Electronic device for generating corrected image and operating method for electronic device

Abstract

This electronic device for generating a corrected image comprises a memory storing at least one instruction and at least one processor including processing circuitry, wherein when the at least one processor executes the at least one instruction stored in the memory, the electronic device may: obtain an input image corresponding to a first region and a second region adjacent to the first region and a frame image corresponding to the second region; generate a masking map including information on an object of interest displayed in the second region of the input image on the basis of the input image and the frame image; and generate a corrected image including the input image in the first region and the object of interest and the frame image in the second region by adding the input image and the frame image on the basis of the masking map.

Description

Electronic device for generating a corrected image and method of operating the electronic device

[0001] The present disclosure relates to an electronic device and a method of operating the electronic device. Specifically, the present disclosure relates to an electronic device for generating a corrected image and a method of operating the electronic device.

[0002] With the advancement of electronic device technology, various electronic devices that provide images to users are being developed and distributed.

[0003] At this time, technologies are being developed and used to provide users with diverse user experiences when providing images through electronic devices.

[0004] For example, when providing video through electronic devices, a framed image, such as a black image, a picture frame, or a window frame, can be displayed around the image to provide the user with the experience. Technologies are being developed to enhance the immersion of users of electronic devices in the video provided.

[0005] One embodiment of the present disclosure provides an electronic device for generating a corrected image. The electronic device may include a memory storing at least one instruction. The electronic device may include at least one processor including processing circuitry. By having at least one processor individually or collectively execute at least one instruction stored in the memory, the electronic device may obtain an input image corresponding to a first region and a second region adjacent to the first region, and a frame image corresponding to the second region. By having at least one processor execute at least one instruction, the electronic device may generate a masking map including information about an object of interest displayed in the second region of the input image based on the input image and the frame image. By having at least one processor execute at least one instruction, the electronic device may generate a corrected image including the input image in the first region and the object of interest and the frame image in the second region by adding the input image and the frame image based on the masking map.

[0006] One embodiment of the present disclosure provides an operating method of an electronic device for generating a corrected image. The operating method of the electronic device may include a step of acquiring an input image corresponding to a first region and a second region adjacent to the first region, and a frame image corresponding to the second region. The operating method of the electronic device may include a step of generating a masking map including information about an object of interest displayed in the second region of the input image based on the input image and the frame image. The operating method of the electronic device may include a step of adding the input image and the frame image based on the masking map to generate a corrected image including the input image in the first region and including the object of interest and the frame image in the second region.

[0007] As one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one method of the disclosed operating method on a computer may be provided.

[0008] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0009] The present disclosure may be understood in conjunction with the following detailed description and accompanying drawings, wherein reference numerals refer to structural elements.

[0010] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 2 is a drawing for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 4 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 5 is a flowchart illustrating an operation of generating a masking map according to one embodiment of the present disclosure.

[0015] FIG. 6A is a drawing for explaining a sales map according to one embodiment of the present disclosure.

[0016] FIG. 6b is a diagram illustrating an operation of extracting an object of interest candidate according to one embodiment of the disclosure.

[0017] FIG. 7 is a drawing for explaining a depth map according to one embodiment of the present disclosure.

[0018] FIG. 8 is a drawing for explaining a reduction ratio and a reduced input image according to one embodiment of the present disclosure.

[0019] FIG. 9 is a drawing for explaining a masking map according to one embodiment of the present disclosure.

[0020] FIG. 10 is a drawing for explaining a correction image including an input image in a first region and an object of interest and a frame image in a second region according to one embodiment of the present disclosure.

[0021] FIG. 11 is a flowchart illustrating an operation of generating a correction image by adding a reduced input image and a frame image based on a masking map according to one embodiment of the present disclosure.

[0022] FIG. 12 is a flowchart illustrating an operation of generating a motion compensation image including a moving object according to one embodiment of the present disclosure.

[0023] FIG. 13 is a flowchart illustrating an operation of extracting a moving object based on the size of the object included in an input image according to one embodiment of the present disclosure.

[0024] FIG. 14 is a diagram for explaining an operation of generating a movement compensation image including a moving object according to one embodiment of the present disclosure.

[0025] FIG. 15 is a diagram for explaining an operation of extracting a moving object from an input image based on motion information of the input image according to one embodiment of the present disclosure.

[0026] FIG. 16 is a diagram for explaining an operation of calculating an expected position of a moving object in the next frame according to one embodiment of the present disclosure.

[0027] FIG. 17 is a drawing for explaining a moving correction image including a moving object according to one embodiment of the present disclosure.

[0028] FIG. 18 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0029] The terms used in this disclosure will be briefly described, and one embodiment of the present disclosure will be specifically described.

[0030] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0031] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0032] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like described herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0033] The expression “configured to” as used herein can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system is “capable of” doing something together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0034] Additionally, when a component is referred to as being “connected” or “connected” to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0035] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0036] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0037] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0038] Referring to FIG. 1, in one embodiment of the present disclosure, an electronic device (100) may be a device that displays an image (200) and provides it to a user. The electronic device (100) may be a device that projects the image (200) onto space. The electronic device (100) may project the image (200) onto space and provide the image (200) to a user using the electronic device (100).

[0039] In one embodiment of the present disclosure, FIG. 1 illustrates an electronic device (100) as a projector that projects an image (200) onto a space. In one embodiment of the present disclosure, an area in space where the image (200) is projected may be referred to as a projection area (230). The projection area (230) may be an area where the image (200) projected from the electronic device (100) is displayed. In one embodiment of the present disclosure, the projection area (230) may include not only a screen, but also objects, walls, ceilings, etc. included in the space, and is not limited to any one of them.

[0040] In one embodiment of the present disclosure, the electronic device (100) may be a fixed projector that is fixed to a specific location or a mobile projector that can be placed at a desired location within a space.

[0041] In one embodiment of the present disclosure, the electronic device (100) is illustrated in FIG. 1 as having a circular shape, but the present disclosure is not limited thereto. The electronic device (100) may have various shapes, such as a square shape.

[0042] Additionally, although FIG. 1 illustrates that the electronic device (100) is positioned at a specific location facing the projection area (230), the present disclosure is not limited thereto. Depending on how the electronic device (100) provides the image (200), the relationship between the location where the electronic device (100) is positioned and the projection area (230) may be determined. For example, the projection area (230) may be positioned in an upward direction from the location where the electronic device (100) is positioned.

[0043] In one embodiment of the present disclosure, the electronic device (100) may include an image display unit (110) that projects an image (200). The electronic device (100) may provide the image (200) to a user through the image display unit (110).

[0044] However, the present disclosure is not limited thereto. The electronic device (100) may be implemented as an electronic device of various shapes, such as a television, a mobile device, a smart phone, a laptop computer, a desktop, a tablet PC, a digital broadcasting terminal, and a wearable device. In one embodiment of the present disclosure, the electronic device (100) may be implemented as an electronic device of various shapes, and may display an image (200) through an image display unit (110) to provide the image (200) to a user (400).

[0045] In one embodiment of the present disclosure, the image display unit (110) may include a display area. In one embodiment of the present disclosure, the image (200) shown in FIG. 1 may be projected through the image display unit (110) and displayed on a projection area (230). In one embodiment of the present disclosure, the area of ​​the projection area (230) may be larger than the area of ​​the display area. As the distance between the image display unit (110) and the projection area (230) increases, the area of ​​the projection area (230) may become larger.

[0046] However, the present disclosure is not limited thereto. If the electronic device (100) is an electronic device including a display that displays an image in a display area, such as a television, a mobile device, a smart phone, a laptop computer, a desktop, a tablet PC, a digital broadcasting terminal, or a wearable device, it goes without saying that the image displayed in the display area can be provided to the user.

[0047] In one embodiment of the present disclosure, the projection area (230) may include a first projection area (210) and a second projection area (220). The display area may include a first area and a second area.

[0048] In one embodiment of the present disclosure, the spatial projection area (230) may be an area corresponding to the display area of ​​the image display unit (110). The first spatial projection area (210) may be an area corresponding to the first area of ​​the image display unit (110). The second spatial projection area (220) may be an area corresponding to the second area of ​​the image display unit (110).

[0049] Hereinafter, for convenience of explanation, the projection area (230) illustrated in FIG. 1 is described as a display area (230). The first projection area (210) is described as a first area (210). The second projection area (220) is described as a second area (220).

[0050] However, if the electronic device (100) is a projector that projects an image (200), the image displayed in the first area of ​​the image display unit (110) is projected onto the first projection area (210), and the image displayed in the second area of ​​the image display unit (110) is projected onto the second projection area (220) and provided to the user.

[0051] In one embodiment of the present disclosure, a first image (211) may be displayed in a first region (210). A second image (221) may be displayed in a second region (220). In one embodiment of the present disclosure, the first image (211) may be an image generated based on an input image. The second image (221) may be an image generated based on a frame image.

[0052] In one embodiment of the present disclosure, the "input image" may be an image corresponding to the first region (210) and a second region (220) adjacent to the first region (210). In one embodiment of the present disclosure, the "frame image" may be an image corresponding to the second region (220). The frame image may be an image displayed around the input image and serving as a background for the input image.

[0053] In one embodiment of the present disclosure, since a second image (221) corresponding to a frame image is displayed in the second region (220), an image corresponding to the second region (220) among the input images may not be displayed. An image corresponding to the first region (210) among the input images may be displayed in the first region (210) as a first image (211).

[0054] However, the present disclosure is not limited thereto. The first image (211) may be an image whose resolution is adjusted so that the input image corresponds to the first area (210). In one embodiment of the present disclosure, the first image (211) may be an image whose resolution is adjusted so that the input image is entirely displayed in the first area (210). In addition, the first image (211) may be an image whose resolution is adjusted so that a portion of the image corresponding to the second area (220) among the input images is displayed in the first area (210).

[0055] In one embodiment of the present disclosure, the electronic device (100) can generate a corrected image including a first image (211), which is an image corresponding to a first area (210) of an input image, and a second image (221) generated based on a frame image. The electronic device (100) can project the corrected image through the image display unit (110) and provide it to a user.

[0056] In one embodiment of the present disclosure, the electronic device (100) can display a first image (211) generated based on an input image in a first area (210) and provide a second image (221) generated based on a frame image in a second area (220) surrounding the first area (210) to a user. Accordingly, the electronic device (100) can provide a high sense of immersion to a user using the electronic device (100).

[0057] In one embodiment of the present disclosure, the input image may be an image including a first object (212), a second object (213), and a background image. The first object (212) may be an object image corresponding to the first area (210). The second object (213) may be an object image at least partially corresponding to the second area (220).

[0058] In one embodiment of the present disclosure, the electronic device (100) can obtain a saliency map including saliency information of the input image based on the input image. The electronic device (100) can obtain a depth map including depth information of the input image based on the input image.

[0059] Below, a detailed explanation of the salience map and depth map will be provided later.

[0060] In one embodiment of the present disclosure, the electronic device (100) may extract an object of interest from an input image based on a saliency map, a depth map, and a frame image. In this case, the "object of interest" may be an object with a high saliency to the user among the second objects (213) corresponding to the second region (220). In addition, the object of interest may be an object with a large depth value among the objects with a high saliency to the user, i.e., an object determined to be close to the user.

[0061] In this case, the electronic device (100) can generate a second image (221) based on the frame image and the object of interest. The electronic device (100) can generate a corrected image including a first image (211), which is an image corresponding to a first area (210) of the input image, and a second image (221) including a portion corresponding to a second area (220) of the frame image and the object of interest. The electronic device (100) can project the corrected image through the image display unit (110) and provide it to the user.

[0062] In one embodiment of the present disclosure, the electronic device (100) may display a first image (211) generated based on an image corresponding to the first region (210) among input images in the first region (210). At this time, the first image (211) may include a portion corresponding to the first region (210) among objects of interest. In one embodiment of the present disclosure, if all objects of interest correspond to the second region (220), the first image (211) may not include the objects of interest.

[0063] In one embodiment of the present disclosure, the electronic device (100) can display a second image (221) generated based on a frame image and a portion of an object of interest corresponding to the second area (220) in the second area (220).

[0064] In one embodiment of the present disclosure, the electronic device (100) can provide a three-dimensional image (200) to the user by displaying an object of interest that is judged to have high user saliency and is close to the user in a second area (220) through a corrected image.

[0065] However, the present disclosure is not limited thereto. Even if an object of interest is extracted from an input image, if the resolution of the input image is adjusted to correspond to the first region (210), the second image (221) may only include frame images.

[0066] In one embodiment of the present disclosure, the input image may include a third object (214). The third object (214) may be an object whose position moves across a plurality of frames.

[0067] In one embodiment of the present disclosure, the electronic device (100) can obtain motion information by estimating the movement of the input image based on the input image. Based on the obtained motion information, the electronic device (100) can extract a third object (214) from the input image. At this time, the electronic device (100) can also extract a third object (214) having a size greater than a preset value among the third objects (214) as a moving object.

[0068] In one embodiment of the present disclosure, the electronic device (100) can calculate the position of a moving object in the next frame. When the calculated position of the moving object is located in the second area (220), the electronic device (100) can generate a second image (221) including the moving object in the next frame.

[0069] In one embodiment of the present disclosure, the electronic device (100) may generate a first image (211) corresponding to a first region (210) of an input image, and generate a motion correction image including a frame image and a portion corresponding to a second region (220) of a moving object. However, the present disclosure is not limited thereto, and the motion correction image may of course include a frame image, a portion corresponding to a second region (220) of an object of interest, and a portion corresponding to the second region (220) of a moving object.

[0070] In one embodiment of the present disclosure, the electronic device (100) can project a movement correction image through the image display unit (110) and provide it to the user.

[0071] In one embodiment of the present disclosure, when a third object (214) moving across a plurality of frames is expected to be located in a second area (220), the electronic device (100) can provide a three-dimensional image (200) to a user by displaying the third object (214) in the second area (220) through a motion compensation image.

[0072] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the present disclosure.

[0073] FIG. 2 is a drawing for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0074] Referring to FIGS. 1 and 2, in one embodiment of the present disclosure, an electronic device (100) may include a video display unit (110), a memory (120), at least one processor (130), an input / output interface (140), and a communication interface (150). However, not all of the components illustrated in FIG. 2 are essential components.

[0075] In one embodiment of the present disclosure, the electronic device (100) may be implemented with more components than those illustrated in FIG. 2, or may be implemented with fewer components. The image display unit (110), the memory (120), at least one processor (130), the input / output interface (140), and the communication interface (150) may each be electrically and / or physically connected to one another.

[0076] In one embodiment of the present disclosure, the image display unit (110) can project the image (200) by generating light to display the image (200). The image display unit (110) may also be referred to as a projection unit or a display unit. The image display unit (110) may include various detailed components such as a light source, a projection lens, and a reflector.

[0077] In one embodiment of the present disclosure, the image display unit (110) can project an image (200) by generating light using various projection methods, for example, a CRT (cathode-ray tube) method, an LCD (Liquid Crystal Display) method, a DLP (Digital Light Processing) method, a laser method, etc.

[0078] In one embodiment of the present disclosure, the image display unit (110) may include various types of light sources. For example, the image display unit (110) may include at least one light source among a lamp, an LED, and a laser.

[0079] In one embodiment of the present disclosure, the image display unit (110) can output the image (200) with a 4:3 screen ratio, a 5:4 screen ratio, a 16:9 wide screen ratio, etc., depending on the purpose of the electronic device (100) or the user's settings. The image display unit (110) can output the image (200) with various resolutions, such as WVGA (854*480), SVGA (800*600), XGA (1024*768), WXGA (1180*720), WXGA (1180*800), SXGA (1180*1024), UXGA (1600*1100), Full HD (1920*1080), UHD (3840*2160), etc., depending on the determined screen ratio.

[0080] In one embodiment of the present disclosure, memory (120) may store instructions, data structures, and program codes that can be read by at least one processor (130). In one embodiment of the present disclosure, there may be more than one memory (120). In the disclosed embodiment, operations performed by at least one processor (130) may be implemented by executing instructions or codes of a program stored in memory (120).

[0081] In one embodiment of the present disclosure, the memory (120) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a Mask ROM, a Flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD).

[0082] In one embodiment of the present disclosure, the memory (120) may not exist separately and may be configured to be included in at least one processor (130).

[0083] The memory (120) may store instructions or program codes for performing functions or operations of the electronic device (100). The instructions, algorithms, data structures, program codes, and application programs stored in the memory (120) may be implemented in a programming or scripting language such as, for example, C, C++, Java, Python, or an assembler.

[0084] In one embodiment of the present disclosure, various types of modules that can be used to generate a correction image or a moving correction image through the image display unit (110) and provide it to a user can be stored in the memory (120).

[0085] In one embodiment of the present disclosure, the memory (120) may include an input image module (121), a correction image generation module (122), a movement correction image generation module (123), and an image display module (124).

[0086] However, not all modules illustrated in FIG. 2 are essential modules. The memory (120) may store more or fewer modules than the modules illustrated in FIG. 2.

[0087] In one embodiment of the present disclosure, a 'module' included in the memory (120) may mean a unit that processes a function or operation performed by at least one processor (130). The 'module' included in the memory (120) may be implemented as software such as instructions, an algorithm, a data structure, or a program code.

[0088] In one embodiment of the present disclosure, the input image module (121) may include commands or program codes relating to an operation or function of obtaining an input image for generating an image (200) to be displayed through the electronic device (100).

[0089] In one embodiment of the present disclosure, at least one processor (130) can obtain an input image for generating an image (200) to be displayed through the electronic device (100) by executing commands or program codes of the input image module (121).

[0090] In one embodiment of the present disclosure, the input image module (121) may include commands or program codes relating to an operation or function of obtaining a frame image for generating an image (200) to be displayed through the electronic device (100).

[0091] In one embodiment of the present disclosure, at least one processor (130) can obtain a frame image for generating an image (200) to be displayed through the electronic device (100) by executing commands or program codes of an input image module (121).

[0092] In one embodiment of the present disclosure, the input image module (121) may include commands or program codes related to an operation or function of obtaining at least one of an input image or a frame image from an external electronic device through an input / output interface (140).

[0093] In one embodiment of the present disclosure, at least one processor (130) can obtain at least one of an input image or a frame image from an external electronic device through an input / output interface (150) by executing commands or program codes of an input image module (121).

[0094] The input image module (121) may include commands or program codes related to an operation or function of obtaining at least one of an input image or a frame image from an external server through a communication interface (150).

[0095] In one embodiment of the present disclosure, at least one processor (130) can obtain at least one of an input image or a frame image from an external server through a communication interface (150) by executing commands or program codes of an input image module (121).

[0096] However, the present disclosure is not limited thereto, and at least one processor (130) may obtain an input image or frame image stored in advance in the memory (120) by executing commands or program codes of the input image module (121).

[0097] In one embodiment of the present disclosure, the correction image generation module (122) may include commands or program codes relating to an operation or function of generating a correction image.

[0098] In one embodiment of the present disclosure, at least one processor (130) can generate a corrected image by executing instructions or program code of a corrected image generation module (122).

[0099] Hereinafter, the correction image generation module (122) and the operation of generating the correction image will be described later with reference to FIGS. 4 to 11.

[0100] In one embodiment of the present disclosure, the motion compensation image generation module (123) may include commands or program codes relating to an operation or function of generating a motion compensation image.

[0101] In one embodiment of the present disclosure, at least one processor (130) can generate a corrected image by executing instructions or program codes of a motion correction image generation module (123).

[0102] Hereinafter, the movement correction image generation module (122) will be described later with reference to FIGS. 12 to 17.

[0103] In one embodiment of the present disclosure, the image display module (124) may include commands or program codes relating to an operation or function for controlling the image display unit (110) to display a corrected image or a moving corrected image.

[0104] In one embodiment of the present disclosure, at least one processor (130) can control the image display unit (110) to display a corrected image or a movement corrected image by executing instructions or program codes of the image display module (124).

[0105] In one embodiment of the present disclosure, at least one processor (130) may be configured as one or more processors, which control a series of processes to cause the electronic device (100) to operate according to the embodiments described below.

[0106] In one embodiment of the present disclosure, at least one processor (130) may be configured as at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), an Application Specific Integrated Circuits (ASICs), a Digital Signal Processor (DSPs), a Digital Signal Processing Device (DSPDs), a Programmable Logic Device (PLDs), a Field Programmable Gate Array (FPGAs), a Communication Processor (CP), a Neural Processing Unit, or an artificial intelligence (AI) processor designed with a hardware structure specialized for learning and processing an artificial intelligence (AI) model, but is not limited thereto.

[0107] In one embodiment of the present disclosure, if one or more processors included in at least one processor (130) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0108] In one embodiment of the present disclosure, at least one processor (130) may be configured as a circuit (Circuitry) such as a System on Chip (SoC) or an Integrated Circuit (IC).

[0109] In one embodiment of the present disclosure, at least one processor (130) can execute various types of modules stored in the memory (120). At least one processor (130) can execute at least one instruction constituting the various types of modules stored in the memory (120). By executing the program or at least one instruction stored in the memory (120), at least one processor (130) can process data according to predefined operation rules or artificial intelligence models.

[0110] In one embodiment of the present disclosure, at least one processor (130) can execute at least one module among an input image module (121), a correction image generation module (122), a motion correction image generation module (123), or an image display module (124) stored in a memory (120).

[0111] In one embodiment of the present disclosure, at least one processor (130) may include a plurality of processors.

[0112] In one embodiment of the present disclosure, at least one module among the input image module (121), the correction image generation module (122), the movement correction image generation module (123), or the image display module (124) stored in the memory (120) can be executed by any one of the plurality of processors.

[0113] The remaining modules among the input image module (121), the correction image generation module (122), the movement correction image generation module (123), or the image display module (124) stored in the memory (120) can be executed by another processor among the plurality of processors.

[0114] In one embodiment of the present disclosure, the input / output interface (140) can receive at least one of image data or audio data from an external electronic device, etc., under the control of at least one processor (130). At least one processor (130) can obtain an input image from the external electronic device through the input / output interface (140).

[0115] In one embodiment of the present disclosure, the input / output interface (140) may perform input / output operations with an external electronic device using at least one of input / output methods including an HDMI port (High-Definition Multimedia Interface port), a DVI (Digital Visual Interface), a component jack, a PC port, or a USB port (Universal Serial Bus port). However, the present disclosure is not limited to the above-described input / output methods.

[0116] In one embodiment of the present disclosure, the communication interface (150) can perform data communication with an external server or an external electronic device under the control of at least one processor (130). The communication interface (150) can perform data communication with an external server or an external electronic device using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0117] In one embodiment of the present disclosure, at least one processor (130) may receive an input image from an external server or an external electronic device via a communication interface (150). At least one processor (130) may also provide a corrected image or a motion corrected image to the external server or an external electronic device via the communication interface (150).

[0118] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0119] Referring to FIGS. 1, 2 and 3, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S100) of obtaining an input image corresponding to a first region (210) and a second region (220) and a frame image corresponding to the second region (220).

[0120] In the step (S100) of obtaining an input image and a frame image, the electronic device (100) can obtain the input image and the frame image through an input / output interface (140) or a communication interface (150).

[0121] Although step (S100) of acquiring an input image and a frame image is illustrated as a single step in FIG. 3, the present disclosure is not limited thereto. It goes without saying that the input image and the frame image may be acquired in separate steps.

[0122] Additionally, the electronic device (100) may also utilize input images and frame images that have already been acquired.

[0123] In one embodiment of the present disclosure, an operating method of an electronic device (100) may include a step (S200) of generating a masking map including information about an object of interest displayed in a second area (220) of an input image based on an input image and a frame image.

[0124] In the step of generating a masking map (S200), the electronic device (100) can generate a masking map including information about an object of interest displayed in a second area (220) of the input image based on the input image and the frame image.

[0125] Hereinafter, the masking map and the step of generating the masking map (S200) will be described later with reference to FIGS. 4 to 9.

[0126] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S300) of adding an input image and a frame image based on a masking map to generate a corrected image including the input image in a first area (210) and including an object of interest and a frame image in a second area (220).

[0127] In the step (S300) of generating a correction image, the electronic device (100) can add an input image and a frame image based on a masking map to generate a correction image including the input image in the first area (210) and including the object of interest and the frame image in the second area (220).

[0128] Hereinafter, the step (S300) of generating a correction image will be described later with reference to FIGS. 4, 10, and 11.

[0129] FIG. 4 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure. FIG. 5 is a flowchart illustrating the operation of generating a masking map according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 3 are assigned the same reference numerals, and redundant descriptions are omitted.

[0130] Referring to FIGS. 1, 2, 4, and 5, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S110) of obtaining a depth map including depth information of an input image based on an input image.

[0131] In one embodiment of the present disclosure, the step of obtaining a depth map (S110) may be performed after the step of obtaining an input image and a frame image (620, see FIG. 6A) (S100).

[0132] In the present disclosure, a 'depth map' means an image including information on a depth value from a reference point of view to the surface of an object. In the present disclosure, a 'depth value' means a value that numerically represents information on the distance from the reference point to the surface of an object. In one embodiment of the present disclosure, a depth value corresponding to a surface located closer to the reference point of view among the surface of an object may be greater than a depth value corresponding to a surface located further from the reference point of view among the surface of the object. In one embodiment of the present disclosure, depth information may mean a depth value.

[0133] In one embodiment of the present disclosure, in the step of obtaining a depth map (S110), the electronic device (100) can obtain a depth map including depth information based on an input image. In one embodiment of the present disclosure, at least one processor (130) can generate the depth map by executing instructions or program codes of a correction image generation module (122).

[0134] In one embodiment of the present disclosure, the correction image generation module (122) may include a depth map module (500). The depth map module (500) may include instructions or program codes related to an operation or function of obtaining a depth map including depth information of the input image based on the input image.

[0135] In one embodiment of the present disclosure, the depth map module (500) may include an artificial intelligence model. The artificial intelligence model included in the depth map module (500) may include a machine learning or deep learning model. In one embodiment of the present disclosure, the artificial intelligence model included in the depth map module (500) may be an artificial intelligence model trained to receive an image as input and infer a depth map including depth information.

[0136] In one embodiment of the present disclosure, at least one processor (130) can obtain a depth map including depth information based on an input image by executing instructions or program code of a depth map module (500).

[0137] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S120) of obtaining a saliency map including saliency information of the input image based on the input image.

[0138] In the present disclosure, a "saliency map" refers to an image containing information on saliency values ​​for objects or background images included in an input image. In one embodiment of the present disclosure, "saliency value" may refer to importance, interest, etc.

[0139] In one embodiment of the present disclosure, an area with a high saliency value may be an area of ​​high user interest or an area judged to be of high importance. An area with a low saliency value may be an area of ​​high user interest or an area judged to be of high importance. In one embodiment of the present disclosure, the saliency information may refer to a saliency value.

[0140] In one embodiment of the present disclosure, in the step of obtaining a saliency map (S120), the electronic device (100) can obtain a saliency map including saliency information based on an input image. In one embodiment of the present disclosure, at least one processor (130) can generate the saliency map by executing commands or program codes of a correction image generation module (122).

[0141] In one embodiment of the present disclosure, the correction image generation module (122) may include a saliency map module (510). The saliency map module (510) may include instructions or program codes related to an operation or function of obtaining a saliency map including saliency information of the input image based on the input image.

[0142] In one embodiment of the present disclosure, the salience map module (510) may include an algorithm that utilizes methods such as fixation prediction or salient object detection.

[0143] In one embodiment of the present disclosure, the saliency map module (510) may include an artificial intelligence model. The artificial intelligence model included in the saliency map module (510) may include a machine learning or deep learning model. In one embodiment of the present disclosure, the artificial intelligence model included in the saliency map module (500) may be an artificial intelligence model trained to receive an image as input and infer a saliency map including saliency information.

[0144] In one embodiment of the present disclosure, the artificial intelligence models included in the depth map module (500) and the saliency map module (510) respectively include a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), a VAE (Variational Auto Encoder), and a transformer, and the artificial intelligence models included in the depth map module (500) and the saliency map module (510) in the present disclosure are not limited to the examples described above.

[0145] In one embodiment of the present disclosure, at least one processor (130) can obtain a saliency map including saliency information based on an input image by executing instructions or program codes of a saliency map module (510).

[0146] In one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step of extracting an object of interest from an input image based on a frame image, a depth map, and a saliency map.

[0147] In one embodiment of the present disclosure, in the step of extracting an object of interest from an input image, the electronic device (100) can extract an object of interest from the input image based on a frame image, a depth map, and a saliency map.

[0148] In one embodiment of the present disclosure, at least one processor (130) can extract an object of interest from an input image by executing instructions or program code of a correction image generation module (122).

[0149] In one embodiment of the present disclosure, the correction image generation module (122) may include an object of interest module (520). The object of interest module (520) may include instructions or program codes related to an operation or function of extracting an object of interest from an input image based on a frame image, a depth map, and a saliency map.

[0150] At least one processor (130) can extract an object of interest from an input image based on a frame image, a depth map, and a saliency map by executing instructions or program codes of an object of interest module (520).

[0151] In one embodiment of the present disclosure, the step of extracting an object of interest from an input image may include a step (S130) of extracting an object of interest candidate from the input image based on a saliency map and a frame image.

[0152] In one embodiment of the present disclosure, in the step (S130) of extracting an object of interest candidate from an input image, at least one processor (130) may extract an object of interest candidate from the input image based on a saliency map and a frame image by executing commands or program codes of an object of interest module (520).

[0153] In one embodiment of the present disclosure, the step of extracting an object of interest from an input image may include a step (S140) of extracting an object of interest candidate as an object of interest based on a depth map, when the object of interest candidate has a depth value greater than a preset second value.

[0154] In one embodiment of the present disclosure, in the step (S140) of extracting an object of interest candidate as an object of interest, at least one processor (130) can extract an object of interest candidate as an object of interest based on a depth map by executing instructions or program code of an object of interest module (520).

[0155] Hereinafter, the operation of the electronic device (100) in the step of extracting an object of interest candidate from an input image (S130) and the step of extracting an object of interest candidate as an object of interest (S140) will be described later with reference to FIGS. 6a to 8.

[0156] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S150) of calculating a reduction ratio for lowering the input resolution of an input image based on an object of interest.

[0157] In one embodiment of the present disclosure, the input image may be an image having an input resolution. The first region (210) may be an area having a first resolution. In this case, the first region (210) having a first resolution may mean that the first region (210) is an area in which an image having the first resolution can be displayed.

[0158] In one embodiment of the present disclosure, the input resolution may be greater than the first resolution. The input resolution may be a resolution corresponding to the first region (210) and the second region (220). The first resolution may be a resolution corresponding to the first region (210).

[0159] In the step (S150) of calculating a reduction ratio for lowering the input resolution of the input image, the electronic device (100) can calculate a reduction ratio for lowering the input resolution of the input image based on the object of interest.

[0160] In one embodiment of the present disclosure, at least one processor (130) can calculate a reduction ratio for lowering the input resolution of an input image based on an object of interest by executing instructions or program code of a correction image generation module (122).

[0161] In one embodiment of the present disclosure, the correction image generation module (122) may include an image reduction judgment module (530). The image reduction judgment module (520) may include instructions or program codes relating to an operation or function of calculating a reduction ratio that reduces the input resolution of an input image based on an object of interest.

[0162] At least one processor (130) can calculate a reduction ratio for lowering the input resolution of an input image based on an object of interest by executing instructions or program codes of an image reduction judgment module (530).

[0163] Hereinafter, the operation of the electronic device (100) in the step (S150) of calculating the reduction ratio for lowering the input resolution of the input image will be described later in FIG. 8.

[0164] In one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S210) of generating a masking map based on a reduction ratio, a frame image, and an object of interest.

[0165] In one embodiment of the present disclosure, in the step of generating a masking map (S210), the electronic device (100) can generate a masking map based on a reduction ratio, a frame image, and an object of interest.

[0166] In one embodiment of the present disclosure, at least one processor (130) can generate a masking map based on a reduction ratio, a frame image, and an object of interest by executing instructions or program code of a correction image generation module (122).

[0167] In one embodiment of the present disclosure, the correction image generation module (122) may include a masking map generation module (540). The masking map generation module (540) may include instructions or program codes relating to an operation or function of generating a masking map based on a reduction ratio, a frame image, and an object of interest.

[0168] At least one processor (130) can generate a masking map based on a reduction ratio, a frame image, and an object of interest by executing instructions or program code of a masking map generation module (540).

[0169] Hereinafter, the operation of the electronic device (100) in the step (S210) of generating a masking map based on the reduction ratio, frame image, and object of interest will be described later in FIG. 9.

[0170] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S300) of generating a correction image by adding an input image and a frame image based on a masking map generated based on a reduction ratio, a frame image, and an object of interest.

[0171] In one embodiment of the present disclosure, at least one processor (130) can generate a corrected image by adding an input image and a frame image based on a masking map generated based on a reduction ratio, a frame image, and an object of interest by executing instructions or program codes of a corrected image generation module (122).

[0172] In one embodiment of the present disclosure, the correction image generation module (122) may include an image generation module (560). The image generation module (560) may include instructions or program codes relating to an operation or function of generating a correction image by adding an input image and a frame image based on a masking map.

[0173] At least one processor (130) can generate a correction image by adding an input image and a frame image based on a masking map by executing instructions or program codes of an image generation module (560).

[0174] Hereinafter, the operation of the electronic device (100) in the step (S300) of generating a correction image by adding an input image and a frame image based on a masking map will be described later in FIG. 10.

[0175] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S220, see FIG. 11) of generating a reduced input image by lowering the input resolution of the input image based on a reduction ratio.

[0176] In one embodiment of the present disclosure, the resolution of the reduced input image may be equal to or greater than the first resolution of the first region (210).

[0177] In one embodiment of the present disclosure, in the step (S230) of generating a reduced input image by lowering the input resolution of the input image based on the reduction ratio, the electronic device (100) can generate a reduced input image by lowering the input resolution of the input image based on the reduction ratio.

[0178] In one embodiment of the present disclosure, at least one processor (130) can generate a reduced input image by lowering the input resolution of the input image based on a reduction ratio by executing instructions or program code of the correction image generation module (122).

[0179] In one embodiment of the present disclosure, the correction image generation module (122) may include an image reduction module (550). The image reduction module (550) may include instructions or program codes relating to an operation or function of generating a reduced input image by reducing the input resolution of the input image based on a reduction ratio.

[0180] At least one processor (130) can generate a reduced input image by lowering the input resolution of the input image based on a reduction ratio by executing instructions or program code of the image reduction module (550).

[0181] In one embodiment of the present disclosure, in the step (S300) of generating a corrected image by lowering the input resolution of the input image to generate a reduced input image, the corrected image can be generated by adding the reduced input image and the frame image based on the masking map.

[0182] Hereinafter, the operation of the electronic device (100) in the step (S210) of generating a reduced input image by lowering the input resolution of the input image based on the reduction ratio and the step (S310) of generating a corrected image by adding the reduced input image and the frame image will be described later with reference to FIGS. 8, 10, and 11.

[0183] However, the present disclosure is not limited thereto. It goes without saying that at least one of the steps (S100 to S300) of the operating method of the electronic device (100) illustrated in FIG. 4 may be omitted, or at least one step may be added. Furthermore, two or more of the steps (S100 to S300) may be performed in parallel. Furthermore, it goes without saying that the steps may be performed in a different order than the order illustrated in FIG. 4.

[0184] FIG. 6A is a diagram illustrating a sales map according to one embodiment of the present disclosure. FIG. 6A is a diagram illustrating an operation for extracting an object of interest candidate according to one embodiment of the present disclosure.

[0185] Referring to FIG. 6A, in one embodiment of the present disclosure, FIG. 6A illustrates an input image (600) and a saliency map (610) generated based on the input image (600).

[0186] Referring to FIGS. 2, 5, and 6A, in one embodiment of the present disclosure, an input image (600) may include a first object (601) and a second object (602). However, the present disclosure is not limited thereto, and the input image (600) may also include at least one other object and a background image.

[0187] In one embodiment of the present disclosure, at least one processor (130) can obtain a saliency map (610) including saliency information of an input image (600) by executing instructions or program codes of a saliency map module (510).

[0188] In one embodiment of the present disclosure, the saliency map (610) may include a first saliency value (611) of a first object and a second saliency value (612) of a second object. However, the present disclosure is not limited thereto, and as the input image (600) further includes a background image and other objects, the saliency map (610) may further include saliency information of the background image and other objects.

[0189] In one embodiment of the present disclosure, an object that is determined to have a high user interest or is determined to have a high importance in the input image (600) may have a high salience value.

[0190] In one embodiment of the present disclosure, FIG. 6A illustrates that objects with large saliency values ​​are brighter than objects with small saliency values ​​in the saliency map (610). However, this is for convenience of explanation and is not limited thereto. In one embodiment of the present disclosure, the saliency map (610) may be generated so that the input image (600) is divided into objects with saliency values ​​greater than a preset reference value and objects with saliency values ​​equal to or less than the reference value.

[0191] In one embodiment of the present disclosure, the second saliency value (612) may be greater than the first saliency value (611). The second object (602) may be an object determined to have a higher user interest than the first object (601). The second object (602) may be an object determined to have a higher importance than the first object (601).

[0192] In one embodiment of the present disclosure, at least one processor (130) may extract an object having a saliency value greater than a preset first value among the input image (600) based on the saliency map (610). In this case, the "preset first value" may be a value that is preset because it is judged to be of high interest to the user or of high importance within the input image (600) to the extent that it is displayed and provided to the user in the second area (220) illustrated in FIG. 6B.

[0193] In one embodiment of the present disclosure, when it is determined that the second saliency value (612) is greater than the preset first value, at least one processor (130) can extract the second object (602) from the input image (600).

[0194] Referring to FIG. 6B, in one embodiment of the present disclosure, a saliency map (610) and a frame image (620) are illustrated in FIG. 6B. In one embodiment of the present disclosure, for convenience of explanation, the frame image (620) is displayed overlapping the saliency map (610).

[0195] In one embodiment of the present disclosure, the area where the frame image (620) is displayed may be the second area (220). The frame image (620) may be displayed around the first area (210). Although FIG. 6B illustrates the frame image (620) as an image surrounding the first area (210), the present disclosure is not limited thereto. In one embodiment of the present disclosure, the frame image (620) may not surround the first area (210). In one embodiment of the present disclosure, the frame image (620) may be displayed only in some areas among areas adjacent to the first area (210) and may not be displayed in the remaining areas. In one embodiment of the present disclosure, the frame image (620) may have a frame width (621). In this case, the width of the second area (220) may also be determined by the frame width (621). In one embodiment of the present disclosure, the width of the frame image (620) is depicted as being constant as the frame width (621) in FIG. 6B, but the present disclosure is not limited thereto. It goes without saying that the frame image (620) may have different frame widths depending on the location.

[0196] In addition, although FIG. 6B illustrates that the frame image (620) includes only an image corresponding to the second region (220), the present disclosure is not limited thereto. The frame image (620) may have images corresponding to the first region (210) and the second region (220), respectively. The image corresponding to the first region (210) among the frame image (620) may be masked by a masking map (900, see FIG. 9) to be described later. An image of an area among the frame image (620) that is not masked by the masking map (900) may be provided to the user together with the input image (600).

[0197] Referring to FIGS. 6A and 6B, in one embodiment of the present disclosure, at least one processor (130) can extract, as a candidate object of interest, an object that has a saliency value greater than a preset first value and corresponds to a second area (220) among at least one object included in an input image (600).

[0198] In one embodiment of the present disclosure, at least one processor (130) may extract the second object (602) as an object of interest candidate as the second object (602) corresponds to the second area (220) in which the frame image (620) is displayed.

[0199] In one embodiment of the present disclosure, at least one processor (130) can extract a second object (602) from an input image (600) as an object of interest candidate based on a frame image (620) and a saliency map (610) by executing instructions or program codes of an object of interest module (520, see FIG. 5).

[0200] In one embodiment of the present disclosure, at least one processor (130) may not extract the second object (602) as an object of interest candidate if the second saliency value (612) is equal to or less than the preset first value. In this case, an object to be extracted as an object of interest candidate may not exist in the input image (600).

[0201] FIG. 7 is a drawing for explaining a depth map according to one embodiment of the present disclosure.

[0202] Referring to FIG. 7, in one embodiment of the present disclosure, FIG. 7 illustrates an input image (600) and a depth map (700) generated based on the input image (600).

[0203] Referring to FIGS. 2, 5, 6a and 7, in one embodiment of the present disclosure, an input image (600) may include a first object (601) and a second object (602).

[0204] In one embodiment of the present disclosure, at least one processor (130) can obtain a depth map (700) including depth information of an input image (600) by executing instructions or program code of a depth map module (500).

[0205] In one embodiment of the present disclosure, the depth map (700) may include a first depth value (710) of a first object and a second depth value (720) of a second object. However, the present disclosure is not limited thereto, and as the input image (600) further includes a background image and other objects, the depth map (700) may further include depth information of the background image and other objects.

[0206] In one embodiment of the present disclosure, the further an object included in an input image (600) is from a reference point, the smaller the depth value the object may have. In one embodiment of the present disclosure, the reference point may be the location of a camera that acquired the input image. However, the present disclosure is not limited thereto, and the reference point may be a location arbitrarily set as a reference point for generating the input image.

[0207] In one embodiment of the present disclosure, FIG. 7 illustrates that objects with larger depth values ​​in the depth map (700) are brighter than objects with smaller depth values. However, this is for convenience of explanation and is not limited thereto.

[0208] In one embodiment of the present disclosure, the second depth value (720) may be greater than the first depth value (710). The distance between the second object (602) and the reference point may be shorter than the distance between the first object (601) and the reference point. The user may feel that the second object (602) is closer than the first object (601).

[0209] In one embodiment of the present disclosure, at least one processor (130) may extract an object having a depth value greater than a preset second value among the input image (600) based on the depth map (700). In this case, the “preset second value” may be a preset value determined to be close enough from the user that the user will not feel a sense of incongruity even if an image having the corresponding depth value is displayed and provided to the user in the second area (220) illustrated in FIG. 6B.

[0210] In one embodiment of the present disclosure, when it is determined that the second depth value (720) is greater than a preset second value, at least one processor (130) can extract a second object (602) from the input image (600).

[0211] Referring to FIGS. 6B and 7, in one embodiment of the present disclosure, at least one processor (130) may compare a depth value of an object of interest candidate extracted based on a saliency map (610) and a frame image (620) based on a depth map (700) with a preset second depth value. When the at least one processor (130) determines that the depth value of the object of interest candidate is greater than the preset second depth value, the at least one processor (130) may extract the object of interest candidate as the object of interest. In this case, extracting as the object of interest may mean defining the object of interest candidate as the object of interest.

[0212] In one embodiment of the present disclosure, at least one processor (130) can extract an object of interest candidate as an object of interest based on a depth map (700) by executing instructions or program code of an object of interest module (520, see FIG. 5).

[0213] In one embodiment of the present disclosure, at least one processor (130) can extract an object corresponding to a second region (220) among at least one object included in an input image (600) and having depth information that does not cause a sense of incongruity to a user as an object of interest.

[0214] Accordingly, even if the object of interest is provided to the user as a second image (221, see FIG. 1) together with the frame image (620) in the second area (220), the user can receive an image (200) having a natural three-dimensional effect from the electronic device (100).

[0215] In one embodiment of the present disclosure, at least one processor (130) may not extract an object of interest candidate as an object of interest if the depth value of the object of interest candidate is equal to or less than a preset second depth value. In this case, an object extracted as an object of interest in the input image (600) may not exist.

[0216] FIG. 8 is a drawing for explaining a reduction ratio and a reduced input image according to one embodiment of the present disclosure.

[0217] Referring to FIG. 8, in one embodiment of the present disclosure, FIG. 8 illustrates an input image (800) having an input resolution and a reduced input image (810) having a reduced resolution based on a reduction ratio.

[0218] Referring to FIGS. 2, 5, 6b, and 8, in one embodiment of the present disclosure, an input image (800) may include a first object (801) and a second object (802). In this case, the second object (802) may be extracted as an object of interest. Hereinafter, for convenience of explanation, the second object (802) will be described as an object of interest (802).

[0219] In one embodiment of the present disclosure, at least one processor (130) can calculate a reduction ratio for lowering the input resolution of an input image (800) based on the saliency map (610), the frame image (620), and the depth map (700) by executing instructions or program codes of the image reduction judgment module (530).

[0220] In one embodiment of the present disclosure, at least one processor (130) can calculate a reduction ratio that reduces the input resolution of the input image (800) based on an object of interest (802) extracted from the input image (800).

[0221] In one embodiment of the present disclosure, at least one processor (130) can calculate a distribution of objects of interest (802) within an input image (800). At least one processor (130) can calculate the distribution of objects of interest (802) within the input image (800) using a module including an algorithm for calculating spatial distribution.

[0222] In one embodiment of the present disclosure, at least one processor (130) can calculate a histogram of an input image (800) and a histogram of an object of interest (802). Using the histogram of the input image (800) and the histogram of the object of interest (802), the at least one processor (130) can determine whether the object of interest (802) is distributed throughout the input image (800) or is densely distributed in a specific area.

[0223] However, the present disclosure is not limited thereto, and at least one processor (130) may calculate the distribution of the object of interest (802) using various algorithms capable of calculating the distribution of the object of interest (802) within the input image (800).

[0224] At least one processor (130) can determine whether the object of interest (802) is widely distributed or densely distributed in the input image (800) through the distribution of the object of interest (802) in the input image (800).

[0225] In one embodiment of the present disclosure, at least one processor (130) can calculate a distribution of objects of interest (802) in a second region (220). The at least one processor (130) can also use the calculated distribution of objects of interest (802) in the second region (220) to determine whether objects of interest (802) are distributed throughout the second region (220) or are densely distributed in a specific region.

[0226] In one embodiment of the present disclosure, the calculated value including the distribution information of the object of interest (802) may have a larger value as the degree to which the object of interest (802) is distributed throughout the second region (220) increases. The calculated value including the distribution information of the object of interest (802) may have a smaller value as the degree to which the object of interest (802) is densely distributed in a specific region among the second region (220) increases.

[0227] In one embodiment of the present disclosure, at least one processor (130) may calculate a reduction ratio for lowering the input resolution of an input image (800) based on the size of a calculated value including distribution information of an object of interest (802). At this time, the reduction ratio may be a real number greater than or equal to 1.

[0228] In one embodiment of the present disclosure, at least one processor (130) may determine a reduction ratio so that the input resolution of the input image (800) is maintained when it is determined that the calculated value is smaller than a predetermined distribution reference value. In one embodiment of the present disclosure, when the reduction ratio is “1”, it may mean that the input resolution of the input image (800) is maintained. However, the present disclosure is not limited thereto, and at least one processor (130) may also determine not to reduce the input image (800) when it is determined that the calculated value is smaller than a predetermined distribution reference value.

[0229] In one embodiment of the present disclosure, the reduced input image (810) is illustrated in FIG. 8 as being reduced in size to have a lower resolution than the input image (800), but the present disclosure is not limited thereto. In one embodiment of the present disclosure, since the reduction ratio is determined such that the input resolution of the input image (800) is maintained, the resolution of the reduced input image (810) may be the same as the input resolution of the input image (800).

[0230] In one embodiment of the present disclosure, at least one processor (130) may determine a reduction ratio so that the input resolution of the input image (800) is reduced when it is determined that the calculated value is equal to or greater than a predetermined distribution reference value. In one embodiment of the present disclosure, when the reduction ratio has a value greater than “1”, the input resolution of the input image (800) may be reduced.

[0231] In one embodiment of the present disclosure, since the reduction ratio is determined so that the input resolution of the input image (800) becomes smaller, the resolution of the reduced input image (810) may be smaller than the input resolution of the input image (800). In one embodiment of the present disclosure, the reduced input image (810) having a resolution smaller than that of the input image (800) may be an image reduced to correspond to an area smaller than that of the input image (800).

[0232] In one embodiment of the present disclosure, reducing the input image (800) may mean lowering the input resolution of the input image (800) so that the input image (800) corresponds to a region with a small area.

[0233] In one embodiment of the present disclosure, at least one processor (130) may calculate a reduction ratio for reducing the input image (800) so that the object of interest (802) corresponding to the second region (220) corresponds to the first region (210) when it is determined that the calculated value is equal to or greater than a predetermined distribution reference value.

[0234] In this case, at least one processor (130) may calculate a reduction ratio so that the object of interest (802) included in the reduced input image (810) is displayed in the first area (210) rather than the second area (220). Referring to FIG. 8, at least one processor (130) may calculate a reduction ratio so that the object of interest (802) can be displayed in the first area (210) by reducing the input image (800). The reduction ratio may be calculated based on the location of the object of interest (802) within the second area (220), the size of the object of interest (802), etc.

[0235] In one embodiment of the present disclosure, when an object extracted as an object of interest is not included in the input image (800), at least one processor (130) can calculate a reduction ratio that reduces the input resolution of the input image (800).

[0236] In one embodiment of the present disclosure, at least one processor (130) can calculate a reduction ratio for reducing an input image (800) corresponding to the first region (210) and the second region (220) to correspond to the first region (210).

[0237] In this case, the reduction ratio can be determined by the sum of the first area (210) and the second area (220) and the ratio of the first area (210).

[0238] In one embodiment of the present disclosure, a reduced input image (810) reduced according to a calculated reduction ratio may correspond to a first region (210). An input image (800) corresponding to the first region (210) and the second region (220) may be reduced to correspond to the first region (210).

[0239] Accordingly, the reduced object of interest included in the reduced input image (810) may correspond to the first region (210). The reduced input image (810) may be displayed as a first image (211) in the first region (210), and the frame image (620) may be displayed as a second image (221) in the second region (220).

[0240] FIG. 9 is a diagram illustrating a masking map according to one embodiment of the present disclosure. FIG. 10 is a diagram illustrating a correction image including an input image in a first region and an object of interest and a frame image in a second region according to one embodiment of the present disclosure.

[0241] Referring to FIG. 9, in one embodiment of the present disclosure, a masking map (900) is illustrated in FIG. 9. In one embodiment of the present disclosure, the masking map (900) may include a first masking area (910) and a second masking area (920).

[0242] Referring to FIGS. 2, 5, 6b, 8 and 9, in one embodiment of the present disclosure, a masking map (900) may be a map generated to mask the input image (600) and the frame image (620), respectively, when generating a correction image (1000) based on the input image (600) and the frame image (620).

[0243] In one embodiment of the present disclosure, at least one processor (130) can generate a masking map (900) based on an object of interest and a frame image (620).

[0244] In one embodiment of the present disclosure, the first masking area (910) may refer to an area corresponding to an object of interest in the first area (210) and the second area (220). The first masking area (910) may be an area that selects components of an input image (600) and masks components of a frame image (620).

[0245] The second masking area (920) may be an area corresponding to the remaining area of ​​the second area (220) excluding the area corresponding to the object of interest. The second masking area (920) may be an area for selecting components of the frame image (620) and masking components of the input image (600).

[0246] In one embodiment of the present disclosure, the masking map (900) can mask the remaining areas of the second region (220) of the input image (600) except for the area corresponding to the object of interest. The input image (600) masked by the masking map (900) can be displayed in the area corresponding to the object of interest among the first region (210) and the second region (220).

[0247] In one embodiment of the present disclosure, the masking map (900) can mask an area corresponding to an object of interest in a first area (210) and a second area (220) of a frame image (620). The frame image (620) masked by the masking map (900) can be displayed in an area remaining in the second area (220) except for the area corresponding to the object of interest.

[0248] However, the present disclosure is not limited thereto, and it goes without saying that the first masking area (910) and the second masking area (920) may vary depending on whether the object of interest is included in the input image (600) and whether the reduced object of interest corresponds to the second area (220) according to the reduction ratio of the input image (600). In one embodiment of the present disclosure, the masking map (900) may be generated using the object of interest and the frame image (620) when the reduction ratio is “1”.

[0249] Referring to FIG. 10, in one embodiment of the present disclosure, a correction image (1000) is illustrated in a first region (210) of FIG. 10. The correction image (1000) may include a first image (211) displayed in the first region (210) and a second image (221) displayed in the second region (220).

[0250] Referring to FIGS. 2, 5, 9, and 10, in one embodiment of the present disclosure, at least one processor (130) may generate a correction image (1000) by adding an input image (600, see FIG. 6a) and a frame image (620, see FIG. 6b) based on a masking map (900).

[0251] At this time, the correction image (1000) can be generated by adding the input image and the masked frame image, each masked by the masking map (900). In one embodiment of the present disclosure, the masking map (900) can include different masking coefficients in the first masking area (910) and the second masking area (920), respectively. At least one processor (130) can generate the correction image (1000) by using the input image (600), the frame image (620), and the masking map (900) having different masking coefficients in the first masking area (910) and the second masking area (920).

[0252] In one embodiment of the present disclosure, the first image (211) may include an image corresponding to a first region (210) among the input image (600). The first image (211) may include an image corresponding to a first region (210) among the objects of interest (213) included in the input image (600).

[0253] In one embodiment of the present disclosure, the second image (221) may include an image corresponding to a second region (220) among the objects of interest (213) included in the input image (600). The second image (221) may include an image corresponding to the remaining region of the second region (220) among the frame images (620) excluding the region corresponding to the object of interest (213).

[0254] In one embodiment of the present disclosure, at least one processor (130) can control the image display unit (110) to display a corrected image (1000). The electronic device (100) can provide the corrected image (1000) to the user through the image display unit (110).

[0255] In one embodiment of the present disclosure, by providing a user with a corrected image (1000) including a first image (211) in a first region (210) and a second image (221) in a second region (220), a high sense of immersion can be provided to a user using an electronic device (100). In addition, by providing frame images (620) of various types (e.g., theater backgrounds, picture frames, windows, audiences, animals, landscapes, etc.) together with an input image (600), a unique experience can be provided to the user.

[0256] In addition, by displaying the object of interest (213) in addition to the frame image (620) in the second area (220), a three-dimensional effect can be provided to the user as if the object of interest (213) is located in front of the first image (211). Accordingly, a three-dimensional image can be provided to the user.

[0257] However, the present disclosure is not limited thereto, and the masking map (900) may also include a third masking area. The third masking area may be an area located between the first masking area (910) and the second masking area (920). The third masking area may be an area that masks some of the components of the input image (600) and some of the components of the frame image (620). The third masking area may be an area for displaying the remaining parts of the components of the input image (600) and the remaining parts of the components of the frame image (620) together. Through this, the input image (600) and the frame image (620) included in the corrected image (1000) may be naturally provided to the user.

[0258] Additionally, the present disclosure is not limited thereto, and at least one processor (130) may generate a masking map (900) based on a reduction ratio, a frame image (620), and an object of interest.

[0259] Referring again to FIGS. 2, 5, 6b, and 9, in one embodiment of the present disclosure, a reduction ratio may be calculated such that the input resolution of the input image (800) becomes smaller depending on the size of the calculated value including the degree of distribution of the object of interest (802). Depending on the reduction ratio, the object of interest included in the input image (600) may be reduced to correspond to the first region (210).

[0260] In this case, the first masking area (910) included in the masking map (900) may be an area corresponding to the first area (210). The second masking area (920) may be an area corresponding to the second area (220).

[0261] In one embodiment of the present disclosure, the masking map (900) can mask an area corresponding to the second area (220) among the input image (600). The input image (600) masked by the masking map (900) can be displayed in the first area (210).

[0262] In one embodiment of the present disclosure, the masking map (900) can mask an area corresponding to the first area (210) among the frame images (620). The frame images (620) masked by the masking map (900) can be displayed in the second area (220).

[0263] In one embodiment of the present disclosure, by providing a user with a corrected image (1000) including a first image (211) in a first region (210) and a second image (221) in a second region (220), a high sense of immersion can be provided to a user using an electronic device (100).

[0264] FIG. 11 is a flowchart illustrating an operation of generating a corrected image by adding a reduced input image and a frame image based on a masking map according to one embodiment of the present disclosure. Hereinafter, steps identical to those described in FIG. 4 are assigned the same reference numerals, and any redundant descriptions are omitted.

[0265] Referring to FIGS. 2, 4, 5, 8, and 11, in one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S220) of generating a reduced input image by lowering the input resolution of the input image based on the calculated reduction ratio.

[0266] In one embodiment of the present disclosure, the step (S220) of generating a reduced input image by lowering the input resolution of the input image may be performed after the step (S210) of generating a masking map. However, the present disclosure is not limited thereto, and the step (S220) of generating a reduced input image by lowering the input resolution of the input image may also be performed after the step (S150) of calculating a reduction ratio.

[0267] In one embodiment of the present disclosure, in the step (S220) of generating a reduced input image by lowering the input resolution of the input image, at least one processor (130) may generate a reduced input image by lowering the input resolution of the input image based on a reduction ratio.

[0268] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S310) of generating a correction image by adding a reduced input image and a frame image based on a masking map. At this time, the masking map used may be a masking map generated based on a reduction ratio, a frame image (620), and an object of interest.

[0269] In one embodiment of the present disclosure, in the step (S310) of generating a correction image, at least one processor (130) can generate a correction image by adding a reduced input image and a frame image based on a masking map.

[0270] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S400) of controlling the image display unit (110) to display a corrected image. The electronic device (100) may display the corrected image through the image display unit (110).

[0271] In one embodiment of the present disclosure, the corrected image displayed in the step (S400) of controlling the image display unit (110) to display a corrected image may be an image in which a reduced input image is displayed in a first area (210) and a frame image is displayed in a second area (220).

[0272] In one embodiment of the present disclosure, at least one processor (130) can control the image display unit (110) to display a corrected image.

[0273] In one embodiment of the present disclosure, the input image (600) may include a plurality of sub-input images, each corresponding to a plurality of frames. The input image (600) may be a video including a plurality of sub-input images.

[0274] In one embodiment of the present disclosure, at least one processor (130) may perform the operations of FIGS. 6A to 8 for each of a plurality of sub-input images to extract an object of interest and calculate a reduction ratio for each of the plurality of sub-input images.

[0275] In one embodiment of the present disclosure, at least one processor (130) may generate a masking map corresponding to each of the plurality of sub-input images by performing the operations of FIGS. 9 to 11 based on a reduction ratio of each of the plurality of sub-input images. At least one processor (130) may generate a plurality of corrected images corresponding to each of the plurality of frames using the masking map.

[0276] In one embodiment of the present disclosure, at least one processor (130) may display and provide a plurality of corrected images to a user via the image display unit (110). Through this, the electronic device (100) may provide an immersive image or a three-dimensional image to the user by considering whether an object of interest is included and the distribution of the object of interest during a plurality of frames.

[0277] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the present disclosure.

[0278] FIG. 12 is a flowchart illustrating an operation for generating a motion compensation image including a moving object according to an embodiment of the present disclosure. FIG. 13 is a flowchart illustrating an operation for extracting a moving object based on the size of an object included in an input image according to an embodiment of the present disclosure. Hereinafter, steps identical to those illustrated in FIG. 3 are assigned the same reference numerals, and redundant descriptions are omitted.

[0279] Referring to FIGS. 2, 3, and 12, in one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S500) of obtaining motion information by estimating the movement of the input image (600, see FIG. 6) based on the input image (600).

[0280] In one embodiment of the present disclosure, the step (S500) of obtaining motion information by estimating the movement of the input image (600) may be performed after the step (S100) of obtaining the input image (600) and the frame image (620).

[0281] In one embodiment of the present disclosure, “motion information” may mean information about the movement of at least one object included in an input image (600). In one embodiment of the present disclosure, when the input image (600) includes a plurality of sub-input images each corresponding to a plurality of frames, the motion information may include information about the direction and distance in which at least one object included in the input image (600) moves during the plurality of frames.

[0282] In one embodiment of the present disclosure, the motion information may include optical flow. In one embodiment of the present disclosure, the optical flow may be a vector component that includes information about the direction and distance an object has moved over a plurality of frames.

[0283] In one embodiment of the present disclosure, the direction of the optical flow may correspond to the direction in which an object included in the input image (600) moves during a plurality of frames. In one embodiment of the present disclosure, the magnitude of the optical flow may correspond to the magnitude of the distance in which an object included in the input image (600) moves during a plurality of frames. In one embodiment of the present disclosure, the optical flow may be estimated based on two plurality of sub-input images corresponding to two adjacent frames.

[0284] In one embodiment of the present disclosure, the motion information may include the velocity of at least one object included in the input image (600).

[0285] In one embodiment of the present disclosure, in the step (S500) of obtaining motion information of an input image (600), the electronic device (100) can obtain motion information by estimating the movement of the input image (600).

[0286] In one embodiment of the present disclosure, at least one processor (130) can estimate the movement of an input image (600) and obtain motion information by executing instructions or program codes of a motion compensation image generation module (123).

[0287] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S600) of extracting a moving object from an input image (600) based on motion information.

[0288] In one embodiment of the present disclosure, in the step (S600) of extracting a moving object from an input image (600), the electronic device (100) can extract a moving object from the input image (600) based on motion information.

[0289] In one embodiment of the present disclosure, at least one processor (130) can extract a moving object from an input image (600) based on motion information by executing instructions or program codes of a motion correction image generation module (123).

[0290] Referring to FIGS. 12 and 13, in one embodiment of the present disclosure, the step (S600) of extracting a moving object from an input image (600) may include the step (S610) of calculating the size of at least one object included in the input image (600). In the step (S610) of calculating the size of at least one object included in the input image (600), the electronic device (100) may detect at least one object included in the input image (600) and calculate the size of the detected object.

[0291] In one embodiment of the present disclosure, in the step (S610) of detecting the size of at least one object included in the input image (600), the electronic device (100) can calculate the size of at least one object included in the input image (600). The electronic device (100) can detect at least one object included in the input image (600) and calculate the size of the detected object.

[0292] In one embodiment of the present disclosure, at least one processor (130) can calculate the size of at least one object included in an input image (600) by executing instructions or program code of a motion compensation image generation module (123). At least one processor (130) can detect at least one object included in an input image (600) and calculate the size of the detected object by executing instructions or program code of a motion compensation image generation module (123).

[0293] In one embodiment of the present disclosure, a "moving object" may mean an object having a size smaller than a preset third value among at least one object included in an input image (600). In one embodiment of the present disclosure, a moving object may mean an object having a size smaller than a preset third value among at least one object included in an input image (600) whose position moves across a plurality of frames.

[0294] In one embodiment of the present disclosure, if an object that is too large among at least one object included in the input image (600) is extracted as a moving object, an awkward experience may be provided to the user when displaying the moving object, which will be described later, in the second area (220). Therefore, the third value may be a preset size so that the user does not feel a sense of incongruity even when displaying the moving object in the second area (220). At this time, in one embodiment of the present disclosure, the third value may be set to have 1 / 10 times the area of ​​the first area (210), but the present disclosure is not limited thereto.

[0295] In one embodiment of the present disclosure, the step (S600) of extracting a moving object from an input image (600) based on motion information may include a step (S620) of extracting an object, among at least one object included in the input image (600), whose size is smaller than a preset third value, as a moving object based on the motion information and the size of the object.

[0296] In one embodiment of the present disclosure, in the step (S600) of extracting a moving object from an input image (600), the electronic device (100) may extract an object, whose size is smaller than a preset third value among at least one object included in the input image (600), as a moving object based on motion information and the size of the object.

[0297] In one embodiment of the present disclosure, a moving object may mean an object having a size smaller than a third value among input images (600) and whose position moves across a plurality of frames.

[0298] In one embodiment of the present disclosure, at least one processor (130) may execute instructions or program codes of a motion correction image generation module (123) to extract, as a moving object, at least one object included in an input image (600) whose size is smaller than a preset third value and whose position changes over a plurality of frames based on motion information and the size of the object.

[0299] In one embodiment of the present disclosure, in the step (S600) of extracting a moving object from an input image (600), a depth map (700, see FIG. 7) including depth information of the input image (600) may be used to extract a usable object from among at least one object included in the input image (600).

[0300] In one embodiment of the present disclosure, the electronic device (100) may extract, as a moving object, at least one object included in an input image (600) that has a size smaller than a preset third value, a position that changes over a plurality of frames, and a depth value greater than a preset second value.

[0301] Referring again to FIG. 12, in one embodiment of the present disclosure, the method of operating the electronic device (100) may include a step (S700) of calculating an expected position of a moving object in the next frame.

[0302] In one embodiment of the present disclosure, the moving object extracted in step S600 may be an object included in the input image (600) in the current frame. The moving object extracted in step S600 may include location information in the current frame.

[0303] In one embodiment of the present disclosure, in the step (S700) of calculating the expected position of a moving object in the next frame, the expected position of the moving object in the next frame can be calculated using the velocity of the moving object in the current frame and the position information of the moving object in the current frame included in the motion information.

[0304] In one embodiment of the present disclosure, in the step (S700) of calculating the expected position of the moving object, the electronic device (100) can calculate the expected position of the moving object in the next frame.

[0305] In one embodiment of the present disclosure, at least one processor (130) can calculate an expected position of a moving object in the next frame by executing instructions or program codes of a motion compensation image generation module (123). At least one processor (130) can calculate an expected position of a moving object in the next frame by using the velocity of the moving object in the current frame included in the motion information and the position information of the moving object in the current frame.

[0306] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S800) of determining whether the expected location in the next frame is located in the second area (220, see FIG. 1).

[0307] In one embodiment of the present disclosure, the electronic device (100) can determine whether the expected position in the next frame is located in the second area (220). At least one processor (130) can determine whether the expected position in the next frame is located in the second area (220) by executing instructions or program codes of the motion compensation image generation module (123).

[0308] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S900) of generating a moving correction image including a moving object located at an expected location in a second area (220) of the correction image based on the correction image and the moving object, when it is determined that the expected location in the next frame is located in the second area (220).

[0309] In one embodiment of the present disclosure, the correction image used in the step (S900) of generating a movement correction image may mean a correction image (1000, see FIG. 10) generated through FIGS. 3 to 11.

[0310] In one embodiment of the present disclosure, when it is determined that the expected position in the next frame is located in the second area (220), the electronic device (100) can generate a moving correction image including a moving object located at the expected position in the second area (220) of the correction image based on the correction image and the moving object.

[0311] In one embodiment of the present disclosure, at least one processor (130) may execute instructions or program codes of a motion correction image generation module (123) to generate a motion correction image including a moving object located at an expected location in a second area (220) of the correction image based on the correction image and the moving object, when it is determined that the expected location in the next frame is located in the second area (220).

[0312] In one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step of controlling an image display unit (110) to display a generated motion correction image. The electronic device (100) may control the image display unit (110) to display the generated motion correction image. At least one processor (130) may control the image display unit (110) to display the motion correction image by executing instructions or program codes of an image display module (124).

[0313] In one embodiment of the present disclosure, the motion compensation image may be an image displayed through the image display unit (110) in the next frame. At least one processor (130) may control the image display unit (110) to display the motion compensation image in the next frame by executing commands or program codes of the image display module (124).

[0314] Through this, if a moving object that is not located in the second area (220) in the current frame is expected to be located in the second area (220) in the next frame based on motion information, etc., the electronic device (100) can display a movement correction image and provide it to the user.

[0315] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step of controlling the image display unit (110) to display a corrected image (1000) generated through FIGS. 3 to 11 when it is determined that the expected position in the next frame is not located in the second area (220).

[0316] The electronic device (100) can display a corrected image (1000) through the image display unit (110) when it is determined that the expected position in the next frame is not located in the second area (220).

[0317] However, the present disclosure is not limited thereto. When the input image (600) includes a plurality of sub-input images spanning a plurality of frames, steps S500 to S800 may be performed for each frame. In one embodiment of the present disclosure, the method of operating the electronic device (100) may, of course, perform steps S500 to S800 again in the next frame when it is determined in step S800 that the expected location in the next frame is not located in the second area (220).

[0318] FIG. 14 is a diagram for explaining an operation of generating a motion compensation image including a moving object according to an embodiment of the present disclosure. FIG. 15 is a diagram for explaining an operation of extracting a moving object from an input image based on motion information of the input image according to an embodiment of the present disclosure. FIG. 16 is a diagram for explaining an operation of calculating an expected position of a moving object in a next frame according to an embodiment of the present disclosure. FIG. 17 is a diagram for explaining a motion compensation image including a moving object according to an embodiment of the present disclosure.

[0319] Hereinafter, the same configuration as that described in FIGS. 5 and 6b is given the same drawing reference numerals, and redundant descriptions are omitted.

[0320] Referring to FIGS. 2, 12, and 14, in one embodiment of the present disclosure, the motion compensation image generation module (123) may include a motion information module (1410). The motion information module (1410) may include instructions or program codes related to an operation or function of estimating the movement of an input image (600) to obtain motion information.

[0321] In one embodiment of the present disclosure, the motion information module (1410) may include an artificial intelligence model. The artificial intelligence model included in the motion information module (1410) may include a machine learning or deep learning model. In one embodiment of the present disclosure, the artificial intelligence model included in the motion information module (1410) may be an artificial intelligence model trained to infer motion information by receiving images, videos, or the like as input.

[0322] In one embodiment of the present disclosure, at least one processor (130) can obtain motion information based on an input image (600) by executing instructions or program code of a motion information module (1410).

[0323] In one embodiment of the present disclosure, the motion correction image generation module (123) may include a moving object extraction module (1420). The moving object extraction module (1420) may include commands or program codes related to an operation or function of extracting a moving object from an input image (600) based on motion information.

[0324] In one embodiment of the present disclosure, the moving object extraction module (1420) may include instructions or program codes relating to an operation or function of detecting at least one object included in an input image (600) and calculating a size of the detected object.

[0325] In one embodiment of the present disclosure, the moving object extraction module (1420) may include instructions or program codes relating to an operation or function of extracting an object, among at least one object included in an input image (600), whose size is smaller than a preset third value, as a moving object based on motion information and the size of the object.

[0326] In one embodiment of the present disclosure, the moving object extraction module (1420) may include instructions or program codes relating to an operation or function of extracting, as a moving object, at least one object included in an input image (600) having a size smaller than a preset third value and whose position moves across multiple frames based on motion information and the size of the object.

[0327] Referring to FIG. 15, in one embodiment of the present disclosure, a moving object (1510) detected in an input image (600) is illustrated in FIG. 15. At this time, the moving object (1510) may be an object having a size smaller than a preset third value. If the third value is set to the size of the area, the area of ​​the moving object (1510) may be smaller than the third value. If the third value is set to the size of the diameter, the size of the diameter of the moving object (1510) may be smaller than the third value. However, the present disclosure is not limited thereto, and it goes without saying that the comparison of the sizes of the moving object (1510) and the third value may be performed in various ways.

[0328] In one embodiment of the present disclosure, motion information (1520) of a moving object (1510) is illustrated in FIG. 15. At this time, the motion information (1520) may mean velocity information of the moving object (1510) in the current frame.

[0329] Referring to FIGS. 14 and 15, in one embodiment of the present disclosure, at least one processor (130) may extract a moving object (1510) from an input image (600) based on motion information by executing instructions or program codes of a moving object extraction module (1420). At least one processor (130) may extract a moving object (1510) from an input image (600) based on motion information and the size of the object by executing instructions or program codes of a moving object extraction module (1420).

[0330] Referring again to FIG. 14, in one embodiment of the present disclosure, the motion compensation image generation module (123) may include an expected position calculation module (1430). The expected position calculation module (1430) may include instructions or program codes relating to operations or functions for calculating an expected position of a moving object (1510) in the next frame based on motion information of the moving object.

[0331] In one embodiment of the present disclosure, the expected position calculation module (1430) may include instructions or program codes relating to an operation or function of calculating an expected position of a moving object (1510) in a next frame based on position information of the moving object (1510) in the current frame and motion information of the moving object (1510) in the current frame.

[0332] Referring to FIG. 16, in one embodiment of the present disclosure, FIG. 16 illustrates a frame image (620) displayed in a second area (220) and a moving object (1510) located at an expected location in the next frame.

[0333] Referring to FIGS. 14 and 16, in one embodiment of the present disclosure, at least one processor (130) may execute instructions or program codes of an expected position calculation module (1430) to calculate an expected position of a moving object (1510) in a next frame based on position information of the moving object (1510) in the current frame and motion information of the moving object (1510) in the current frame.

[0334] In one embodiment of the present disclosure, at least one processor (130) may store the expected position of the moving object (1510) in the next frame calculated by executing instructions or program codes of the expected position calculation module (1430) in the memory (120). The at least one processor (130) may determine whether the expected position is located in the second area (220) and may use the expected position of the moving object (1510) in the next frame stored in the memory (120) when generating a motion correction image (1700, see FIG. 17).

[0335] Referring again to FIG. 14, in one embodiment of the present disclosure, the motion compensation image generation module (123) may include a motion compensation judgment module (1440). The motion compensation judgment module (1440) may include instructions or program codes relating to an operation or function for judging whether the calculated expected position is located in the second area (220).

[0336] In one embodiment of the present disclosure, the movement correction judgment module (1440) may include instructions or program codes relating to an operation or function of judging whether the expected location is located in the second area (220) by using the location information of the calculated expected location and the location information of the second area (220).

[0337] Referring to FIG. 16, in one embodiment of the present disclosure, FIG. 16 illustrates a moving object (1510) located at a calculated expected location, located in a second area (220). The moving object (1510) located at the expected location may overlap with a frame image (620) displayed in the second area (220).

[0338] Referring to FIGS. 14 and 16, in one embodiment of the present disclosure, at least one processor (130) can determine whether the expected location is located in the second area (220) by executing instructions or program codes of the movement correction determination module (1440).

[0339] Again, referring to FIG. 14, in one embodiment of the present disclosure, the movement correction image generation module (123) may include a movement image generation module (1450). The movement image generation module (1450) may include instructions or program codes relating to an operation or function of generating a movement correction image based on a movement object (1510) and a correction image (1000) generated using the correction image generation module (122).

[0340] In one embodiment of the present disclosure, the moving image generation module (1450) may include instructions or program codes relating to an operation or function of generating a moving correction image by adding a correction image (1000) and a moving object (1510) located at an expected location.

[0341] In one embodiment of the present disclosure, the moving image generation module (1450) may include instructions or program codes relating to an operation or function of generating a moving correction image by adding a moving object (1510) located at the expected location to the correction image (1000) when the expected location is determined to be located in the second area (220) through the movement correction judgment module (1440).

[0342] Referring to FIGS. 10, 16, and 17, in one embodiment of the present disclosure, a motion correction image (1700) is illustrated in FIG. 17. The motion correction image (1700) may be an image including the correction image (1000) illustrated in FIG. 10 and a moving object (1510) located at a calculated expected location.

[0343] In one embodiment of the present disclosure, the first image (211) may include an image corresponding to a first area (210) among the input image (600, see FIG. 15). The first image (211) may include an image corresponding to a first area (210) among the objects of interest (213) included in the input image (600). The first image (211) may include an image corresponding to a first area (210) among the moving objects (1510) located at the expected location.

[0344] In one embodiment of the present disclosure, the second image (221) may include an image corresponding to a second region (220) among the objects of interest (213) included in the input image (600). The second image (221) may include an image corresponding to a remaining region of the second region (220) of the frame image (620) excluding the region corresponding to the object of interest (213). The second image (221) may include an image corresponding to the second region (210) among the moving objects (1510) located at the expected location.

[0345] Referring to FIGS. 14 and 17, in one embodiment of the present disclosure, at least one processor (130) may execute instructions or program codes of a moving image generation module (1450) to generate a moving corrected image (1700) including a moving object located at an expected position in a second area (220) of the corrected image based on the corrected image and the moving object, as the calculated expected position is located in the second area (220).

[0346] In one embodiment of the present disclosure, when a moving object (1510) moving during a plurality of frames in an input image (600) is expected to be located in a second area (220) in the next frame and thus be covered by the frame image (620), a motion compensation image (1700) is provided to the user, thereby providing a three-dimensional effect to the user using the electronic device (100).

[0347] Additionally, the user's sense of immersion can be improved by providing the user with a feeling that the moving object (1510) is moving out of the first area (210) or the display area.

[0348] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the present disclosure.

[0349] FIG. 18 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0350] Referring to FIG. 18, in one embodiment of the present disclosure, an electronic device (1800) is illustrated as an electronic device including a display (1810). In one embodiment of the present disclosure, the electronic device (1800) may include a shape such as a television, a mobile device, a smart phone, a laptop computer, or the like.

[0351] In one embodiment of the present disclosure, it goes without saying that the electronic device (1800) illustrated in FIG. 18 may also include a plurality of configurations illustrated in FIG. 2. Furthermore, it goes without saying that the electronic device (1800) illustrated in FIG. 18 may also perform the operations illustrated in FIGS. 2 to 17.

[0352] In one embodiment of the present disclosure, an electronic device (1800) can display a corrected image (1000, see FIG. 10) or a movement corrected image (1700, see FIG. 17) through a display (1810). At least one processor (130, see FIG. 2) can control the display (1810) to display the corrected image (1000) or the movement corrected image (1700). In this case, the display (1810) can correspond to the image display unit (110) of FIG. 2.

[0353] In one embodiment of the present disclosure, the display (1810) may include a display area (1810). The display area (1810) may include a first area (1820) and a second area (1830). In one embodiment of the present disclosure, the second area (1830) may be located above and below the first area (1820). The second area (1830) may be an area adjacent to the first area (1820). However, the present disclosure is not limited thereto, and the second area (1830) may be an area adjacent to the first area (1820) in only one direction, or may be an area adjacent to the first area (1820) in all directions.

[0354] In one embodiment of the present disclosure, a first image (1821) may be displayed in a first region (1820). A second image (1831) may be displayed in a second region (1830). The first image (1821) may include an image corresponding to the first region (1820) among the input image (600, see FIG. 6A). The first image (1821) may include an image corresponding to the first region (1820) among the objects of interest (1822) included in the input image (600).

[0355] In one embodiment of the present disclosure, the second image (1831) may include an image corresponding to a second region (1830) among the objects of interest (1822) included in the input image (600). The second image (1831) may include an image corresponding to the remaining region of the second region (1830) among the frame images (620, see FIG. 6B) excluding the region corresponding to the object of interest (1822).

[0356] However, the present disclosure is not limited thereto, and the electronic device (1800) may also provide a movement correction image (1700) including an object of interest (1510, see FIG. 17) to the user by displaying the image through the display (1810).

[0357] In order to solve the above-described technical problem, in one embodiment of the present disclosure, an electronic device for generating a corrected image is provided. The electronic device may include a memory storing at least one instruction. The electronic device may include at least one processor including processing circuitry. By having at least one processor individually or collectively execute at least one instruction stored in the memory, the electronic device may obtain an input image corresponding to a first region and a second region adjacent to the first region, and a frame image corresponding to the second region. By having at least one processor execute at least one instruction, the electronic device may generate a masking map including information about an object of interest displayed in the second region of the input image based on the input image and the frame image. By having at least one processor execute at least one instruction, the electronic device may generate a corrected image including the input image in the first region and including the object of interest and the frame image in the second region by adding the input image and the frame image based on the masking map.

[0358] In one embodiment of the present disclosure, an electronic device may include an image display unit including a display area. A first area and a second area may be included in the display area. At least one processor may control the image display unit to display a generated corrected image in the display area by executing at least one command.

[0359] In one embodiment of the present disclosure, at least one processor may obtain a depth map including depth information of an input image based on an input image by executing at least one command. At least one processor may obtain a saliency map including saliency information of the input image based on the input image by executing at least one command. At least one processor may extract an object of interest from the input image based on the frame image, the depth map, and the saliency map by executing at least one command.

[0360] In one embodiment of the present disclosure, at least one processor may extract, by executing at least one command, an object corresponding to a second area and having a saliency value greater than a preset first value among at least one object included in an input image based on a saliency map and a frame image, as an object candidate of interest. At least one processor may extract, by executing at least one command, an object candidate of interest as an object of interest based on a depth map, if the object candidate of interest has a depth value greater than a preset second value.

[0361] In one embodiment of the present disclosure, the first region may have a first resolution, and the input image may have an input resolution. At least one processor may calculate a reduction ratio for reducing the input resolution of the input image based on an object of interest by executing at least one command. At least one processor may generate a masking map based on the reduction ratio, the frame image, and the object of interest by executing at least one command.

[0362] In one embodiment of the present disclosure, at least one processor may generate a reduced input image by reducing the input resolution of an input image based on a reduction ratio by executing at least one command. At least one processor may generate a corrected image by adding the reduced input image and the frame image based on a masking map by executing at least one command.

[0363] In one embodiment of the present disclosure, the input resolution may be greater than the first resolution. The resolution of the reduced input image may be equal to or greater than the first resolution.

[0364] In one embodiment of the present disclosure, a masking map can be generated to mask each of the reduced input image and the frame image to generate a correction image including the reduced input image in a first region and the frame image in a second region.

[0365] In one embodiment of the present disclosure, at least one processor can obtain motion information by estimating a motion of an input image based on the input image by executing at least one command. At least one processor can extract a moving object from the input image based on the obtained motion information by executing at least one command. At least one processor can calculate an expected position of the moving object in a next frame by executing at least one command. At least one processor can generate a moving corrected image including a moving object located at an expected position in a second region of the corrected image based on the corrected image and the moving object, as the calculated expected position is located in a second region by executing at least one command.

[0366] In one embodiment of the present disclosure, at least one processor may calculate the size of at least one object included in an input image by executing at least one command. At least one processor may extract, as a moving object, an object whose size is smaller than a preset third value among the at least one object based on motion information and the size of the at least one object by executing at least one command.

[0367] In order to solve the above-described technical problem, one embodiment of the present disclosure provides an operating method of an electronic device for generating a corrected image. The operating method of the electronic device may include a step of acquiring an input image corresponding to a first region and a second region adjacent to the first region and a frame image corresponding to the second region. The operating method of the electronic device may include a step of generating a masking map including information on an object of interest displayed in the second region of the input image based on the input image and the frame image. The operating method of the electronic device may include a step of generating a corrected image including the input image in the first region and including the object of interest and the frame image in the second region by adding the input image and the frame image based on the masking map.

[0368] In one embodiment of the present disclosure, a method of operating an electronic device may include a step of displaying a generated correction image in a display area through an image display unit including a display area. A first area and a second area may be included in the display area.

[0369] In one embodiment of the present disclosure, a method of operating an electronic device may include a step of obtaining a depth map including depth information of an input image based on an input image. The method of operating an electronic device may include a step of obtaining a saliency map including saliency information of the input image based on the input image. The method of operating an electronic device may include a step of extracting an object of interest from an input image based on a frame image, a depth map, and a saliency map.

[0370] In one embodiment of the present disclosure, the step of extracting an object of interest may include a step of extracting, based on a saliency map and a frame image, an object, among at least one object included in an input image, that has a saliency value greater than a preset first value and corresponds to a second area, as an object of interest candidate. The step of extracting an object of interest may include a step of extracting, based on a depth map, an object of interest candidate as an object of interest if the object of interest candidate has a depth value greater than a preset second value.

[0371] In one embodiment of the present disclosure, a first region may have a first resolution, and an input image may have an input resolution. The method of operating an electronic device may include calculating a reduction ratio that reduces the input resolution of the input image based on an object of interest. In the step of generating a masking map, the masking map may be generated based on the reduction ratio, the frame image, and the object of interest.

[0372] In one embodiment of the present disclosure, a method of operating an electronic device may include a step of generating a reduced input image by reducing the input resolution of an input image based on a reduction ratio. In the step of generating a corrected image, a reduced input image and a frame image may be added based on a masking map to generate the corrected image.

[0373] In one embodiment of the present disclosure, in the step of generating a masking map, the masking map may be generated to mask each of the reduced input image and the frame image to generate a correction image including a reduced input image in a first region and a frame image in a second region.

[0374] In one embodiment of the present disclosure, a method for operating an electronic device may include a step of obtaining motion information by estimating a motion of an input image based on an input image. The method for operating the electronic device may include a step of extracting a moving object from the input image based on the obtained motion information. The method for operating the electronic device may include a step of calculating an expected position of the moving object in a next frame. The method for operating the electronic device may include a step of generating a moving compensation image including a moving object located at an expected position in a second area of ​​the compensation image based on the compensation image and the moving object, when the calculated expected position is located in a second area.

[0375] In one embodiment of the present disclosure, the step of extracting a moving object may include a step of calculating a size of at least one object included in an input image. The step of extracting a moving object may include a step of extracting an object, among at least one object, whose size is smaller than a preset third value, as a moving object, based on the motion information and the size of the at least one object.

[0376] In order to solve the above-described technical problem, a computer-readable recording medium having recorded thereon a program for performing at least one method of an embodiment of an operating method of an electronic device disclosed in the present disclosure on a computer can be provided.

[0377] The program executed by the electronic device described in this disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0378] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0379] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The storage media may be readable by a computer, stored in a memory, and executed by a processor.

[0380] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage media and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0381] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0382] A computer program product may include a software program and a computer-readable storage medium storing the software program. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by an electronic device manufacturer or through an electronic marketplace (e.g., the Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily generated. In this case, the storage medium may be a server of the electronic device manufacturer, a server of the electronic marketplace, or a storage medium of an intermediary server that temporarily stores the software program.

[0383] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0384] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the present disclosure.

Claims

1. In an electronic device (100) that generates a correction image, A memory (120) storing at least one instruction; and At least one processor comprising a processing circuit, The electronic device, by causing the at least one processor to individually or collectively execute the at least one instruction stored in the memory, Obtain an input image corresponding to a first region (210) and a second region (220) adjacent to the first region (210) and a frame image corresponding to the second region (220), Based on the input image and the frame image, a masking map is generated that includes information about an object of interest displayed in the second area (220) of the input image, An electronic device (100) that generates a correction image including the input image in the first area (210) and including the object of interest and the frame image in the second area (220) by adding the input image and the frame image based on the masking map.

2. In paragraph 1, The above electronic device (100) further includes a video display unit (110) including a display area, The first area (210) and the second area (220) are included in the display area, The at least one processor (130) executes the at least one instruction, An electronic device (100) that controls the image display unit (110) to display the generated correction image in the display area.

3. In either of paragraphs 1 or 2, The at least one processor (130) executes the at least one instruction, Based on the input image, a depth map including depth information of the input image is obtained, Based on the input image, a saliency map including saliency information of the input image is obtained, An electronic device (100) that extracts the object of interest from the input image based on the frame image, the depth map, and the saliency map.

4. In paragraph 3, The at least one processor (130) executes the at least one instruction, Based on the above saliency map and the frame image, an object corresponding to the second area, having a saliency value greater than a preset first value among at least one object included in the input image, is extracted as a candidate object of interest, An electronic device (100) that extracts the object of interest candidate as the object of interest based on the depth map, when the object of interest candidate has a depth value greater than a preset second value.

5. In either of the third or fourth paragraphs, The first region has a first resolution, and the input image has an input resolution, The at least one processor (130) executes the at least one instruction, Based on the object of interest, a reduction ratio is calculated to lower the input resolution of the input image, An electronic device (100) that generates the masking map based on the reduction ratio, the frame image, and the object of interest.

6. In paragraph 5, The at least one processor (130) executes the at least one instruction, Based on the above reduction ratio, the input resolution of the input image is lowered to generate a reduced input image, An electronic device (100) that generates the corrected image by adding the reduced input image and the frame image based on the masking map.

7. In paragraph 6, The above input resolution is greater than the first resolution, The resolution of the above reduced input image is an electronic device (100) equal to or greater than the first resolution.

8. In either of paragraphs 6 or 7, The above masking map is, An electronic device (100) generated to mask each of the reduced input image and the frame image to generate the correction image including the reduced input image in the first region and the frame image in the second region.

9. In any one of the clauses 1 to 8, The at least one processor (130) executes the at least one instruction, Based on the input image, motion information is obtained by estimating the movement of the input image, Based on the acquired motion information, a moving object is extracted from the input image, Compute the expected position of the above moving object in the next frame, An electronic device (100) that generates a moving correction image including the moving object located at the expected position in the second area of ​​the correction image based on the correction image and the moving object, as the calculated expected position is located in the second area.

10. In paragraph 9, The at least one processor (130) executes the at least one instruction, Compute the size of at least one object included in the input image, An electronic device (100) that extracts an object among the at least one object whose size is smaller than a preset third value as a moving object based on the motion information and the size of the at least one object.

11. In the operating method of an electronic device (100) that generates a correction image, A step (S100) of obtaining an input image corresponding to a first region (210) and a second region (220) adjacent to the first region (210) and a frame image corresponding to the second region (220); A step (S200) of generating a masking map including information about an object of interest displayed in the second area (220) of the input image based on the input image and the frame image; and An operating method of an electronic device (100), comprising a step (S300) of adding the input image and the frame image based on the masking map to generate a correction image including the input image in the first area (210) and including the object of interest and the frame image in the second area (220).

12. In paragraph 11, The operating method of the above electronic device (100) is: A step of displaying the generated correction image in the display area through an image display unit (110) including a display area, The above first area (210) and the above second area (220) are operating methods of an electronic device (100) included in the above display area.

13. In either of paragraphs 11 or 12, The operating method of the above electronic device (100) is: A step of obtaining a depth map including depth information of the input image based on the input image; A step of obtaining a saliency map including saliency information of the input image based on the input image; and An operating method of an electronic device (100), comprising a step of extracting the object of interest from the input image based on the frame image, the depth map, and the saliency map.

14. In paragraph 13, The step of extracting the object of interest is as follows: A step of extracting an object corresponding to the second area (220) and having a saliency value greater than a preset first value among at least one object included in the input image based on the saliency map and the frame image as an object of interest candidate; and An operating method of an electronic device (100), comprising a step of extracting the object of interest candidate as the object of interest based on the depth map, when the object of interest candidate has a depth value greater than a preset second value.

15. A computer-readable recording medium having recorded thereon a program for performing the method of operation described in any one of clauses 11 to 14 on a computer.

Images