Motion-based image enhancement method and electronic equipment

Abstract

Accordingly, embodiments herein provide a method for motion-based image enhancement by an electronic device (100). The method includes receiving a plurality of image frames including a subject to perform an action. The method includes determining a plurality of keypoints associated with a subject of the plurality of image frames, and detecting an action performed by the subject by using the plurality of estimated keypoints. The method includes determining a motion characteristic associated with the plurality of estimated keypoints. The method includes identifying one or more regions to be enhanced from a plurality of regions in the plurality of image frames based on motion characteristics determined with the plurality of estimated keypoints and the detected action. The method includes generating an enhanced image comprising one or more enhanced regions compared to one or more regions of the plurality of regions of the received image frame.

Description

Technical Field

[0001] The present invention relates to an electronic device, and more particularly to a motion-based image enhancement method and an electronic device. This application is based on and claims priority to Indian Provisional Application No. 202241048869 filed on August 26, 2022, the disclosure of which is incorporated herein by reference. Background Art

[0002] Image enhancement has attracted a lot of attention recently, especially in the smartphone consumer market. Leading smartphone vendors have recently made remarkable progress in image enhancement areas such as high dynamic range (HDR) and low-light denoising. However, image capture of moving subjects (e.g., humans) often results in artifacts such as blur (1) and, in the absence of good lighting conditions, often results in artifacts such as low-light noise (2). Figure 1 shown.

[0003] Image enhancement via artifact reduction is crucial for both aesthetics and downstream computer vision tasks. Multi-frame algorithms, such as multi-frame noise removal (MFNR) and HDR, are commonly used in image enhancement methods. To avoid artifacts such as blur (1) / low-light noise (2) / ghosting (3) during image processing, multi-frame algorithms often compute motion maps. Motion maps are often computed using photometric difference-based methods or human keypoint-based methods. However, these methods often result in false positive motion in the presence of blur (1) / low-light noise (2) / ghosting (3). As a result, the output image has more noise (2) or lower dynamic range (4).

[0004] The photometric difference based method uses photometric alignment of each pixel (optional for HDR) and then uses photometric difference. Motion maps are generated in the presence of noise, which is prone to errors. As a result, large areas of false positive motion are generated. Large areas of false positive motion lead to areas with less mixing, which further leads to dynamic range loss or noise increase, such as Figure 2a and Figure 2b shown.

[0005] Keypoint-based methods estimate human pose by computing keypoints of the human body, which are then analyzed to detect motion. In the presence of high noise / blur, the estimated keypoints are erroneous, which further leads to classifying static areas as moving (false positive motion). Subsequently, this results in lower dynamic range (4) or higher noise (2).

[0006] It would therefore be desirable to address the above-mentioned shortcomings or other deficiencies, or at least provide a useful alternative to motion-based image enhancement. Summary of the invention

Technical issues

[0007] The main purpose of the embodiments herein is to intelligently generate an image by identifying one or more regions with image artifacts (e.g., blurred regions, regions with a lot of motion, etc.) to be enhanced from multiple regions in a received image frame using multiple estimated key points based on motion characteristics associated with the multiple estimated key points (the key points are associated with a subject of the received image frame) and the actions performed by the subject. Therefore, when compared to one or more regions of the multiple regions of the received image frame, the enhanced image includes one or more enhanced regions without image artifacts, which enhances the user experience.

[0008] Another object of the embodiments herein is to determine an optimal motion map from a plurality of optimal image frames by predicting a local motion region (e.g., a user's legs) in a received image frame based on a detected motion (e.g., a user's jump) and a plurality of estimated key points, wherein the plurality of optimal image frames include a peak motion of the detected motion (e.g., a user jumping in the air). An enhanced image (e.g., an HDR image, a denoised image, a blur-corrected image, a reflection-removed image, etc.) is generated using the optimal motion map. [Solution to the problem]

[0009] Accordingly, embodiments herein provide a method for motion-based image enhancement. The method includes: receiving, by an electronic device, an image frame including a subject performing an action. In addition, the method includes: determining, by an electronic device, a plurality of key points associated with the subject of the received image frame. In addition, the method includes: detecting, by an electronic device, an action performed by the subject using a plurality of estimated key points. In addition, the method includes: determining, by an electronic device, motion characteristics associated with a plurality of estimated key points. In addition, the method includes: identifying, by an electronic device, one or more regions to be enhanced from a plurality of regions in the received image frame based on the motion characteristics determined using a plurality of estimated key points and the detected action. In addition, the method includes: generating, by an electronic device, an enhanced image including one or more enhanced regions compared to one or more regions in a plurality of regions of the received image frame. In addition, the method includes: storing, by an electronic device, an enhanced image including one or more enhanced regions in a plurality of regions.

[0010] In an embodiment, wherein, identifying, by the electronic device, one or more regions to be enhanced from a plurality of regions in a received image frame based on motion characteristics determined using a plurality of estimated key points and a detected motion comprises: determining, by the electronic device, a plurality of optimal image frames from the received image frame based on the detected motion, wherein the plurality of optimal image frames comprise peak motion of the detected motion. In addition, the method comprises: predicting, by the electronic device, a local motion region in the received image frame based on the detected motion. In addition, the method comprises: determining, by the electronic device, a best motion map from a plurality of optimal image frames based on the predicted local motion region and a plurality of estimated key points. In addition, the method comprises: performing, by the electronic device, localization of space-time artifacts for a plurality of optimal image frames based on the determined best motion map, the determined motion characteristics associated with the plurality of estimated key points, and the detected motion. In addition, the method comprises: identifying, by the electronic device, one or more regions to be enhanced from a plurality of regions in the received image frame based on localization of space-time artifacts, wherein the one or more regions comprise image artifacts, and the image artifacts comprise blurred regions, noise regions, dark regions, and motion regions.

[0011] In an embodiment, wherein determining, by an electronic device, the best motion map among multiple best image frames based on a predicted local motion region and multiple estimated key points includes: generating, by an electronic device, an initial motion map of multiple best image frames based on an image restoration mechanism. In addition, the method includes: generating, by an electronic device, a digital skeleton by connecting multiple estimated key points. In addition, the method includes: retrieving, by an electronic device, the motion probabilities of the key points and bones of the generated digital skeleton from a predefined dictionary of a database of the electronic device for the detected motion. For each action, the motion probability for each key point is calculated. For example, for jumping, since the limbs move faster than the body, their probability will be higher. In addition, the method includes: updating, by an electronic device, the generated digital skeleton based on the retrieved motion probabilities of the key points and bones. In addition, the method includes: determining, by an electronic device, the best motion map based on the predicted local motion region, the generated initial motion map, and the updated digital skeleton.

[0012] In an embodiment, wherein performing the localization of space-time artifacts for multiple best image frames by the electronic device includes: determining the noise standard deviation of the multiple best image frames by the electronic device using a classical learning mechanism and a deep learning mechanism to identify one or more regions to be enhanced from multiple regions in the received image frames. The standard deviation of the image in each region can be set as a noise estimate. In addition, the method includes: determining at least one static region from multiple regions in at least one received image frame by the electronic device. In addition, the method includes: determining at least one change key point in the at least one static region by the electronic device. In addition, the method includes: determining the motion parameters of each key point in the predicted local motion region by the electronic device based on the post-estimation error and the multiple estimated key points, wherein the motion parameters include displacement, velocity and acceleration. In addition, the method includes: determining the motion between subsequent frames of the multiple best image frames by the electronic device using the determined motion parameters. In addition, the method includes: determining the size of the blur kernel by the electronic device based on the determined motion to identify one or more regions to be enhanced from multiple regions in the received image frames.

[0013] In an embodiment, the enhanced image generated by applying the image enhancement mechanism includes a high dynamic range (HDR) image, a denoised image, a blur-corrected image, and a reflection-removed image.

[0014] In an embodiment, generating an HDR image includes: clustering, by an electronic device, one or more regions identified from a plurality of regions in a received image frame based on determined motion characteristics associated with a plurality of estimated key points and detected motion, and clustering the received image frames into a plurality of frame groups, wherein the plurality of frame groups include a plurality of frames with a lowest displacement, a plurality of frames with a medium displacement, and a plurality of frames with a highest displacement. In addition, the method includes: generating, by an electronic device, a high-exposure frame from a plurality of frames with a lowest displacement. In addition, the method includes: generating, by an electronic device, a medium-exposure frame from a plurality of frames with a medium displacement. In addition, the method includes: generating, by an electronic device, a low-exposure frame from a plurality of frames with a highest displacement. In addition, the method includes: mixing, by an electronic device, the generated high-exposure frame, the generated medium-exposure frame, and the generated low-exposure frame to generate an HDR image.

[0015] In an embodiment, a denoised image is generated by utilizing an optimal motion map.

[0016] In an embodiment, generating the blur-corrected image includes: determining, by the electronic device, whether the motion parameter exceeds a predefined threshold. Blur correction is only required if the motion parameter is above the predefined threshold. In addition, the method includes: applying, by the electronic device, blur correction to an area around a key point where the measured motion parameter exceeds the predefined threshold. In addition, the method includes: generating the blur-corrected image by applying, by the electronic device, blur correction to an area around a key point where the measured motion parameter exceeds the predefined threshold.

[0017] In an embodiment, wherein generating the reflection removed image comprises: determining, by an electronic device, a correlation between a motion characteristic determined using a plurality of estimated key points of a first subject and a motion characteristic determined using a plurality of estimated key points of a second subject. In addition, the method comprises: classifying, by the electronic device, highly correlated key points of the second subject as reflection key points. In addition, the method comprises: generating, by the electronic device, a reflection map using the classified highly correlated key points. In addition, the method comprises: generating, by the electronic device, a reflection removed image using the generated reflection map.

[0018] In an embodiment, identifying, by an electronic device, one or more regions to be enhanced from multiple regions in at least one received image frame based on at least one motion characteristic determined using multiple estimated key points and at least one detected action includes: comparing, by the electronic device (100), calculated values ​​of one or more motion characteristics associated with each of the multiple estimated key points with expected values; determining, by the electronic device (100), a deviation of the calculated value of each of the multiple estimated key points from the expected value; and determining, by the electronic device (100), a first group of key points among the multiple estimated key points having a deviation greater than a threshold.

[0019] Accordingly, embodiments herein provide an electronic device for motion-based image enhancement. The electronic device includes an image processing controller coupled to a processor and a memory. The image processing controller receives an image frame including a subject performing an action. The image processing controller determines a plurality of key points associated with the subject of the received image frame. The image processing controller detects the action performed by the subject using a plurality of estimated key points. The image processing controller determines motion characteristics associated with the plurality of estimated key points. The image processing controller identifies one or more regions to be enhanced from a plurality of regions in the received image frame based on the motion characteristics determined using the plurality of estimated key points and the detected action. The image processing controller generates an enhanced image including one or more enhanced regions compared to one or more regions in the plurality of regions of the received image frame. The image processing controller stores the enhanced image including the one or more enhanced regions in the plurality of regions.

[0020] These and other aspects of the embodiments herein will be better understood and appreciated when considered in conjunction with the following description and accompanying drawings. However, it should be understood that although the following description indicates a preferred embodiment and many of its specific details, the following description is given by way of illustration and not limitation. Various changes and modifications may be made within the scope of the embodiments herein, and the embodiments herein include all such modifications. [Beneficial effects of the invention]

[0021] In one embodiment, the electronic device may obtain an enhanced image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention is illustrated in the accompanying drawings, and like reference numerals indicate corresponding parts in the various drawings throughout the drawings. The embodiments herein will be better understood from the following description with reference to the accompanying drawings, in which:

[0023] Figure 1 The problems caused by the presence of a moving subject in a conventional image enhancement mechanism according to the prior art are shown;

[0024] Figure 2a and Figure 2b is an example scenario illustrating problems in existing HDR image enhancement mechanisms according to the prior art;

[0025] Figure 3 A block diagram of an electronic device for motion-based image enhancement according to embodiments disclosed herein is shown;

[0026] Figure 4 is a flow chart illustrating a method for motion-based image enhancement according to embodiments disclosed herein;

[0027] Figure 5 is a system flow chart illustrating a method for motion-based image enhancement according to embodiments disclosed herein;

[0028] Figure 6 Various operations associated with a motion-based artifact region locator for motion-based image enhancement according to embodiments disclosed herein are shown;

[0029] Figure 7 Various operations for motion-based image enhancement associated with a peak action identifier and a local motion predictor according to embodiments disclosed herein are shown;

[0030] Figure 8 Various operations for motion-based image enhancement associated with a region identifier for a motion locator are shown in accordance with embodiments disclosed herein;

[0031] Fig. 9 Various operations associated with a spatio-temporal artifact locator for motion-based image enhancement according to embodiments disclosed herein are shown;

[0032] Fig.10 Various operations associated with an image enhancer for generating an HDR image according to embodiments disclosed herein are shown;

[0033] Fig.11 is a flow chart illustrating a method for generating a blur-corrected image using an image enhancer according to embodiments disclosed herein;

[0034] Fig.12 illustrates various operations associated with an image enhancer for generating a denoised image according to embodiments disclosed herein; and

[0035] Fig.13a and Fig.13b is an example flow chart illustrating a method for motion-based image enhancement according to embodiments disclosed herein. DETAILED DESCRIPTION

[0036] The embodiments herein and their various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. The description of known components and processing techniques is omitted so as not to unnecessarily confuse the embodiments herein. In addition, since some embodiments can be combined with one or more other embodiments to form new embodiments, the various embodiments described herein are not necessarily mutually exclusive. Unless otherwise indicated, the term "or" used herein refers to non-exclusive or. The examples used herein are intended only to facilitate understanding of the manner in which the embodiments herein are practiced, and further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

[0037] As is common in the art, embodiments may be described and illustrated around blocks that perform one or more functions described. These blocks, which may be referred to herein as managers, units, modules, hardware components, etc., are physically implemented by analog and / or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and may optionally be driven by firmware. For example, the circuit may be implemented in one or more semiconductor chips, or on a substrate support such as a printed circuit board. The circuits that make up the blocks may be implemented by dedicated hardware or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware for performing some functions of the block and a processor for performing other functions of the block. Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of the present disclosure, the blocks of the embodiment may be physically combined into more complex blocks.

[0038] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, the present disclosure should be interpreted as extending to any changes, equivalents and substitutes other than those specifically set forth in the accompanying drawings. Although the terms "first", "second" etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0039] Figure 2a and Figure 2b is an example scenario illustrating problems in existing HDR image enhancement mechanisms according to the prior art.

[0040] An existing HDR image enhancement mechanism receives a plurality of image frames (5 and 6) including a subject (e.g., a human) performing an action (e.g., jumping). The existing HDR image enhancement mechanism then performs exposure alignment (7 and 8) on the received plurality of image frames (5 and 6). The existing HDR image enhancement mechanism then determines the photometric difference of the exposure aligned (7 and 8) frames. The existing HDR image enhancement mechanism then generates an initial motion map (10). The generated initial motion map (10) is prone to errors. As a result, large areas of false positive motion (11) are generated. Large areas of false positives result in reduced mixing of these areas, resulting in dynamic range loss / increased noise / dark artifacts, which negatively impact the user experience. To address these issues, a novel method is proposed for image enhancement that uses action recognition to locate motion areas, which is resistant to artifacts such as noise and blur.

[0041] Accordingly, embodiments herein provide a method for motion-based image enhancement. The method includes: receiving, by an electronic device, an image frame including a subject performing an action. In addition, the method includes: determining, by an electronic device, a plurality of key points associated with the subject of the received image frame. In addition, the method includes: detecting, by an electronic device, an action performed by the subject using a plurality of estimated key points. In addition, the method includes: determining, by an electronic device, motion characteristics associated with a plurality of estimated key points. In addition, the method includes: identifying, by an electronic device, one or more regions to be enhanced from a plurality of regions in the received image frame based on the motion characteristics determined using a plurality of estimated key points and the detected action. In addition, the method includes: generating, by an electronic device, an enhanced image including one or more enhanced regions compared to one or more regions in a plurality of regions of the received image frame. In addition, the method includes: storing, by an electronic device, an enhanced image including one or more enhanced regions in a plurality of regions.

[0042] Accordingly, embodiments herein provide an electronic device for motion-based image enhancement. The electronic device includes an image processing controller coupled to a processor and a memory. The image processing controller receives an image frame including a subject performing an action. The image processing controller determines a plurality of key points associated with the subject of the received image frame. The image processing controller detects the action performed by the subject using a plurality of estimated key points. The image processing controller determines motion characteristics associated with the plurality of estimated key points. The image processing controller identifies one or more regions to be enhanced from a plurality of regions in the received image frame based on the motion characteristics determined using the plurality of estimated key points and the detected action. The image processing controller generates an enhanced image including one or more enhanced regions compared to one or more regions in the plurality of regions of the received image frame. The image processing controller stores an enhanced image including one or more enhanced regions in the plurality of regions.

[0043] Unlike existing methods and systems, the proposed method enables an electronic device to intelligently generate an image by identifying one or more regions with image artifacts (e.g., blurred regions, regions with a lot of motion, etc.) to be enhanced from a plurality of regions in a received image frame using a plurality of estimated key points, based on motion characteristics associated with the plurality of estimated key points (the key points are associated with a subject of the received image frame) and an action performed by the subject. As a result, the enhanced image includes one or more enhanced regions without image artifacts when compared to one or more regions of the plurality of regions of the received image frame, which enhances the user experience.

[0044] Unlike existing methods and systems, the proposed method enables an electronic device to determine an optimal motion map among multiple optimal image frames by predicting a local motion region (e.g., a user's legs) in a received image frame based on a detected action (e.g., a user's jump) and multiple estimated key points, wherein the multiple optimal image frames include a peak action of the detected action (e.g., a user jumping in the air). The optimal motion map is used to generate an enhanced image (e.g., an HDR image, a denoised image, a blur-corrected image, a reflection-removed image, etc.).

[0045] Referring now to the drawings, and more particularly to Figure 3 to Figure 1 3, wherein like reference numerals consistently denote corresponding features shown in the preferred embodiment throughout the accompanying drawings.

[0046] Figure 3 A block diagram of an electronic device (100) for motion-based image enhancement according to an embodiment disclosed herein is shown. The electronic device (100) may be, but is not limited to, for example, a smart phone, a laptop, a desktop, a smart watch, a smart TV, an augmented reality device (AR device), a virtual reality device (VR device), an Internet of Things (IoT) device, etc.

[0047] In an embodiment, an electronic device (100) includes a memory (110), a processor (120), a communicator (130), a display (140), an image processing controller (150), and a camera (160).

[0048] In an embodiment, the memory (110) stores a plurality of image frames having a subject, a plurality of key points associated with the subject in a key point motion storage (111) of the memory (110), information associated with skeletal motion in a skeletal motion storage (112) of the memory (110), actions performed by the subject, a plurality of optimal image frames, optimal motion maps in a plurality of optimal image frames, one or more regions having image artifacts, and an enhanced image having one or more enhanced regions in a plurality of regions. The memory (110) stores instructions executed by the processor (120). The memory (110) may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of an electrically programmable memory (EPROM) or an electrically erasable programmable memory (EEPROM). In addition, in some examples, the memory (110) may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not implemented in a carrier wave or a propagating signal. However, the term "non-transitory" should not be interpreted as the memory (110) being non-removable. In some examples, the memory (110) can be configured to store a larger amount of information than the storage. In some examples, a non-transitory storage medium can store data that can change over time (e.g., in random access memory (RAM) or cache). The memory (110) can be an internal storage unit, or can be an external storage unit of the electronic device (100), cloud storage, or any other type of external storage device.

[0049] The processor (120) communicates with the memory (110), the communicator (130), the display (140), the image processing controller (150), and the camera (160). The camera (160) includes a main camera (160a) and auxiliary cameras (160b-160n) for capturing image frames. The processor (120) is configured to execute instructions stored in the memory (110) and perform various processes. The processor (120) may include one or more processors, which may be a general-purpose processor (e.g., a central processing unit (CPU), an application processor (AP), etc.), a pure graphics processing unit (e.g., a graphics processing unit (GPU)), a visual processing unit (VPU), and / or an artificial intelligence (AI) dedicated processor (e.g., a neural processing unit (NPU)).

[0050] The communicator (130) is configured to communicate internally between internal hardware components and to communicate with external devices (eg, servers) via one or more networks (eg, radio technologies). The communicator (130) includes electronic circuitry specific to a standard that enables wired or wireless communication.

[0051] The display (140) may be a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), or other types of displays that may also accept user input. Touch, swipe, drag, gesture, voice command, and other user input are examples of user input.

[0052] The image processing controller (150) is implemented by processing circuits (e.g., logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may optionally be driven by firmware. For example, the circuits may be implemented in one or more semiconductor chips, or on a substrate support such as a printed circuit board.

[0053] In an embodiment, an image processing controller (150) includes a pose estimator (151), a motion recognizer (152), a motion-based artifact region locator (153), an image enhancer (154), and an artificial intelligence (AI) engine (155).

[0054] The pose estimator (151) receives an image frame including a subject (e.g., a human, a plant, an animal, etc.) performing an action (e.g., jumping). The pose estimator (151) determines a plurality of key points associated with the subject of the received image frame. The action recognizer (152) detects the action performed by the subject using the plurality of estimated key points. The action recognizer (152) determines a motion characteristic. The motion characteristic associated with the plurality of estimated key points may be, for example, but is not limited to, velocity, acceleration, displacement.

[0055] The motion-based artifact region locator (153) identifies one or more regions to be enhanced from a plurality of regions in a received image frame based on motion characteristics determined using a plurality of estimated key points and a detected motion. The motion-based artifact region locator (153) determines a plurality of optimal image frames from the received image frames based on the detected motion. The plurality of optimal image frames are image frames including peak motion of the detected motion. The motion-based artifact region locator (153) predicts a local motion region in the received image frame based on the detected motion. The motion-based artifact region locator (153) determines an optimal motion map among the plurality of optimal image frames based on the predicted local motion region and the plurality of estimated key points. The motion-based artifact region locator (153) performs localization of space-time artifacts for the plurality of optimal image frames based on the determined optimal motion map, the determined motion characteristics associated with the plurality of estimated key points, and the detected motion. Localization of space-time artifacts refers to localizing space-time artifacts at specific locations within the optimal motion map. A motion-based artifact region locator (153) identifies one or more regions to be enhanced from a plurality of regions in a received image frame based on the location of spatio-temporal artifacts, wherein the one or more regions include image artifacts, and the image artifacts include, for example, blurred regions, noisy regions, dark regions, and motion regions.

[0056] The motion-based artifact region locator (153) generates an initial motion map of a plurality of optimal image frames based on an image restoration mechanism. The motion-based artifact region locator (153) generates a digital skeleton by connecting a plurality of estimated key points. The motion-based artifact region locator (153) retrieves the motion probabilities of the key points and bones of the generated digital skeleton from a predefined dictionary of a database (e.g., a key point motion repository (111), a skeleton motion repository (112), etc.) of the electronic device (100) for the detected motion. The motion-based artifact region locator (153) updates the generated digital skeleton based on the retrieved motion probabilities of the key points and bones. The motion-based artifact region locator (153) determines the optimal motion map based on the predicted local motion region, the generated initial motion map, and the updated digital skeleton.

[0057] The motion-based artifact region locator (153) uses a classical learning mechanism and a deep learning mechanism to determine the noise standard deviation of multiple best image frames to identify one or more regions to be enhanced from multiple regions in the received image frames. The motion-based artifact region locator (153) determines at least one static region from multiple regions in at least one received image frame. In addition, the motion-based artifact region locator (153) determines at least one changing key point in at least one static region. In motion such as standing still, the key points should be static. However, due to errors in the initial pose estimation, the estimated key points will change. However, since these key points should be static, the expected changes / errors in the key point estimation (pose estimation) can be determined. The errors in the pose estimation can be modeled as a Gaussian distribution. The mean and variance of the model can be estimated using the key point data in the static region. The motion-based artifact region locator (153) determines motion parameters of each key point in the predicted local motion region based on the post-estimation error and the multiple estimated key points, wherein the motion parameters include displacement, velocity, and acceleration. The motion-based artifact region locator (153) uses the determined motion parameters to determine motion between subsequent frames of the plurality of best image frames. The motion-based artifact region locator (153) determines a size of a blur kernel based on the determined motion to identify one or more regions to be enhanced from the plurality of regions in the received image frame.

[0058] The image enhancer (154) generates an enhanced image including one or more enhanced regions compared to one or more regions of the plurality of regions of the received image frame. The image enhancer (154) generates the enhanced image by applying an image enhancement mechanism, the enhanced image including, for example, a high dynamic range (HDR) image, a denoised image, a blur-corrected image, and a reflection-removed image.

[0059] The image enhancer (154) clusters one or more regions identified from a plurality of regions in the received image frames based on the determined motion characteristics associated with the plurality of estimated key points and the detected motion, and clusters the received image frames into a plurality of frame groups, wherein the plurality of frame groups include a plurality of frames with the lowest displacement, a plurality of frames with the middle displacement, and a plurality of frames with the highest displacement. The image enhancer (154) generates a high exposure frame from the plurality of frames with the lowest displacement. The image enhancer (154) generates a middle exposure frame from the plurality of frames with the middle displacement. The image enhancer (154) generates a low exposure frame from the plurality of frames with the highest displacement. The image enhancer (154) mixes the generated high exposure frame, the generated middle exposure frame, and the generated low exposure frame to generate an HDR image.

[0060] The image enhancer (154) generates a denoised image by utilizing the optimal motion map.

[0061] The image enhancer (154) determines whether the motion parameter exceeds a predefined threshold. The image enhancer (154) applies blur correction to the area around the key point where the measured motion parameter exceeds the predefined threshold. The image enhancer (154) generates a blur-corrected image by applying blur correction to the area around the key point where the measured motion parameter exceeds the predefined threshold.

[0062] The image enhancer (154) determines a correlation between a motion characteristic determined using a plurality of estimated keypoints of the first subject and a motion characteristic determined using a plurality of estimated keypoints of the second subject. The image enhancer (154) classifies highly correlated keypoints of the second subject as reflection keypoints. The image enhancer (154) generates a reflection map using the classified highly correlated keypoints. The image enhancer (154) generates a reflection removed image using the generated reflection map.

[0063] In an embodiment, the image enhancer (154) compares the calculated values ​​of one or more motion characteristics associated with each of the plurality of estimated keypoints with expected values, wherein the expected values ​​are pre-calculated. In addition, the image enhancer (154) determines a deviation of the calculated values ​​of each of the plurality of estimated keypoints from the expected values. In addition, the image enhancer (154) determines a first set of keypoints in the plurality of estimated keypoints having deviations greater than a threshold. The first set of keypoints is an initial set of keypoints calculated using an existing method.

[0064] Functions associated with the AI ​​engine (155) (or ML model) can be performed by non-volatile memory, volatile memory and processor (120). One or more processors control the processing of input data according to predefined operating rules or AI models stored in non-volatile memory and volatile memory. The predefined operating rules or AI models are provided by training or learning. Here, providing by learning means formulating predefined operating rules or AI engines (155) of desired characteristics by applying a learning algorithm to multiple learning data. Learning can be performed in the device itself that executes the AI ​​according to the embodiment, and / or can be implemented by a separate server / system. A learning algorithm is a method for training a predetermined target device (e.g., a robot) using multiple learning data to prompt, allow, or control the target device to make a decision or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0065] The AI ​​engine (155) may be composed of a plurality of neural network layers. Each layer has a plurality of weighted values, and layer operations are performed by calculations of previous layers and operations on a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0066] In an embodiment, the processor (120) may include an image processing controller (150).

[0067] In an embodiment, an image processing controller (150) is configured to receive an image frame including a subject performing an action. The image processing controller (150) is configured to determine a plurality of key points associated with the subject of the received image frame. The image processing controller (150) is configured to detect the action performed by the subject using a plurality of estimated key points. The image processing controller (150) is configured to determine motion characteristics associated with the plurality of estimated key points. The image processing controller (150) is configured to identify one or more regions to be enhanced from a plurality of regions in the received image frame based on the motion characteristics determined using the plurality of estimated key points and the detected action. The image processing controller (150) is configured to generate an enhanced image including one or more enhanced regions compared to one or more regions of the plurality of regions of the received image frame. The image processing controller (150) is configured to store the enhanced image including the one or more enhanced regions of the plurality of regions.

[0068] In an embodiment, an image processing controller (150) is configured to receive an image frame including a subject performing an action. The image processing controller (150) is configured to estimate the posture of the subject (e.g., a human body) by using an AI engine (155) (a deep neural network). The subject consists of multiple key points (e.g., k approximate key points) for each part of the subject (e.g., head, wrist, etc.). Since the received image frame may be damaged due to blur, noise, and other factors, the multiple key points generated by the posture estimator (151) can only be approximate. The image processing controller (150) is configured to detect the action performed by the subject (e.g., jumping, squatting, throwing, etc.) using the multiple estimated key points by using the AI ​​engine (155), and generate an action label corresponding to the detected action.

[0069] In an embodiment, an image processing controller (150) is configured to determine the type and intensity of image artifacts of one or more regions (e.g., M regions) to be enhanced from among multiple regions in an optimal image frame (e.g., N frames) based on an action tag, a plurality of key points, and received image frames. The image processing controller (150) is configured to determine motion characteristics associated with a plurality of estimated key points, identify one or more regions to be enhanced from among multiple regions in the received image frames based on the motion characteristics determined using the plurality of estimated key points and the detected motion (e.g., a set of N image frames having M artifact regions and (1×M) vectors representing the intensity of each of these artifacts (or the included motions)), and generate an enhanced image (optimal frame) comprising one or more enhanced regions compared to one or more regions in the multiple regions of the received image frames. The image processing controller (150) is configured to minimize artifacts (image artifacts) and generate an enhanced image by utilizing a combination of received image frames.

[0070] In an embodiment, the image processing controller (150) is configured to determine a plurality of best image frames (N frames) from the received image frames (601) based on the detected action (action label), wherein the plurality of best image frames include the peak action of the detected action. The image processing controller (150) is configured to identify the peak action / peak frame for the corresponding detected action. For example, in a jumping action, the peak frame would be the highest point of the jump. For example, in a javelin throwing action, the peak frame would be the moment when the javelin leaves the user's hand. Due to the peak action / peak frame identification, the total processing time for generating the enhanced image is reduced, which is one of the advantages of the proposed method. If the peak action is not identified, calculations need to be performed for each group of k frames (k is predefined), while in the proposed method, only one calculation needs to be performed.

[0071] The image processing controller (150) is configured to predict local motion regions in received image frames based on detected motion. The image processing controller (150) is configured to include large motion for detected motion using a predefined lookup table. For example, a limb in a jump. The image processing controller (150) is configured to determine an optimal motion map (e.g., (x, y) coordinates of an area around each key point) in a plurality of optimal image frames (N frames) based on the predicted local motion regions and a plurality of estimated key points (e.g., a set of key points with possible motion).

[0072] The image processing controller (150) is configured to perform localization of space-time artifacts for multiple best image frames based on the determined best motion map, the determined motion characteristics associated with multiple estimated key points, and the detected motion, and identify one or more regions to be enhanced from multiple regions in the received image frames based on the localization of space-time artifacts.

[0073] The image processing controller (150) is configured to determine whether the one or more regions (e.g., M regions and N frames) are corrupted by certain artifacts (e.g., image artifacts) (e.g., noise / blur). The image processing controller (150) is configured to return an intensity / counter value / counter action associated with the artifact in response to determining that the one or more regions are corrupted by certain artifacts. For example, when the image processing controller (150) is configured to detect that one or more regions are corrupted by artifacts (i.e., blurred regions), the image processing controller (150) is configured to return a kernel size representing the intensity of the blur. In another example, when the image processing controller (150) is configured to detect that one or more regions are corrupted by artifacts (i.e., noise), the image processing controller (150) is configured to return a standard deviation of the noise.

[0074] The image processing controller (150) is configured to receive an image frame (701) including a subject performing an action. The image processing controller (150) is configured to identify a peak action / peak frame (702) for a corresponding detected action from the received image frame (701) based on an action tag. For example, in a jumping action, the peak frame (702) would be the highest point of the jump. The image processing controller (150) is configured to predict a local motion region (703) in the received image frame (701) based on the detected action. The local motion region (703) includes large motions (e.g., motions associated with the legs) for the detected action using a predefined lookup table.

[0075] The image processing controller (150) is configured to generate an initial motion map of a plurality of optimal image frames (801 and 802) based on an image restoration mechanism (e.g., HDR / motion deblurring). The image processing controller (150) is configured to generate a digital skeleton by connecting a plurality of estimated key points. The image processing controller (150) is configured to retrieve the motion probabilities of the key points and bones of the generated digital skeleton from a predefined dictionary of a database (i.e., a key point motion repository (111) and a skeleton motion repository (112)) of the electronic device (100) for the detected action. The motion probabilities / values ​​of the key points and bones are selected from a pre-computed lookup table (LUT) for each action.

[0076] The image processing controller (150) is configured to update the generated digital skeleton based on the retrieved key points and the motion probability of the skeleton. The image processing controller (150) is configured to perform an expansion process on the updated digital skeleton. The image processing controller (150) is configured to perform a smoothing process on the expanded digital skeleton (804). The image processing controller (150) is configured to determine an optimal motion map (805) based on the predicted local motion area / motion probability, the generated initial motion map, and the updated / expanded / smoothed digital skeleton (804). The image processing controller (150) is configured to generate by combining the initial motion map and the value of the motion probability.

[0077] The image processing controller (150) is configured to detect that one or more regions in a plurality of best image frames (N frames) are corrupted by artifacts (i.e., noise). The image processing controller (150) then uses a classical learning mechanism and a deep learning mechanism to determine a standard deviation of the noise of the plurality of best image frames. The image processing controller (150) then returns an intensity / counter value / counter action associated with the artifact to the image processing controller (150) in response to determining that the one or more regions are corrupted by certain artifacts (i.e., noise).

[0078] The image processing controller (150) is configured to determine motion parameters (e.g., displacement, velocity, and acceleration) of each key point in the predicted local motion region based on the post-estimation error and the plurality of estimated key points. The image processing controller (150) is configured to determine the post-estimation error by analyzing changes in key points in the static region in the plurality of best image frames (the previous stage gives an estimate of the low / no motion region, which can be used to determine the post-estimation error).

[0079] The image processing controller (150) is configured to use the determined motion parameters to determine motion between subsequent frames of the plurality of best image frames. The image processing controller (150) then determines a size of a blur kernel based on the determined motion to identify one or more regions to be enhanced from a plurality of regions in the received image frame. The image processing controller (150) then returns an intensity / counter value / counter action associated with the artifact to the image processing controller (150) in response to determining that the one or more regions are corrupted by certain artifacts (i.e., blur).

[0080] The image processing controller (150) is configured to cluster one or more regions identified from a plurality of regions in a received image frame based on the determined motion characteristics associated with the plurality of estimated key points and the detected actions, and cluster the received image frames into a plurality of frame groups. The plurality of frame groups include a plurality of frames with the lowest displacement (K1), a plurality of frames with the middle displacement (K2), and a plurality of frames with the highest displacement (K3). The relationship between the displacements is "K1>K2>K3".

[0081] The image processing controller (150) is configured to generate a high exposure frame (K4) from a plurality of frames having the lowest displacement. The image processing controller (150) is configured to generate a medium exposure frame (K5) from a plurality of frames having a medium displacement. The image processing controller (150) is configured to generate a low exposure frame (K6) from a plurality of frames having a highest displacement. The frames (K4, K5, and K6) are added using weighted addition of all frames. The weighted addition is performed using a motion map to remove ghosting while blending.

[0082] The image processing controller (150) is configured to blend the generated high exposure frame (K4), the generated medium exposure frame (K5) and the generated low exposure frame (K6) to generate an HDR image (1002). The high, low and medium exposure images / frames are created by blending the frames for displacement of key points to reduce ghosting and HDR. A smaller number of frames with large displacement are blended to create low exposure frames, and vice versa.

[0083] The image processing controller (150) is configured to receive N image frames and information associated with artifact measurements for M regions and N frames. The image processing controller (150) is configured to determine an average blur in each image frame based on a current blur region. The image processing controller (150) is configured to sort the image frames in ascending order of average blur. The image processing controller (150) is configured to store the sorted image frames in a memory (110). The image processing controller (150) is configured to retrieve one or more image frames from the sorted image frames. The image processing controller (150) is configured to determine a maximum blur from the retrieved one or more image frames. The image processing controller (150) is configured to determine whether the maximum blur is less than a predefined threshold (t). If the maximum blur is less than the preset threshold (t), the image enhancer (154) returns the best frame (i-th frame). Otherwise, the sorted image frames (1104-1105) list is checked until this constraint is met.

[0084] The image processing controller (150) is configured to generate a motion map based on the estimated displacement after receiving N image frames and image artifacts (motion) / measurements for M regions in the N frames. The image processing controller (150) represents the regions where the motion of the subject is detected. The motion map is typically a grayscale image with values ​​ranging from 0 to 255. The higher the value, the higher the confidence in the motion in the region. The motion map is generated using artifact measurements for the N frames.

[0085] The image processing controller (150) is configured to compensate for multi-frame motion denoising after receiving the motion map and generate a denoised image. The image processing controller (150) is configured to use the motion map to blend N frames together using weighted addition. While blending, areas with more motion are given less weighting.

[0086] The image processing controller (150) is configured to receive an image frame including a subject performing an action. The image processing controller (150) is configured to perform exposure alignment on the received image frame and generate an initial motion map. The image processing controller (150) is configured to determine a plurality of key points (human pose / digital skeleton) associated with the subject of the received image frame. The image processing controller (150) is configured to update the determined digital skeleton based on the retrieved key points and the motion probability of the skeleton. The image processing controller (150) is configured to generate an intermediate motion map based on the generated initial motion map. The image processing controller (150) is configured to generate an optimal / final motion map based on the intermediate motion map and the updated digital skeleton / multiple key points, the optimal / final motion map being generated by combining the initial motion map and the value of the motion probability.

[0087] although Figure 3 Various hardware components of the electronic device (100) are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the electronic device (100) may include fewer or greater numbers of components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope of the present invention. One or more components may be combined to perform the same or substantially similar functions for motion-based image enhancement.

[0088] Figure 4 is a flow chart (400) illustrating a method for motion-based image enhancement according to embodiments disclosed herein. The electronic device (100) performs various steps for motion-based image enhancement.

[0089] At step 401, the method includes receiving an image frame including a subject performing an action. At step 402, the method includes determining a plurality of key points associated with the subject of the received image frame. At step 403, the method includes detecting the action performed by the subject using a plurality of estimated key points. At step 404, the method includes determining motion characteristics associated with the plurality of estimated key points. At step 405, the method includes identifying one or more regions to be enhanced from a plurality of regions in the received image frame based on the motion characteristics determined using the plurality of estimated key points and the detected action. At step 406, the method includes generating an enhanced image including one or more enhanced regions compared to one or more regions of the plurality of regions of the received image frame. At step 407, the method includes storing the enhanced image including the one or more enhanced regions of the plurality of regions.

[0090] Figure 5 is a system flow diagram illustrating a method for motion-based image enhancement according to embodiments disclosed herein.

[0091] At steps 501-502, a pose estimator (151) receives an image frame including a subject performing an action. The pose estimator (151) estimates the pose of a subject (e.g., a human body) by using an AI engine (155) (a deep neural network). The subject consists of a plurality of key points (e.g., k approximate key points) for each part of the subject (e.g., head, wrist, etc.). Since the received image frame may be corrupted due to blur, noise, and other factors, the plurality of key points generated by the pose estimator (151) can only be approximate. At step 503, an action recognizer (152) detects the action performed by the subject (e.g., jumping, squatting, throwing, etc.) by using the AI ​​engine (155) and utilizing the plurality of estimated key points, and generates an action label corresponding to the detected action.

[0092] At step 504, the motion-based artifact region locator (153) determines the image artifact type and intensity of one or more regions (e.g., M regions) to be enhanced among a plurality of regions in the best image frames (e.g., N frames) based on the motion label, the plurality of key points, and the received image frames. Figure 6Additional information related to the motion-based artifact region locator (153) is shown. At steps 505-506, the image enhancer (154) determines motion characteristics associated with multiple estimated key points, identifies one or more regions to be enhanced from multiple regions in the received image frames based on the motion characteristics determined using the multiple estimated key points and the detected motion (e.g., a set of N image frames having M artifact regions and (1×M) vectors representing the intensity of each of these artifacts (or included motion)), and generates an enhanced image (optimal frame) that includes one or more enhanced regions compared to one or more regions in the multiple regions of the received image frames. The image enhancer (154) minimizes artifacts (image artifacts) and generates an enhanced image by utilizing a combination of the received image frames.

[0093] Figure 6 Various operations for motion-based image enhancement associated with a motion-based artifact region locator (153) are shown in accordance with embodiments disclosed herein.

[0094] In an embodiment, the motion-based artifact region locator (153) includes a peak motion identifier (153a), a local motion predictor (153b), a region identifier for a motion locator (153c), and a spatio-temporal artifact locator (153d).

[0095] A peak action identifier (153a) determines a plurality of best image frames (N frames) from the received image frames (601) based on the detected action (action label), wherein the plurality of best image frames include a peak action of the detected action. The peak action identifier (153a) identifies the peak action / peak frame for the corresponding detected action. For example, in a jumping action, the peak frame would be the highest point of the jump. For example, in a javelin throwing action, the peak frame would be the moment when the javelin leaves the user's hand. Due to the peak action / peak frame identification, the total processing time for generating an enhanced image is reduced, which is one of the advantages of the proposed method. If the peak action is not identified, calculations need to be performed for each group of k frames (k is predefined), whereas in the proposed method, only one calculation is required.

[0096] A local motion predictor (153b) predicts local motion regions in received image frames based on detected motion. The local motion regions include large motions for detected motions using a predefined lookup table. For example, a limb in a jump. A region identifier (153c) for a motion locator determines an optimal motion map (e.g., (x, y) coordinates of a region around each keypoint) in a plurality of optimal image frames (N frames) based on the predicted local motion regions and a plurality of estimated keypoints (e.g., a set of keypoints with possible motions).

[0097] A space-time artifact locator (153d) performs space-time artifact localization for multiple best image frames based on the determined best motion map, the determined motion characteristics associated with multiple estimated key points, and the detected actions, and identifies one or more regions to be enhanced from multiple regions in the received image frames based on the location of the space-time artifacts.

[0098] The space-time artifact locator (153d) determines whether the one or more regions (e.g., M regions and N frames) are corrupted by certain artifacts (e.g., image artifacts) (e.g., noise / blur). In response to determining that the one or more regions are corrupted by certain artifacts, the space-time artifact locator (153d) returns an intensity / counter value / counter action associated with the artifact. For example, when the space-time artifact locator (153d) detects that one or more regions are corrupted by artifacts (i.e., blurred regions), the space-time artifact locator (153d) returns a kernel size representing the intensity of the blur. In another example, when the space-time artifact locator (153d) detects that one or more regions are corrupted by artifacts (i.e., noise), the space-time artifact locator (153d) returns the standard deviation of the noise.

[0099] Figure 7 Various operations for motion-based image enhancement associated with a peak action identifier (153a) and a local motion predictor (153b) are shown in accordance with embodiments disclosed herein.

[0100] The peak action identifier (153a) receives an image frame (701) including a subject performing an action. The peak action identifier (153a) identifies a peak action / peak frame (702) for a corresponding detected action from the received image frame (701) based on the action label. For example, in a jumping action, the peak frame (702) would be the highest point of the jump. The local motion predictor (153b) predicts a local motion region (703) in the received image frame (701) based on the detected action. The local motion region (703) includes large motions (e.g., motions associated with the legs) for the detected action using a predefined lookup table.

[0101] Figure 8 Various operations for motion based image enhancement associated with a region identifier (153c) for a motion locator are shown in accordance with embodiments disclosed herein.

[0102] The region identifier (153c) for the motion localizer includes an initial motion map creator (153ca), a digital skeleton creator (153cb), a keypoint / bone strength updater (153cc), a dilation engine (153cd), a Gaussian smoother (153ce) and a final motion map creator (153cf).

[0103] An initial motion map creator (153ca) generates an initial motion map of a plurality of optimal image frames (801 and 802) based on an image restoration mechanism (e.g., HDR / motion deblurring). A digital skeleton creator (153cb) generates a digital skeleton by connecting a plurality of estimated key points. A key point / bone strength updater (153cc) retrieves motion probabilities of key points and bones of the generated digital skeleton from a predefined dictionary of a database (i.e., a key point motion repository (111) and a bone motion repository (112)) of the electronic device (100) for the detected motion. The motion probabilities / values ​​of the key points and bones are selected from a pre-computed lookup table (LUT) for each motion.

[0104] The key point / bone strength updater (153cc) updates the generated digital skeleton based on the retrieved key points and motion probabilities of the bones. The dilation engine (153cd) performs a dilation process on the updated digital skeleton. The Gaussian smoother (153ce) performs a smoothing process on the dilated digital skeleton (804). The final motion map creator (153cf) determines an optimal motion map (805) based on the predicted local motion regions / motion probabilities, the generated initial motion map, and the updated / dilated / smoothed digital skeleton (804). The final motion map creator (153cf) is generated by combining the initial motion map and the values ​​of the motion probabilities.

[0105] Fig. 9 Various operations associated with a spatio-temporal artifact locator (153d) for motion-based image enhancement according to embodiments disclosed herein are shown. The spatio-temporal artifact locator (153d) includes a noise analyzer (153da), a motion analyzer (153db), a pose controller (153dc), and a blur kernel (153dd).

[0106] The noise analyzer (153da) detects that one or more regions in the plurality of best image frames (N frames) are corrupted by artifacts (i.e., noise). The noise analyzer (153da) then determines the standard deviation of the noise of the plurality of best image frames using a classical learning mechanism and a deep learning mechanism. The noise analyzer (153da) then returns an intensity / counter value / counter action associated with the artifact to the image enhancer (154) in response to determining that the one or more regions are corrupted by certain artifacts (i.e., noise).

[0107] The motion analyzer (153db) determines motion parameters (e.g., displacement, velocity, and acceleration) of each key point in the predicted local motion region based on the post-estimation error and the plurality of estimated key points, wherein the posture controller (153dc) determines the post-estimation error by analyzing the changes of key points in the static region in the plurality of best image frames (the previous stage gives an estimate of the low / no motion region, which can be used to determine the post-estimation error).

[0108] The blur kernel (153dd) uses the determined motion parameters to determine motion between subsequent frames of the plurality of best image frames. The blur kernel (153dd) then determines a size of the blur kernel based on the determined motion to identify one or more regions to be enhanced from a plurality of regions in the received image frame. The blur kernel (153dd) then returns an intensity / counter value / counter action associated with the artifact to the image enhancer (154) in response to determining that the one or more regions are corrupted by certain artifacts (i.e., blur).

[0109] Fig.10 Various operations associated with an image enhancer (154) for generating an HDR image according to embodiments disclosed herein are shown. The image enhancer (154) includes a cluster generator (154a), motion compensated addition-1 (154b), motion compensated addition-2 (154c), motion compensated addition-3 (154d), and an HDR merger (154e).

[0110] The cluster generator (154a) clusters one or more regions identified from a plurality of regions in the received image frames based on the determined motion characteristics associated with the plurality of estimated key points and the detected actions, and clusters the received image frames into a plurality of frame groups. The plurality of frame groups include a plurality of frames with the lowest displacement (K1), a plurality of frames with the middle displacement (K2), and a plurality of frames with the highest displacement (K3). The relationship between the displacements is "K1>K2>K3".

[0111] The cluster generator (154a) generates a high exposure frame (K4) from the plurality of frames having the lowest displacement. The cluster generator (154a) generates a medium exposure frame (K5) from the plurality of frames having the medium displacement. The cluster generator (154a) generates a low exposure frame (K6) from the plurality of frames having the highest displacement. The frames (K4, K5, and K6) are added using weighted addition of all frames. The weighted addition is performed using a motion map to remove ghosting while blending.

[0112] The HDR merger (154e) blends the generated high exposure frame (K4), the generated medium exposure frame (K5), and the generated low exposure frame (K6) to generate an HDR image (1002). The high, low, and medium exposure images / frames are created by blending the frames for displacement of key points to reduce ghosting and HDR. A smaller number of frames with large displacements are blended to create low exposure frames, and vice versa. Fig.10 A comparison (1000) between a conventional HDR image (1001) and a proposed HDR image (1002) is shown. Compared to the conventional HDR image (1001), the proposed HDR (1002) has no ghosting effects / image artifacts.

[0113] Fig.11 is a flow chart (1100) illustrating a method for generating a blur-corrected image using an image enhancer (154) according to embodiments disclosed herein.

[0114] At step 1101, the method includes receiving N image frames and information associated with artifact measurements for M regions and N frames. At step 1102, the method includes determining an average blur in each image frame based on a current blur region. At step 1103, the method includes sorting the image frames in ascending order of average blur. At step 1104, the method includes storing the sorted image frames in a memory (110). At step 1105, the method includes retrieving one or more image frames from the sorted image frames. At step 1106, the method includes determining a maximum blur from the retrieved one or more image frames. At steps 1107-1108, the method includes determining whether the maximum blur is less than a predefined threshold (t). If the maximum blur is less than the preset threshold (t), the image enhancer (154) returns the best frame (i-th frame). Otherwise, the sorted list of image frames (1104-1105) is checked until this constraint is met.

[0115] Fig.12 Various operations associated with an image enhancer (154) for generating a denoised image in accordance with embodiments disclosed herein are shown. The image enhancer (154) includes a displacement motion mapper (154f) and a noise reduction engine (154g).

[0116] The displacement motion mapper (154f) generates a motion map based on the estimated displacement after receiving N image frames and image artifact (motion) / measurements for M regions in the N frames. The motion map represents the regions where motion of the subject is detected. The motion map is typically a grayscale image with values ​​ranging from 0 to 255. The higher the value, the higher the confidence of motion in the region. The motion map is generated using artifact measurements for the N frames.

[0117] The denoising engine (154g) compensates for multi-frame motion denoising upon receiving the motion map and generates a denoised image. The denoising engine (154g) uses the motion map to blend the N frames together using weighted addition. While blending, areas with more motion are given less weighting. This ensures that there will be no ghosting or blurring in the final output (denoised image). In addition, in areas where there is no motion, the noise will be greatly reduced. Fig.12 A comparison (1200) between a conventional denoised image (1201) and a proposed denoised image (1202) is shown. Compared to the conventional denoised image (1201), the proposed denoised image (1202) has no noise effects / image artifacts.

[0118] The photometric difference between images is used in traditional motion map generation methods. Due to the presence of high noise, this may lead to false positive motion regions in larger areas. As a result, the final output is noisy (1201). However, the proposed method accurately detects the motion regions and the frame with the least motion can be selected for motion compensated denoising (1202).

[0119] Fig.13a and Fig.13b is an example flow chart (1300) illustrating a method for motion-based image enhancement according to embodiments disclosed herein. The electronic device (100) performs various steps for motion-based image enhancement.

[0120] At step 1301, the method includes receiving an image frame including a subject performing an action. At steps 1302-1303, the method includes performing exposure alignment on the received image frame and generating an initial motion map. At step 1304, the method includes determining a plurality of key points (human pose / digital skeleton) associated with the subject of the received image frame. At step 1305, the method includes updating the determined digital skeleton based on the retrieved key points and motion probabilities of the skeleton. At step 1306, the method includes generating an intermediate motion map based on the generated initial motion map. At steps 1307-1308, the method includes generating an optimal / final motion map based on the intermediate motion map and the updated digital skeleton / multiple key points, the optimal / final motion map being generated by combining the initial motion map and the values ​​of the motion probabilities.

[0121] The various actions, behaviors, blocks, steps, etc. in the flowcharts (400, 1100, and 1300) may be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some of the actions, behaviors, blocks, steps, etc. may be omitted, added, modified, or skipped without departing from the scope of the present invention.

[0122] The embodiments disclosed herein may be implemented using at least one hardware device and performing a network management function to control elements.

[0123] The above description of the specific embodiments fully reveals the general nature of the embodiments herein, so that other technicians can easily modify and / or adjust these specific embodiments for various applications without departing from the overall concept by applying existing knowledge, and therefore such adjustments and modifications should and are intended to be included in the meaning and scope of the equivalents of the disclosed embodiments. It is understood that the phrases or terms used herein are for descriptive purposes and not restrictive. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced by modification within the scope of the embodiments as described herein.

Claims

1. A motion-based image enhancement method, wherein: The method comprises: Receiving, by an electronic device (100), a plurality of image frames, the plurality of image frames including at least one subject performing at least one action; estimating, by the electronic device (100), a plurality of key points associated with the at least one subject included in a plurality of received image frames; detecting, by the electronic device (100), the at least one action performed by the at least one subject using a plurality of estimated key points; determining, by the electronic device (100), at least one motion characteristic associated with each of the plurality of estimated key points; identifying, by the electronic device (100), one or more regions to be enhanced from a plurality of regions in at least one received image frame among the plurality of received image frames based on at least one determined motion characteristic associated with each of the plurality of estimated key points and at least one detected action; and The electronic device (100) generates an enhanced image including one or more enhanced regions by applying at least one image enhancement to the identified one or more regions.

2. The method according to claim 1, wherein: Identifying, by the electronic device (100), the one or more regions to be enhanced from the plurality of regions in the at least one received image frame among the plurality of received image frames based on the at least one determined motion characteristic associated with each of the plurality of estimated key points and the at least one detected action comprises: Determining, by the electronic device (100), an optimal motion map using a plurality of optimal image frames based on at least one predicted local motion region and the plurality of estimated key points; performing, by the electronic device (100), localization of spatio-temporal artifacts for the plurality of best image frames based on the determined best motion map, the at least one determined motion characteristic associated with the plurality of estimated key points, and the at least one detected action; and The electronic device (100) identifies one or more regions to be enhanced from the multiple regions in at least one received image frame among the multiple received image frames based on the positioning of the space-time artifact, wherein the one or more regions include at least one image artifact.

3. The method according to claim 2, wherein: Determining, by the electronic device (100), the best motion map using the multiple best image frames based on the at least one predicted local motion region and the multiple estimated key points comprises: Determining, by the electronic device (100), the plurality of best image frames from the plurality of received image frames based on the at least one detected motion; at least one local motion region in at least one best image frame among a plurality of best image frames determined by the electronic device (100) based on the at least one detected motion prediction; Determining, by the electronic device (100), a digital skeleton using the plurality of estimated key points; and The electronic device (100) determines the optimal motion map using the multiple optimal image frames based on the at least one predicted local motion region and the digital skeleton.

4. The method according to claim 1, wherein: Generating the enhanced image by applying the at least one image enhancement includes generating at least one of a high dynamic range (HDR) image, a denoised image, a blur-corrected image, or a reflection-removed image.

5. The method according to claim 4, wherein: Generating the HDR image comprises: The electronic device (100) clusters one or more regions identified from the plurality of regions in the at least one received image frame and clusters the plurality of received image frames into a plurality of frame groups, based on the at least one determined motion characteristic associated with the plurality of estimated key points and the at least one detected action, respectively, wherein the plurality of frame groups include a first frame group, a second frame group, and a third frame group, the first frame group includes a plurality of frames with a lowest displacement, the second frame group includes a plurality of frames with a medium displacement, and the third frame group includes a plurality of frames with a highest displacement; The electronic device (100) generates a high exposure frame using a plurality of frames in the first frame group; The electronic device (100) generates a medium exposure frame using a plurality of frames in the second frame group; The electronic device (100) generates a low exposure frame using a plurality of frames in the third frame group; and The electronic device (100) mixes the generated high exposure frame, the generated medium exposure frame and the generated low exposure frame to generate the HDR image.

6. The method according to claim 4, wherein: Generating the denoised image includes generating a motion map based on the at least one determined motion characteristic associated with each of the plurality of estimated keypoints.

7. The method according to claim 4, wherein: Generating the blur-corrected image comprises: Determining, by the electronic device (100), whether at least one motion characteristic exceeds a predefined threshold; and The blur-corrected image is generated by the electronic device (100) by applying blur correction to one or more areas around key points whose motion characteristics exceed the predefined threshold.

8. The method according to claim 4, wherein: Generating the reflection removed image comprises: determining, by the electronic device (100), a correlation between at least one determined motion characteristic associated with each of a plurality of estimated key points of a first subject and at least one determined motion characteristic associated with each of a plurality of estimated key points of a second subject; The electronic device (100) classifies at least one highly relevant key point of the second body as a reflection key point; The electronic device (100) generates a reflection map using the classified at least one highly correlated key point; and The electronic device (100) generates the reflection-removed image using the generated reflection map.

9. The method according to claim 1, wherein: Identifying, by the electronic device (100), the one or more regions to be enhanced from the plurality of regions in the at least one received image frame among the plurality of received image frames based on the at least one determined motion characteristic associated with each of the plurality of estimated key points and the at least one detected action comprises: comparing, by the electronic device (100), the calculated value of the at least one determined motion characteristic associated with each of the plurality of estimated key points with an expected value of the motion characteristic associated with each of the plurality of estimated key points; determining, by the electronic device (100), a deviation of the calculated value from the expected value for each of the plurality of estimated key points; and A first group of key points having a deviation greater than a threshold value among the plurality of estimated key points is determined by the electronic device (100).

10. An electronic device (100) for motion-based image enhancement, wherein: The electronic device (100) comprises: Memory (110); a processor (120); and An image processing controller (150), coupled to the memory (110) and the processor (120), is configured to: receiving a plurality of image frames, the plurality of image frames including at least one subject performing at least one action; estimating a plurality of key points associated with the at least one subject included within a plurality of received image frames; detecting the at least one action performed by the at least one subject using a plurality of estimated key points; determining at least one motion characteristic associated with each of the plurality of estimated keypoints; identifying one or more regions to be enhanced from a plurality of regions in at least one received image frame among the plurality of received image frames based on at least one determined motion characteristic associated with each of the plurality of estimated keypoints and at least one detected action; and An enhanced image including one or more enhanced regions is generated by applying at least one image enhancement to the identified one or more regions.

11. The electronic device according to claim 10, wherein: The image processing controller is further configured to: determining the pose of a subject in a captured scene; identifying a plurality of key points from the posture; measuring a plurality of motion parameters for each of the plurality of key points; checking whether the measured motion parameter exceeds a predefined threshold; as well as Blur correction is applied to areas surrounding keypoints where the measured motion parameter exceeds the predefined threshold.

12. The electronic device according to claim 10, wherein: The image processing controller is further configured to: determining an optimal motion map using a plurality of optimal image frames based on at least one predicted local motion region and the plurality of estimated key points; performing localization of spatio-temporal artifacts for the plurality of best image frames based on the determined best motion map, the at least one determined motion characteristic associated with the plurality of estimated keypoints, and the at least one detected action; as well as Based on the location of the spatio-temporal artifact, the one or more regions to be enhanced are identified from the plurality of regions in the at least one received image frame among the plurality of received image frames, wherein the one or more regions include at least one image artifact.

13. The electronic device according to claim 12, wherein: The image processing controller is further configured to: determining the plurality of best image frames from the plurality of received image frames based on the at least one detected motion; predicting at least one local motion region in at least one best image frame among the determined plurality of best image frames based on the at least one detected motion; determining a digital skeleton using the plurality of estimated key points; and The optimal motion map is determined using the plurality of optimal image frames based on at least one predicted local motion region and the digital skeleton.

14. The electronic device according to claim 10, wherein: The image processing controller is further configured to generate at least one of a high dynamic range (HDR) image, a denoised image, a blur-corrected image, or a reflection-removed image.

15. The electronic device according to claim 14, wherein: The image processing controller is further configured to: Based on the at least one determined motion characteristic associated with the plurality of estimated key points and the at least one detected action, clustering one or more regions identified from the plurality of regions in the at least one received image frame and clustering the plurality of received image frames into a plurality of frame groups, respectively, wherein the plurality of frame groups include a first frame group including a plurality of frames with a lowest displacement, a second frame group including a plurality of frames with a medium displacement, and a third frame group including a plurality of frames with a highest displacement; generating a high exposure frame using a plurality of frames in the first frame group; generating a medium exposure frame using a plurality of frames in the second frame group; generating a low exposure frame using a plurality of frames in the third frame group; and The generated high exposure frame, the generated medium exposure frame, and the generated low exposure frame are mixed to generate the HDR image.

Images