Device, method, non-transitory computer-readable medium, and apparatus for motion compensation for neural network enhanced images

Abstract

A device includes a memory and one or more processors. The memory is configured to store instructions. The one or more processors are configured to execute instructions to perform the following operations: apply a neural network to a first image to generate an enhanced image. The one or more processors are also configured to execute instructions to perform the following operations: adjust at least a portion of the high-frequency components of the enhanced image based on motion compensation operations to generate adjusted high-frequency image components. The one or more processors are also configured to execute instructions to perform the following operations: combine the low-frequency components of the enhanced image and the adjusted high-frequency image components to generate an adjusted enhanced image.

Description

[Technical Field]

[0001] In summary, the subject matter of this case concerns the application of motion compensation to enhanced images. [Previous Technology]

[0002] Technological advancements have led to smaller and more powerful computing devices. For example, a wide variety of portable personal computing devices exist today, including small, lightweight cordless phones (such as mobile and smartphones, tablets, and laptops) that are easily carried by the user. These devices can transmit voice and data packets over wireless networks. Furthermore, many of these devices incorporate additional functionalities such as digital cameras, digital camcorders, digital recorders, and audio file players. Moreover, such devices can process executable instructions, including software applications that can access the Internet (such as web browser applications). Therefore, these devices can include critical computing capabilities.

[0003] Such computing devices typically incorporate functionality for displaying images. For example, an image stream may include downscaled low-resolution images representing video content. For instance, the image stream may be received from another device (e.g., downloaded) and may include low-resolution images to meet transmission bandwidth limitations. As another example, the image stream may be generated at a lower resolution (e.g., by a camera or graphics processor) and will be displayed at a higher resolution. Super-resolution upscaling can be used to reconstruct a high-resolution image from a low-resolution image. The reconstructed high-resolution image adds textures that are not present in the low-resolution image. However, the added textures may be inconsistent between image frames, resulting in visible flickering in the video. [Summary of the Invention]

[0004] According to one implementation of the present invention, a device includes a memory and one or more processors. The memory is configured to store instructions. The one or more processors are configured to execute the instructions to perform the following operations: applying a neural network to a first image to generate an enhanced image. The one or more processors are also configured to execute the instructions to perform the following operations: adjusting at least a portion of the high-frequency components of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component. The one or more processors are also configured to execute the instructions to perform the following operations: combining the low-frequency components of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0005] According to another implementation of the present invention, a method includes: at a device, applying a neural network to a first image to generate an enhanced image. The method also includes: at the device, adjusting at least a portion of the high-frequency components of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component. The method further includes: at the device, combining the low-frequency components of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0006] According to another implementation of the present invention, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to: apply a neural network to a first image to generate an enhanced image. The instructions, when executed by the one or more processors, also cause the one or more processors to: adjust at least a portion of the high-frequency components of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component. The instructions, when executed by the one or more processors, also cause the one or more processors to: combine the low-frequency components of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0007] According to another implementation of the present invention, an apparatus includes: a unit for applying a neural network to a first image to generate an enhanced image. The apparatus also includes: a unit for adjusting at least a portion of a high-frequency component of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component. The apparatus also includes: a unit for combining a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0008] Other aspects, advantages and features of this application will become apparent after reading the entire application (including the following sections: description of drawings, detailed description and scope of the application).

Implementation Method

[0030] Upscaling is a technique that can be used to generate high-resolution images from low-resolution images. High-resolution images can have added textures that are not present in low-resolution images. However, the added textures may be inconsistent between image frames and may cause visible flicker for viewers of upscaled video, which can be distracting and impair the user experience.

[0031] Systems and methods for performing motion compensation on images enhanced by neural networks are disclosed. For example, the image intensifier includes an image generator and a motion analyzer. The image generator uses a neural network to generate an enhanced image from a low-resolution image. For example, the neural network is used to upscale the low-resolution image and add texture to it to generate the enhanced image. The enhanced image may be referred to as a high-resolution image.

[0032] As used herein, "low resolution" and "high resolution" are relative terms. In a first instance, a first image having a first resolution (e.g., corresponding to 1080p) is upscaled to produce a second image having a second resolution (e.g., corresponding to 7680p). In this instance, the first image may be referred to as a low-resolution image, and the second image may be referred to as a high-resolution image. In a second instance, a first image having a first resolution (e.g., corresponding to 480p) is upscaled to produce a second image having a second resolution (e.g., corresponding to 1080p). In this instance, the first image may be referred to as a low-resolution image, and the second image may be referred to as a high-resolution image. Therefore, an image having a specific resolution (e.g., corresponding to 1080p) may be considered low resolution in some implementations and high resolution in others.

[0033] The image generator also identifies the high-frequency components of the enhanced image. In a specific instance, the high-frequency components of the enhanced image correspond to the difference between the enhanced image and an upscaled version of the low-resolution image. In another instance, the image generator applies a low-pass filter to the enhanced image to extract the high-frequency components.

[0034] The motion analyzer estimates motion information indicating the motion between the current image frame (e.g., a low-resolution image) and a previous image frame (e.g., a previous low-resolution image). Based on the motion information, the motion analyzer performs motion compensation on the previous high-frequency component (e.g., a previously adjusted high-frequency component) to generate a predicted high-frequency component. The motion analyzer generates an adjusted high-frequency component based on the predicted high-frequency component and a high-frequency component extracted from the enhanced image. For example, the motion analyzer adjusts the extracted high-frequency component based on the predicted high-frequency component to generate the adjusted high-frequency component. In some instances, the motion analyzer performs temporal filtering based on the predicted high-frequency component and the extracted high-frequency component to generate the adjusted high-frequency component.

[0035] The motion analyzer combines the low-frequency components of the enhanced image with the adjusted high-frequency components to produce an adjusted enhanced image. Applying motion compensation and temporal filtering to the high-frequency components (rather than the entire image frame) avoids filtering artifacts in the low-frequency components, as these artifacts may be more visible at low frequencies.

[0036] The following description, with reference to the accompanying drawings, describes a particular aspect of the subject matter. In this description, common features are designated by common component symbols. As used herein, various terms are used only to describe a particular implementation and are not intended to limit the implementation. For example, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context explicitly indicates otherwise. Furthermore, some features described herein are singular in some implementations and plural in others. For example, Figure 1 illustrates a device 102 including one or more processors ("processor" 190 in Figure 1), indicating that in some implementations, device 102 includes a single processor 190, while in other implementations, device 102 includes multiple processors 190.

[0037] As used herein, the terms "comprise," "comprises," and "comprising" may be used interchangeably with "include," "includes," or "including." Additionally, the term "wherein" may be used interchangeably with "where." As used herein, "exemplary" indicates an instance, implementation, and / or state, and should not be construed as limiting or indicating a preference or preferred implementation. As used herein, ordinal numbers used to modify elements such as structures, components, operations, etc. (e.g., "first," "second," "third," etc.) do not themselves indicate any priority or order of that element relative to another element, but merely distinguish that element from another element with the same name (but without the ordinal number). As used herein, the term "set" refers to one or more of a particular element, and the term "plural" refers to multiple of a particular element (e.g., two or more).

[0038] As used herein, "coupling" can include "communicationally coupled," "electrically coupled," or "physically coupled," and may (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicationally coupled, electrically coupled, or physically coupled) via one or more other devices, components, lines, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative, non-limiting example, two electrically coupled devices (or components) may be included in the same or different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some implementations, two communicationally coupled devices (or components) (such as those performing electronic communication) may directly or indirectly (via one or more lines, buses, networks, etc.) send and receive signals (e.g., digital or analog signals). As used herein, "directly coupled" can include two devices coupled (e.g., communicationally coupled, electrically coupled, or physically coupled) without intermediate components.

[0039] In this context, terms such as “determine,” “calculate,” “estimate,” “shift,” and “adjust” may be used to describe how one or more operations are performed. It should be noted that such terms should not be construed as restrictive, nor should they imply that similar operations can be performed using other techniques. Furthermore, as cited herein, “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” may be used interchangeably. For example, “generate,” “calculate,” “estimate,” or “determine” a parameter (or signal) may represent actively generating, estimating, calculating, or determining a parameter (or signal), or may represent using, selecting, or accessing a parameter (or signal) that has already been generated, for example, by another component or device.

[0040] Referring to FIG1, a specific illustrative example of a system 100 configured to perform motion compensation for neural network-enhanced images is shown. System 100 includes a device 102 coupled to a display device 104, a camera 126, a device 160, or a combination thereof. Device 102 is configured to perform motion compensation for neural network-enhanced images using an image intensifier 154.

[0041] Device 102 includes one or more processors 190, which include an image intensifier 154. The image intensifier 154 includes an image generator 152 coupled to a motion analyzer 112. In some implementations, one or more processors 190 are configured to receive camera input 127 from camera 126, output an adjusted enhanced image 123 to a display device 104, or perform both operations. A modem 170 is coupled to one or more processors 190 and is configured to enable communication with device 160, such as receiving a first bitstream including input data 161 via wireless transmission, sending a second bitstream including output data 171, or performing both operations.

[0042] Image intensifier 154 is configured to process low-resolution image 107 to produce an adjusted enhanced image 123. In some implementations, low-resolution image 107 corresponds to input data 161 received from device 160. For example, device 160 downscales image data (e.g., video content, burst photos, animation content, etc.) to produce downscaled data and compresses the downscaled data to produce input data 161. Decoders of one or more processors 190 decompress input data 161 to produce low-resolution image 107. In some implementations, camera input 127 (e.g., video content, burst photos, etc.) includes low-resolution image 107. For example, image intensifier 154 is configured to retrieve low-resolution image 107 from a camera buffer. In some implementations, low-resolution image 107 is generated by one or more processors 190 (e.g., graphics processing unit (GPU), gaming application, video editing application, etc.). In one instance, the image intensifier 154 is configured to retrieve a low-resolution image 107 from the GPU buffer.

[0043] Image generator 152 is configured to process low-resolution image 107 to generate enhanced image 111. Image generator 152 includes neural network 156 (e.g., Super-Resolution Generative Adversarial Network (SRGAN)) configured to process low-resolution image 107 (e.g., current image frame) to generate enhanced image 111 (e.g., upscaled and with added texture), as further described with reference to Figures 2A-3A and 3C. Image generator 152 is also configured to generate high-frequency (hi-freq.) image components 115 (e.g., added texture), high-resolution (hi-res.) low-frequency (low-freq.) images 113 (e.g., upscaled current image frame), or both, as further described with reference to Figures 2A-3C. In some implementations, neural network 156 is configured to process low-resolution image 107 (e.g., current image frame) to generate high-frequency image component 115 (e.g., added texture), as further described with reference to Figures 2A-2B and 3B. In these implementations, image generator 152 is configured to generate enhanced image 111 (e.g., upscaled and with added texture) based on high-frequency image component 115 (e.g., added texture) and high-resolution low-frequency image 113 (e.g., upscaled current image frame).

[0044] Motion analyzer 112 is configured to process augmented image 111 to produce adjusted augmented image 123, as further described with reference to Figures 2A-2C, 4 and 5. In a particular instance, motion analyzer 112 includes motion estimator 106 coupled to combiner 114 via motion-compensated adjuster 108. Motion estimator 106 is configured to generate motion vector 117 (e.g., previous low-resolution image or previous high-resolution low-frequency image) by performing motion analysis based on current image frames (e.g., low-resolution image 107 or high-resolution low-frequency image 113) and previous image frames, as further described with reference to Figures 4-5.

[0045] The motion-compensated adjuster 108 is configured to adjust the high-frequency image component 115 (e.g., added texture) based on the motion vector 117 to produce an adjusted high-frequency image component 121, as further described with reference to Figures 4-5. For example, the motion-compensated adjuster 108 is configured to produce a predicted high-frequency image component by applying motion compensation to a previous high-frequency image component. The motion-compensated adjuster 108 is configured to perform temporal filtering based on the predicted high-frequency image component and the high-frequency image component 115 (e.g., added texture) to produce the adjusted high-frequency image component 121, as further described with reference to Figures 4-5. For example, the motion-compensated adjuster 108 adjusts the high-frequency image component 115 based on the predicted high-frequency image component (e.g., which is based on motion compensation operation) to produce the adjusted high-frequency image component 121. Combiner 114 is configured to combine high-resolution low-frequency image 113 (e.g., upscaled current image frame) and adjusted high-frequency image component 121 (e.g., adjusted added texture) to produce adjusted enhanced image 123.

[0046] In some implementations, the motion-compensated adjuster 108 selectively applies temporal filtering based on the accuracy of the motion vector 117, as further described with reference to Figures 4-5. As an example, the motion-compensated adjuster 108 applies motion compensation to a previous image frame based on the motion vector 117 to generate a predicted image frame, and determines the accuracy of the motion vector 117 based on the predicted image frame and the current image frame (e.g., low-resolution image 107 or enhanced image 111), as further described with reference to Figures 4-5. If the motion vector 117 is accurate, the motion-compensated adjuster 108 applies temporal filtering. Alternatively, if the motion vector 117 is inaccurate, the motion-compensated adjuster 108 avoids applying temporal filtering. For example, if the motion vector 117 is inaccurate, the adjusted high-frequency image component 121 corresponds to the high-frequency image component 115 (e.g., no adjustment), while the adjusted enhanced image 123 corresponds to the enhanced image 111 (e.g., no adjustment). In some implementations, the motion-compensated adjuster 108 determines whether to apply temporal filtering on a primitive-by-primitive basis.

[0047] In a particular configuration, the motion-compensated adjuster 108 is configured to selectively generate an adjusted enhanced image 123. For example, the motion-compensated adjuster 108 determines a motion metric based on motion vector 117 (e.g., without performing motion compensation based on motion vector 117 to generate a predicted image frame), and determines whether to generate the adjusted enhanced image 123 based on the motion metric. For example, the motion-compensated adjuster 108 bypasses motion compensation and temporal filtering based on the motion metric (e.g., avoids generating the adjusted enhanced image 123) and outputs an enhanced image 111. The motion metric (e.g., the sum of motion vectors) indicates the motion associated with the low-resolution image 107. The motion-compensated adjuster 108 determines whether to generate the adjusted enhanced image 123 based on the motion indicated by the motion metric. In one particular implementation, the motion-compensated adjuster 108 adjusts the high-frequency image component 115 to produce an adjusted high-frequency image component 121 in response to a determination that the motion metric indicates the motion meets a motion criterion, and produces an adjusted enhanced image 123 based on the adjusted low-frequency image component 121. Alternatively, the motion-compensated adjuster 108 avoids producing the adjusted enhanced image 123 in response to a determination that the motion metric indicates the motion fails to meet a motion criterion, and outputs the enhanced image 111 as the next output image in the output stream. For example, the motion-compensated adjuster 108 provides the enhanced image 111 to a display device 104, a device 160, memory, another device, or a combination thereof. In one particular case, the motion-compensated adjuster 108 determines that the motion meets a motion criterion in response to a determination that the motion is less than a motion threshold (e.g., 5 primitives). For example, in the presence of large motion (e.g., greater than or equal to 5 primitives), artifacts may be less visible. In such cases, motion compensation and temporal filtering can be bypassed (e.g., to generate adjusted enhanced images 123) to save resources, since less visible artifacts have a limited (if any) adverse effect on video quality and user experience.

[0048] In some implementations, device 102 corresponds to or is included in a variety of types of devices. In one illustrative example, one or more processors 190 integrate at least one of the following: a mobile phone or tablet device as described with reference to FIG. 10, a wearable electronic device as described with reference to FIG. 11, a camera device as described with reference to FIG. 12, or a virtual reality, mixed reality, or augmented reality headset as described with reference to FIG. 13. In another exemplary example, one or more processors 190 are integrated into a vehicle such as those described further with reference to FIGS. 14 and 15.

[0049] During operation, image intensifier 154 accesses (e.g., receives or retrieves) low-resolution image 107. In one particular instance, image intensifier 154 receives input data 161 from device 160 via modem 170. One or more processors 190 generate low-resolution image 108 based on input data 161 and provide low-resolution image 107 to image intensifier 154, as further described with reference to FIG. 7. In another instance, image intensifier 154 receives low-resolution image 107 from camera 126. In yet another instance, image intensifier 154 receives low-resolution image 107 from one or more processors 190 (e.g., GPU).

[0050] In one particular implementation, image enhancer 154 receives an image enhancement request from an application (e.g., a video editing application, a game application, a presentation application, etc.). Image enhancer 154, in response to receiving the image enhancement request, retrieves low-resolution image 107 from a buffer. In one particular instance, image enhancer 154 may access multiple buffers. Image enhancer 154 selects a specific buffer based on the image enhancement request, the type of the application, or both, and retrieves low-resolution image 107 from the selected buffer. For example, image enhancer 154 retrieves low-resolution image 107 from a GPU buffer in response to determining that the application has a first type (e.g., a game application) or that the image enhancement request indicates a GPU buffer. As another example, image enhancer 154 retrieves low-resolution image 107 from a camera buffer in response to determining that the application has a second type (e.g., a video editing application) or that the image enhancement request indicates a camera buffer.

[0051] Image generator 152 applies neural network 156 (e.g., SRGAN) to low-resolution image 107 (e.g., current image frame) to generate enhanced image 111 (e.g., upscaled and with added texture), as further described with reference to Figures 2A-3A and 3C. Image generator 152 generates high-resolution low-frequency image 113 (e.g., upscaled current image frame) and high-frequency image component 115 (e.g., added texture) of enhanced image 111, as further described with reference to Figures 2A-3C.

[0052] In some implementations, image generator 152 applies neural network 156 to low-resolution image 107 (e.g., current image frame) to generate high-frequency image component 115 (e.g., added texture), as further described with reference to Figures 2A-2B and 3B. In these implementations, image generator 152 generates enhanced image 111 (e.g., upscaled and with added texture) based on high-frequency image component 115 (e.g., added texture) and high-resolution low-frequency image 113 (e.g., upscaled current image frame). Image generator 152 provides motion analyzer 112 with enhanced image 111 (e.g., upscaled and with added texture), high-resolution low-frequency image 113 (e.g., upscaled), high-frequency image component 115 (e.g., added texture), or a combination thereof.

[0053] Motion analyzer 112 performs motion compensation operations on the previous high-frequency image components to generate predicted high-frequency image components. In one particular case, motion analyzer 112 adjusts the high-frequency image component 115 of enhanced image 111 (e.g., added texture) based on the predicted high-frequency image components to generate adjusted high-frequency image component 121 (e.g., motion-adjusted added texture). For example, motion estimator 106 generates motion vector 117 based on a comparison of high-resolution low-frequency image 113 (e.g., upscaled current image) and previous high-resolution low-frequency image (e.g., upscaled previous image), as described with reference to FIG4. In another instance, motion estimator 106 generates motion vector 117 based on a comparison of low-resolution image 107 and previous low-resolution image (e.g., previous image frame), as further described with reference to FIG5. Generating motion vector 117 based on low-resolution image can use fewer resources (e.g., time and computation cycles). The use of motion vector 117 to indicate motion information is provided as an illustrative example. In some instances, other types of data can be used to indicate motion information.

[0054] The motion estimator 106 provides the motion vector 117 to the motion-compensated adjuster 108. In some implementations, the motion-compensated adjuster 108 determines whether to generate an adjusted augmented image 123 based on the motion vector 117. For example, the motion-compensated adjuster 108 determines a motion metric based on the motion vector 117. The motion analyzer 112 avoids generating the adjusted augmented image 123 in response to a determination that the motion metric indicates motion that fails to meet the motion criteria, and outputs the augmented image 111 as an output image in the output stream. Alternatively, the motion-compensated adjuster 108 continues to generate the adjusted augmented image 123 in response to a determination that the motion metric indicates motion that meets the motion criteria. In a particular case, the motion-compensated adjuster 106 determines that the motion meets the motion criteria in response to a determination that the motion is greater than or equal to a first motion threshold, less than or equal to a second motion threshold, or both.

[0055] The motion-compensated adjuster 108 adjusts at least a portion of the high-frequency image component 115 of the enhanced image 111 based on a motion compensation operation to produce an adjusted high-frequency image component 121, as further described with reference to Figures 4-5. For example, the motion-compensated adjuster 108 generates a predicted high-frequency image component by performing a motion compensation operation on a previous high-frequency image component, and adjusts at least a portion of the high-frequency image component 115 based on the predicted high-frequency image component to produce the adjusted high-frequency image component 121.

[0056] In one particular instance, the motion-compensated adjuster 108 uses a temporal filter to adjust at least a portion of the high-frequency image component 115 based on the predicted high-frequency image component, as further described with reference to Figures 4-5. In another particular case, the motion-compensated adjuster 108 performs temporal filtering on the high-frequency image component 115 (e.g., added texture) and the predicted high-frequency image component based on the estimated accuracy of the motion vector 117, as further described with reference to Figures 4-5. For example, the motion analyzer 112 applies motion compensation to a previous image frame based on the motion vector 117 to generate a predicted image frame, as further described with reference to Figures 4-5. The motion analyzer 112 determines the accuracy of the motion vector 117 (e.g., indicated by a mixing factor) based on a comparison between the predicted image frame and the current image frame (e.g., low-resolution image 107 or enhanced image 111). For example, a closer match between the predicted image frame and the current image frame indicates higher accuracy of the motion vector 117. Motion analyzer 112 applies temporal filtering to generate an adjusted high-frequency image component 121 based on the accuracy of motion vector 117, the predicted high-frequency image component, and high-frequency image component 115 (e.g., added texture). For example, when motion vector 117 is more accurate, the adjusted high-frequency image component 121 corresponds more closely to the predicted high-frequency image component. Alternatively, when motion vector 117 is less accurate, the adjusted high-frequency image component 121 corresponds more closely to the high-frequency image component 115. Therefore, motion analyzer 112 filters (e.g., reduces or removes) temporally inconsistent added texture or noise to generate the adjusted high-frequency image component 121.

[0057] In some formats, in a replay time series of multiple images, a previous image frame (e.g., a previous low-resolution image) may precede the current image frame (e.g., low-resolution image 107). In another format, in a replay time series of multiple images, a previous image frame may follow the current image frame. In a particular format, a previous image frame represents an image frame most recently processed by image intensifier 154 before processing the current image frame (e.g., low-resolution image 107). Combiner 114 combines the adjusted high-frequency image component 121 (e.g., adjusted added texture) and the high-resolution low-frequency image 113 of enhanced image 111 to produce an adjusted enhanced image 123.

[0058] Generating the adjusted high-frequency image component 121 based on a single blending factor generated by a frame-by-frame comparison (e.g., between a predicted image frame and the current image frame) is provided as an illustrative example. In other examples, the motion analyzer 112 determines the accuracy of the motion vector 117 (e.g., as indicated by a plurality of blending factors) based on a comparison of a portion of a predicted image frame with a corresponding portion of the current image frame (e.g., low-resolution image 107 or enhanced image 111). For example, the motion analyzer 112 determines a first blending factor based on a comparison of a first portion of a predicted image frame with a corresponding first portion of the current image frame, a second blending factor based on a comparison of a second portion of a predicted image frame with a corresponding second portion of the current frame, and so on. The motion analyzer 112 applies temporal filtering to generate a first portion of the adjusted high-frequency image component 121 based on the first blending factor, a first portion of the predicted high-frequency image component, and a first portion of the high-frequency image component 115 (e.g., an added texture). Similarly, motion analyzer 112 applies temporal filtering to generate a second portion of the adjusted high-frequency image component 121 based on a second mixing factor, a second portion of the predicted high-frequency image component, and a second portion of the high-frequency image component 115 (e.g., the added texture). Therefore, the various portions of the adjusted high-frequency image component 121 can be temporally filtered based on different mixing factors corresponding to the accuracy of the local motion vectors.

[0059] In a particular state, the adjusted enhanced image 123 thus retains any added texture generated by the neural network 156 in the low-frequency portion of the enhanced image 111 (e.g., high-resolution low-frequency image 113) and includes motion-adjusted added texture for the high-frequency portion of the adjusted enhanced image 123 (e.g., adjusted high-frequency image component 121) to reduce signal interference relative to the previously adjusted enhanced image generated by the image intensifier 154.

[0060] In one particular configuration, the image intensifier 154 is configured to generate output data 171 based on the adjusted enhanced image 123, and to provide the output data 171 to the device 160 via a modem 170, as further described with reference to FIG6. In one particular configuration, the image intensifier 154 provides the adjusted enhanced image 123 to the display device 104, and the display device 104 displays the output image based on the adjusted enhanced image 123. In one particular configuration, the image intensifier 154 stores the adjusted enhanced image 123 in memory or a storage device.

[0061] System 100 reduces signal interference while preserving temporally consistent enhancements (e.g., added textures) generated by the neural network. For example, the high-frequency portion of the enhanced image is adjusted to remove temporally inconsistent enhancements or noise, while the low-frequency portion remains unchanged to avoid adding unnecessary temporal filtering artifacts in the low-frequency portion.

[0062] Figures 2A-2C illustrate illustrative examples of various implementations of the motion-compensated image intensifier 154 of Figure 1. In Figure 2A, a schematic diagram 200 of image intensifier 154A is shown. In Figure 2B, a schematic diagram 250 of image intensifier 154B is shown. In Figure 2C, a schematic diagram 270 of image intensifier 154C is shown. The three implementations of image intensifier 154 are provided as illustrative and non-limiting examples.

[0063] Referring to FIG. 2A, the image intensifier 154A includes an image generator 152A coupled to the motion analyzer 112A. In a particular configuration, the image generator 152A corresponds to the implementation of the image generator 152 of FIG. 1, while the motion analyzer 112A corresponds to the implementation of the motion analyzer 112 of FIG. 1.

[0064] Image generator 152A processes low-resolution image 107B (e.g., current image frame) and low-resolution image 107A (e.g., previous image frame) to produce output 290 including: enhanced image 111 (e.g., current image frame upscaled and with added texture), high-frequency image component 115 (e.g., added texture), high-resolution low-frequency image 113A (e.g., current image frame upscaled), and high-resolution low-frequency image 113B (e.g., previous image frame upscaled), as described with reference to Figures 3A-3B. In a particular case, low-resolution image 107B corresponds to low-resolution image 107 of Figure 1, and high-resolution low-frequency image 113B corresponds to high-resolution low-frequency image 113 of Figure 1.

[0065] Motion analyzer 112A generates an adjusted enhanced image 123B (e.g., where added textures are adjusted) based on output 290, as further described with reference to FIG. 4. For example, motion analyzer 112A determines motion data (e.g., motion vector 117 of FIG. 1) based on performing motion analysis on high-resolution low-frequency image 113A (e.g., upscaled current image frame) and high-resolution low-frequency image 113B (e.g., upscaled previous image frame). Motion analyzer 112A adjusts high-frequency image component 115 (e.g., added texture) at least in part based on motion data to generate adjusted enhanced image 123B. For example, motion analyzer 112A generates predicted high-frequency image components by performing motion compensation on previous high-frequency image components based on motion data, and adjusts high-frequency image component 115 based on predicted high-frequency image components to generate adjusted enhanced image 123B, as further described with reference to FIG. 4. In one particular state, the adjusted enhanced image 123B corresponds to the adjusted enhanced image 123 in FIG1. ​​In one particular state, the motion analyzer 112A generates the adjusted enhanced image 123B independently of accessing the low-resolution images 107A and 107B.

[0066] Referring to FIG. 2B, the image intensifier 154B includes an image generator 152A coupled to the motion analyzer 112B. In a particular configuration, the motion analyzer 112B corresponds to the implementation of the motion analyzer 112 of FIG. 1.

[0067] Image generator 152A generates output 290 (e.g., high-resolution low-frequency image 113A, high-resolution low-frequency image 113B, high-frequency image component 115, or a combination thereof). In the example shown in FIG2B, the enhanced image 111 is not included in output 290. For example, motion analyzer 112B generates adjusted enhanced image 123B independently of accessing enhanced image 111.

[0068] The motion analyzer 112B generates an adjusted enhanced image 123B based on the low-resolution image 107B (e.g., the current image frame) and the low-resolution image 107A (e.g., the previous image frame), as described with reference to FIG5. Compared to motion analysis of high-resolution images (e.g., high-resolution low-frequency images 113A and 113B) performed by the motion analyzer 112A, motion analysis of low-resolution images (e.g., low-resolution images 107A and 107B) can save resources (e.g., time and computation cycles).

[0069] Referring to FIG. 2C, the image intensifier 154C includes an image generator 152B coupled to the motion analyzer 112A. In a particular configuration, the image intensifier 154C corresponds to the implementation of the image intensifier 154 of FIG. 1.

[0070] Image generator 152B generates output 290 (e.g., enhanced image 111, high-frequency image component 115, high-resolution low-frequency image 113A, high-resolution low-frequency image 113B, or a combination thereof) based on low-resolution image 107B (e.g., current image frame) and adjusted enhanced image 123A (e.g., previous output image frame), as further described with reference to FIG3C. For example, during processing of a previous image frame, image intensifier 154C processes low-resolution image 107A (e.g., previous image frame) to generate adjusted enhanced image 123A (e.g., previous output image frame). Subsequently, during processing of the current image frame, image intensifier 154C processes low-resolution image 107B (e.g., current image frame) based on adjusted enhanced image 123A (e.g., previous output image frame) to generate adjusted enhanced image 123B (e.g., current output image frame). For example, image intensifier 154C generates high-resolution low-frequency image 113A based on the adjusted enhanced image 123A (e.g., the low-frequency portion of a previously output image frame). Motion analyzer 112A performs motion analysis on the high-resolution low-frequency image 113A (e.g., the low-frequency portion of a previously output image frame) and the high-resolution low-frequency signal 113B (e.g., an upscaled current image frame). Performing motion analysis on the low-frequency portion of the previously output image frame (instead of an upscaled version of the low-resolution image 107A) can improve the reduction of signal interference between the adjusted enhanced image 123B (e.g., the current output image frame) and the adjusted enhanced image 123A (e.g., the previously output image frame).

[0071] Figures 3A and 3B illustrate an illustrative implementation of image generator 152A. Figure 3C illustrates an illustrative implementation of image generator 152B. In Figures 3A-3B, image generator 152A includes a neural network 156 (e.g., SRGAN), upscalories 304A and 304B. In Figure 3C, image generator 152B includes a neural network 156 (e.g., SRGAN), low-pass filters 306A and 306B. In Figures 3A and 3B, neural network 156 generates an enhanced image 111 (e.g., upscaled and with added texture) based on a low-resolution image 107B (e.g., the current image frame). In Figure 3B, neural network 156 generates a high-frequency image component 115 (e.g., added texture).

[0072] Referring to FIG3A, upscaler 304A upscales low-resolution image 107A (e.g., a previous image frame) to produce high-resolution low-frequency image 313A (e.g., an upscaled previous image frame). In a particular case, high-resolution low-frequency image 313A corresponds to high-resolution low-frequency image 113A of FIG2A-2B.

[0073] Upscaler 304B upscales the low-resolution image 107B (e.g., the current image frame) to produce a high-resolution low-frequency image 113B (e.g., the upscaled current image frame). Upscaler 304B and neural network 156 are coupled to combiner 308. Upscaler 304B provides the high-resolution low-frequency image 113B (e.g., the upscaled current image frame) to combiner 308. Neural network 156 provides the enhanced image 111 (e.g., the upscaled current image frame with added texture) to combiner 308. Combiner 308 generates a high-frequency image component 115 (e.g., the added texture) based on the difference between the enhanced image 111 (e.g., the upscaled current image frame with added texture) and the high-resolution low-frequency image 113B (e.g., the upscaled current image frame).

[0074] Image generator 152A generates output 290. For example, output 290 includes a high-resolution low-frequency image 313A (e.g., a scaled-up previous image frame) as high-resolution low-frequency image 113A. Output 290 also includes a high-frequency image component 115 (e.g., added texture) and a high-resolution low-frequency image 113B (e.g., a scaled-up current image frame). In some implementations, output 290 includes an enhanced image 111. In other implementations, the enhanced image 111 is not included in output 290.

[0075] Referring to FIG. 3B, upscaler 304B and neural network 156 are coupled to combiner 312. Upscaler 304B provides high-resolution low-frequency image 113B (e.g., upscaled current image frame) to combiner 312. Neural network 156 provides high-frequency image component 115 (e.g., added texture) to combiner 312.

[0076] The combiner 312 generates an enhanced image 111 (e.g., an upscaled current image frame) by combining the high-frequency image component 115 (e.g., an added texture) and the high-resolution low-frequency image 113B (e.g., an upscaled current image frame).

[0077] Image generator 152A generates output 290. For example, output 290 includes a high-resolution low-frequency image 313A (e.g., a scaled-up previous image frame) as high-resolution low-frequency image 113A. Output 290 also includes a high-frequency image component 115 (e.g., added texture) and a high-resolution low-frequency image 113B (e.g., a scaled-up current image). In some implementations, output 290 includes an enhanced image 111. In other implementations, the enhanced image 111 is not included in output 290.

[0078] Referring to FIG. 3C, image generator 152B applies a low-pass filter 306A to the adjusted enhanced image 123A (e.g., the previously output image frame) to generate an adjusted high-resolution low-frequency image 323A (e.g., the low-frequency portion of the previously output image frame). In a particular case, the adjusted high-resolution low-frequency image 323A corresponds to the high-resolution low-frequency image 113A of FIG. 2C.

[0079] Image generator 152B applies a low-pass filter 306B to enhanced image 111 (e.g., a current image frame that has been upscaled and has added texture) to generate high-resolution low-frequency image 113B (e.g., the low-frequency portion of enhanced image 111). Low-pass filter 306B and neural network 156 are coupled to combiner 310. Low-pass filter 306B provides high-resolution low-frequency image 113B (e.g., the low-frequency portion of enhanced image 111) to combiner 310. Neural network 156 provides enhanced image 111 to combiner 310. Combiner 310 generates high-frequency image component 115 (e.g., added texture) based on the difference between enhanced image 111 and high-resolution low-frequency image 113B (e.g., the low-frequency portion of enhanced image 111).

[0080] The image generator 152B generates an output 290. For example, the output 290 includes a high-resolution low-frequency image 323A (e.g., the low-frequency portion of a previously output image frame) as a high-resolution low-frequency image 113A. The output 290 also includes an enhanced image 111, a high-frequency image component 115 (e.g., added texture), and a high-resolution low-frequency image 113B (e.g., the low-frequency portion of the enhanced image 111).

[0081] Figures 4-5 illustrate illustrative examples of the implementations of motion analyzers 112A and 112B. In Figure 4, motion analyzer 112A includes a motion estimator 106A coupled to a motion compensation-based adjuster 108A. In Figure 5, motion analyzer 112B includes a motion estimator 106B coupled to the motion compensation-based adjuster 108B.

[0082] Referring to FIG4, the motion-compensated adjuster 108A is coupled to the frame buffer 410 via the combiner 114. The frame buffer 410 is coupled to the combiner 408 and the motion-compensated adjuster 108A. The combiner 408 is also coupled to the motion-compensated adjuster 108A.

[0083] The motion estimator 106A generates a motion vector 117 by performing motion analysis on the high-resolution low-frequency image 113B (e.g., corresponding to the current image frame) and the high-resolution low-frequency image 113A (e.g., corresponding to the previous image frame). For the motion analyzer 112A included in the image intensifier 154A of FIG. 2A, the high-resolution low-frequency image 113B corresponds to an upscaled version of the low-resolution image 107B (e.g., the current image frame), and the high-resolution low-frequency image 113A corresponds to an upscaled version of the low-resolution image 107A (e.g., the previous image frame), as described with reference to FIG. 3A-3B. For the motion analyzer 112A included in the image intensifier 154C of FIG. 2C, the high-resolution low-frequency image 113B corresponds to the low-frequency portion of the enhanced image 111, and the high-resolution low-frequency image 113A corresponds to the low-frequency portion of the adjusted enhanced image 123A, as described with reference to FIG. 3C.

[0084] The combiner 408 generates an adjusted high-frequency image component 121A (e.g., an adjusted added texture of the previous output image) based on the difference between the adjusted enhanced image 123A (e.g., a previously output image frame) and the high-resolution low-frequency image 113A (e.g., a scaled-up previous image or a low-frequency portion of a previously output image). In a particular pattern, the combiner 408 retrieves the adjusted enhanced image 123A from the frame buffer 410.

[0085] The motion-compensated adjuster 108A generates an adjusted high-frequency image component 121B (e.g., an adjusted added texture) based on the enhanced image 111, the high-frequency image component 115, the adjusted enhanced image 123A, the adjusted high-frequency image component 121A, the motion vector 117, or a combination thereof, as described herein. The motion-compensated adjuster 108A provides the adjusted high-frequency image component 121B to the combiner 114. The combiner 114 generates the adjusted enhanced image 123B by combining the high-resolution low-frequency image 113B (e.g., the low-frequency portion of the upscaled current image or enhanced image 111) and the adjusted high-frequency image component 121B (e.g., the adjusted added texture). For example, the adjusted enhanced image 123B corresponds to the adjusted high-frequency image component 121B (e.g., adjusted added texture) applied to the high-resolution low-frequency image 113B (e.g., the upscaled current image or the low-frequency portion of the enhanced image 111). The combiner 114 outputs the adjusted enhanced image 123B, stores the adjusted enhanced image 123B in the frame buffer 410, or performs both operations.

[0086] The motion-compensated adjuster 108A includes a motion compensator 404A coupled to a time filter 412 (e.g., a finite impulse response (FIR) filter or an infinite impulse response (IIR) filter). The motion-compensated adjuster 108A also includes a motion compensator 404B coupled to the time filter 412 via a motion analyzer 414.

[0087] Motion compensator 404A generates predicted high-frequency image component 425 by applying motion vector 117 to adjusted high-frequency image component 121A (e.g., adjusted added texture of a previous output image). In one particular case, adjusted high-frequency image component 121A corresponds to a recently generated previously adjusted enhanced image. For example, adjusted high-frequency image component 121A is based on the difference between adjusted enhanced image 123A (e.g., a frame of a previous output image) and high-resolution low-frequency image 113A (e.g., a scaled-up previous image or a low-frequency portion of a previous output image). In one particular case, image intensifier 154 generates adjusted enhanced image 123A by processing low-resolution image 107A (e.g., a previous low-resolution image). In one particular case, low-resolution image 107A is an image recently processed by image intensifier 154 before processing low-resolution image 107B (e.g., the current low-resolution image). In a particular configuration, the adjusted enhanced image 123A is an adjusted enhanced image recently generated by the image intensifier 154 prior to processing the low-resolution image 107B (e.g., the current low-resolution image).

[0088] In one particular state, in the replay sequence, low-resolution image 107A precedes low-resolution image 107B (e.g., has an earlier timestamp compared to it). In another state, in the replay sequence, low-resolution image 107A follows low-resolution image 107B (e.g., has a later timestamp compared to it). Motion compensator 404A provides the predicted high-frequency image component 425 to time filter 412.

[0089] Motion compensator 404B generates a predicted augmented image 427 by applying motion vector 117 to the adjusted augmented image 123A (e.g., a previously output image). Motion analyzer 414 generates a blending factor 429 based on a comparison of the predicted augmented image 427 and augmented image 111. For example, motion analyzer 414 determines a matching metric for a specific primitive based on a comparison of a specific primitive and one or more adjacent primitives of the predicted augmented image 427 with corresponding primitives of augmented image 111. For example, motion analyzer 414 generates a matching metric for a specific primitive based on the sum of absolute differences (SAD) between the primitive values ​​of a specific primitive window (e.g., a 5-primary-by-5 ​​primitive window centered on the specific primitive at the first primitive coordinate) of the predicted augmented image 427 and the primitive values ​​of the corresponding primitive window (e.g., a primitive window centered on the primitive at the first primitive coordinate) of augmented image 111. Motion analyzer 414 determines a blending factor 429 based on a matching metric of primitives in the enhanced image 427 and enhanced image 111 used for prediction. In one particular implementation, the blending factor 429 is set to a first value (e.g., minimum or 0) in response to a determination that the matching metric (e.g., average SAD value) is greater than a first threshold. Alternatively, motion detector 414 sets the blending factor 429 to a second value (e.g., maximum or 1) in response to a determination that the matching metric (e.g., average SAD value) is less than or equal to the first threshold and greater than or equal to the second threshold. Motion detector 414 sets the blending factor 429 to a value between the first value (e.g., minimum or 0) and the second value (e.g., maximum or 1) in response to a determination that the matching metric (e.g., average SAD value) is less than or equal to the first threshold and greater than or equal to the second threshold. For example, motion detector 414 determines the blending factor 429 by applying a linear function to the matching metric (e.g., average SAD value). In a particular state, motion detector 414 uses a filter (e.g., a Gaussian filter) to smoothly transition between a first value (e.g., a minimum value or 0) and a second value (e.g., a maximum value or 1).

[0090] The mixing factor 429 indicates the difference between the predicted enhanced image 427 and the enhanced image 111. In a given case, the mixing factor 427 corresponds to the accuracy of the motion vector 117. In some implementations, a higher value of the mixing factor 429 indicates a tighter match between the predicted enhanced image 427 and the enhanced image 111, and higher accuracy of the motion vector 117. Alternatively, a lower value of the mixing factor 429 corresponds to a larger difference between the predicted enhanced image 427 and the enhanced image 111, and lower accuracy of the motion vector 118.

[0091] The motion analyzer 414 provides a mixing factor 429 to the temporal filter 412. In some implementations, the temporal filter 412 performs IIR filtering based on the predicted high-frequency image component 425 and the high-frequency image component 115. For example, the temporal filter 412 combines the predicted high-frequency image component 425 (e.g., the predicted adjusted added texture) and the high-frequency image component 115 (e.g., the added texture) based on the mixing factor 429 to produce an adjusted high-frequency image component 121B (e.g., an adjusted added texture). For example, the temporal filter 412 determines a first weight and a second weight based on the mixing factor 429. For example, the first weight is the mixing factor 429, and the temporal filter 412 determines the second weight by subtracting the first weight from a specific number of bits (e.g., the second weight = 1 - the first weight). Temporal filter 412 applies a first weight (e.g., mixing factor 429) to the predicted high-frequency image component 425 to produce a first weighted image component, and applies a second weight (e.g., 1-mixing factor 429) to the high-frequency image component 115 to produce a second weighted image component. Temporal filter 412 combines the first weighted image component and the second weighted image component to produce an adjusted high-frequency image component 121B.

[0092] In some instances, the closer the enhanced image 111 matches the predicted enhanced image 427 (e.g., as indicated by a higher value of the mixing factor 429) (corresponding to higher accuracy of motion vector 117), the greater the weight assigned to the predicted high-frequency image component 425. Alternatively, the greater the difference between the enhanced image 111 and the predicted enhanced image 427 (e.g., as indicated by a lower value of the mixing factor 429) (corresponding to lower accuracy of motion vector 117), the greater the weight assigned to the high-frequency image component 115 (e.g., texture added by neural network 156).

[0093] Therefore, the adjusted enhanced image 123B includes a high-frequency component (e.g., an adjusted high-frequency image component 121B) that is adjusted based on motion detected in the enhanced image 111 relative to the predicted enhanced image 427. In cases of lower accuracy in indicating motion vector 117 (e.g., in cases of greater motion detected in the enhanced image 111 relative to the predicted enhanced image 427), the adjusted enhanced image 123B includes more texture added by the neural network 156. Alternatively, in cases of greater accuracy in indicating motion vector 117 (e.g., in cases of less motion detected in the enhanced image 111 relative to the predicted enhanced image 427), the adjusted enhanced image 123B includes more of the predicted high-frequency image component 425.

[0094] The generation of the adjusted high-frequency image component 121B based on a single blending factor 429 generated from frame-by-frame comparisons of (e.g., predicted enhanced image 427 and enhanced image 111) is provided as an illustrative example. In other examples, the motion analyzer 414 determines the accuracy of the motion vector 117 (e.g., as indicated by a plurality of blending factors) based on comparisons of portions of the predicted enhanced image 427 and corresponding portions of the enhanced image 111. For example, the motion analyzer 414 determines a first blending factor based on a comparison of a first portion of the predicted enhanced image 427 and a corresponding first portion of the enhanced image 111 (e.g., a first matching metric), a second blending factor based on a comparison of a second portion of the predicted enhanced image 427 and a corresponding second portion of the enhanced image 111 (e.g., a second matching metric), and so on. Temporal filter 412 applies temporal filtering to generate a first portion of the adjusted high-frequency image component 121B based on a first mixing factor, a first portion of the predicted high-frequency image component 425, and a first portion of the high-frequency image component 115 (e.g., added texture). Similarly, temporal filter 412 applies temporal filtering to generate a second portion of the adjusted high-frequency image component 121B based on a second mixing factor, a second portion of the predicted high-frequency image component 425, and a second portion of the high-frequency image component 115 (e.g., added texture). Therefore, various portions of the adjusted high-frequency image component 121B can be temporally filtered based on different mixing factors corresponding to the accuracy of the local motion vector.

[0095] The determination of a blending factor 429 (or a plurality of blending factors) based on a comparison (or a comparison of portions thereof) of the predicted enhanced image 427 and the enhanced image 111 is provided as an illustrative example. In some instances, the motion compensator 404B generates a predicted high-resolution low-frequency image by applying a motion vector 117 to the high-resolution low-frequency image 113A, and the motion analyzer 414 determines the blending factor 429 (or a plurality of blending factors) based on the predicted high-resolution low-frequency image and the high-resolution low-frequency image 113B. For example, the motion analyzer 414 determines the blending factor 429 based on a comparison of the predicted high-resolution low-frequency image and the high-resolution low-frequency image 113B. As another example, the motion analyzer 414 determines a plurality of blending factors based on a comparison of portions of the predicted high-resolution low-frequency image and corresponding portions of the high-resolution low-frequency image 113B. The time filter 412 generates the adjusted high-frequency image component 121B based on the mixing factor 429 (or multiple mixing factors).

[0096] Referring to FIG5, an illustrative example of the motion analyzer 112B included in the image intensifier 154B of FIG2B is shown. The motion estimator 106B is coupled to the frame buffer 410 via the combiner 114. The frame buffer 410 is coupled to the motion compensation-based adjuster 108B via the combiner 408.

[0097] The motion estimator 106B generates motion vector 117 based on a comparison of low-resolution image 107B (current image frame) and low-resolution image 107A (previous image frame). Since low-resolution image 107 has fewer primitives compared to high-resolution image 113, generating motion vector 117 based on motion analysis of low-resolution image 107 can save resources (e.g., time, computation cycle, memory usage, etc.) compared to generating motion vector 117 based on motion analysis of high-resolution image 113 (as described in motion estimator 106A with reference to FIG. 4).

[0098] The motion-compensated adjuster 108B is configured to generate the adjusted high-frequency image component 121B (e.g., an added texture adjusted based on motion compensation) based on the high-frequency image component 115, the adjusted high-frequency image component 121A, the motion vector 117, the low-resolution image 107A, the low-resolution image 107B, or a combination thereof, as described herein.

[0099] The motion-compensated adjuster 108B provides the adjusted high-frequency image component 121B to the combiner 114. The combiner 114 generates an adjusted enhanced image 123B by combining the high-resolution low-frequency image 113B (e.g., the upscaled current image) and the adjusted high-frequency image component 121B (e.g., the adjusted added texture). For example, the adjusted enhanced image 123B corresponds to the adjusted high-frequency image component 121B (e.g., the adjusted added texture) applied to the high-resolution low-frequency image 113B (e.g., the upscaled current image). The combiner 114 outputs the adjusted enhanced image 123B, stores the adjusted enhanced image 123B in the frame buffer 410, or performs both operations.

[0100] The motion-compensated adjuster 108B includes a motion compensator 404A coupled to a time filter 412 (e.g., an FIR filter or an IIR filter). The motion-compensated adjuster 108B also includes a motion compensator 504B coupled to the time filter 412 via a motion analyzer 512 and an upscaler 514.

[0101] The motion compensator 404A generates a predicted high-frequency image component 425 by applying a motion vector 117 to the adjusted high-frequency image component 121A (e.g., an adjusted added texture of a previously output image). The motion compensator 404A provides the predicted high-frequency image component 425 to the temporal filter 412.

[0102] The motion compensator 504B generates a predicted low-resolution image 523 by applying motion vector 117 to the low-resolution image 107A (e.g., a previous image frame). Since the low-resolution image 107A has fewer primitives compared to the adjusted enhanced image 123A, applying motion vector 117 to the low-resolution image 107A can use fewer resources (e.g., time, computation cycles, memory, etc.) compared to applying motion vector 117 to the adjusted enhanced image 123A (as described with reference to FIG. 4).

[0103] The motion analyzer 512 generates a mixing factor 525 based on a comparison of the predicted low-resolution image 523 and the low-resolution image 107B. For example, the motion analyzer 512 determines a matching metric for a specific primitive based on a comparison of a specific primitive and one or more adjacent primitives in the predicted low-resolution image 523 with their corresponding primitives in the low-resolution image 107B. For example, the motion analyzer 512 generates a matching metric for a specific primitive based on the sum of the absolute differences between the primitive values ​​of a specific primitive window (e.g., a 5-primary-by-5 ​​primitive window centered on the specific primitive at the first primitive coordinate) in the predicted low-resolution image 523 and the primitive values ​​of the corresponding primitive window (e.g., a primitive window centered on the primitive at the first primitive coordinate) in the low-resolution image 107B. The motion analyzer 512 determines the mixing factor 525 based on the matching metric of primitives in the low-resolution image 523 and the low-resolution image 107B used for prediction.

[0104] The mixing factor 525 indicates the difference between the predicted low-resolution image 523 and the low-resolution image 107B. In a particular case, the mixing factor 429 corresponds to the accuracy of the motion vector 117. In some implementations, a higher value of the mixing factor 525 indicates a tighter match between the predicted low-resolution image 523 and the low-resolution image 107B, and indicates higher accuracy of the motion vector 117. Alternatively, a lower value of the mixing factor 429 corresponds to a larger difference between the predicted enhanced image 427 and the enhanced image 111, and indicates lower accuracy of the motion vector 117.

[0105] Upscaler 514 upscales the blending factor 525 to produce a blending factor 429. For example, upscaler 514 can operate in a similar manner to upscaler 304B of Figures 3A-3B, which upscales the low-resolution image 107B with a specific upscaler factor to produce a high-resolution low-frequency image 113B. Upscaler 514 upscales the blending factor 525 based on the specific upscaler factor to produce a blending factor 429.

[0106] Upscaler 514 provides a blending factor 429 to temporal filter 412. Temporal filter 412 combines the predicted high-frequency image component 425 (e.g., the predicted adjusted added texture) and high-frequency image component 115 based on blending factor 429 to produce adjusted high-frequency image component 121B. For example, temporal filter 412 of FIG5 performs a similar operation as described with reference to temporal filter 412 of FIG4 to produce adjusted high-frequency image component 121B based on blending factor 429, wherein temporal filter 412 of FIG5 receives blending factor 429 from upscaler 514, while temporal filter 412 of FIG4 receives blending factor 429 from motion analyzer 414 of FIG4.

[0107] In some instances, the closer the low-resolution image 107B matches the predicted low-resolution image 523 (e.g., as indicated by a higher value of the mixing factor 429) (corresponding to greater accuracy of motion vector 117), the greater the weight assigned to the predicted low-resolution image 523. Alternatively, the greater the difference between the low-resolution image 107B and the predicted low-resolution image 523 (e.g., as indicated by a lower value of the mixing factor 429) (corresponding to lower accuracy of motion vector 117), the greater the weight assigned to the high-frequency image component 115 (e.g., texture added by neural network 156).

[0108] Therefore, the adjusted enhanced image 123B includes high-frequency components (e.g., adjusted high-frequency image component 121B) adjusted based on motion detected in the low-resolution image 107B relative to the predicted low-resolution image 523. In the case of lower accuracy indicating motion vector 117 (e.g., in the case of more motion detected in the low-resolution image 107B relative to the predicted low-resolution image 523), the adjusted enhanced image 123B includes more texture added by neural network 156. Alternatively, in the case of higher accuracy indicating motion vector 117 (e.g., in the case of less motion detected in the low-resolution image 107B relative to the predicted low-resolution image 523), the adjusted enhanced image 123B includes more texture added by the predicted high-frequency image component 425 (e.g., predicted adjusted added texture).

[0109] The generation of the adjusted high-frequency image component 121B based on a single blending factor 429 generated based on (e.g., a frame-by-frame comparison of the predicted low-resolution image 523 and the low-resolution image 107B) is provided as an illustrative example. In other examples, the motion analyzer 512 determines the accuracy of the motion vector 117 (e.g., as indicated by a plurality of blending factors) based on a comparison of a portion of the predicted low-resolution image 523 and a corresponding portion of the low-resolution image 107B. For example, the motion analyzer 512 determines a first blending factor based on a comparison of a first portion of the predicted low-resolution image 523 and a corresponding first portion of the low-resolution image 107B (e.g., a first matching metric), a second blending factor based on a comparison of a second portion of the predicted low-resolution image 523 and a corresponding second portion of the low-resolution image 107B (e.g., a second matching metric), and so on. Upscaler 514 determines a first upscaled mixing factor by upscaling a first mixing factor, determines a second upscaled mixing factor by upscaling a second mixing factor, and so on.

[0110] Temporal filter 412 applies temporal filtering to generate a first portion of the adjusted high-frequency image component 121B based on a first upscaled mixing factor, a first portion of the predicted high-frequency image component 425, and a first portion of the high-frequency image component 115 (e.g., added texture). Similarly, temporal filter 412 applies temporal filtering to generate a second portion of the adjusted high-frequency image component 121B based on a second upscaled mixing factor, a second portion of the predicted high-frequency image component 425, and a second portion of the high-frequency image component 115 (e.g., added texture). Therefore, each portion of the adjusted high-frequency image component 121B can be temporally filtered based on different mixing factors corresponding to the accuracy of the local motion vector.

[0111] Referring to FIG6, an illustrative configuration of a scalable video transcoder 600 is shown. In one particular configuration, the scalable video transcoder 600 is integrated into one or more processors 190 of FIG1.

[0112] In some implementations, the scalable video transcoder 600 processes the high-resolution image 601 to produce encoded low-resolution image data 605 (e.g., image-encoded data) and thinning data 619, to be transmitted as output data 171 to the device 160 of FIG. 1. In one particular case, the high-resolution image 601 is based on camera input 127 received from camera 126. In one instance, the scalable video transcoder 600 encodes one or more high-resolution images 601 from camera input 127 to stream the data to device 160.

[0113] The scalable video transcoder 600 includes a downscaler 602 coupled to an encoder 604. In a given scenario, the downscaler 602 applies downscaling to a high-resolution image 601 to produce a low-resolution image 603. The low-resolution image 603 has fewer primitives compared to the high-resolution image 601. The encoder 604 encodes the low-resolution image 603 to produce encoded low-resolution image data 605.

[0114] The scalable video transcoder 600 also generates refinement data 619. For example, refinement data 619 is configured for use by the decoder to reduce the difference between the reconstructed image (e.g., the decoded image) generated from the encoded low-resolution image data 605 and the high-resolution image 601. For example, refinement data 619 is generated based on a comparison of the high-resolution image 601 and the predicted image 651 (e.g., the difference between them), which is generated based on partial decoding of the encoded low-resolution image data 605.

[0115] The scalable video transcoder 600 includes a predictor 650 configured to generate a predicted image 651 based on encoded low-resolution image data 605. In one particular configuration, the predictor 650 includes an image enhancement predictor 640. The image enhancement predictor 640 includes a decoder 606 (e.g., a local decoder) coupled to the image enhancer 154. In one particular configuration, the image enhancer 154 corresponds to the image enhancer 154A of FIG. 2A, the image enhancer 154B of FIG. 2B, or the image enhancer 154C of FIG. 2C.

[0116] Decoder 606 decodes the encoded low-resolution image data 605 to produce low-resolution image 107 (e.g., decoded low-resolution image). Image intensifier 154 processes low-resolution image 107 to produce adjusted enhanced image 123, as described with reference to FIG1. ​​In some instances, predictor 650 is also configured to perform inter-frame prediction 610 to produce predicted image 611, perform intra-frame prediction 612 to produce predicted image 613, or both.

[0117] Predictor 650 generates prediction image 651 based on the adjusted enhanced image 123, prediction image 611, prediction image 613, or a combination thereof. For example, predictor 650 generates prediction image 613 by performing intra-frame prediction 612 based on the adjusted enhanced image 123. As another example, predictor 650 generates prediction image 611 by performing inter-frame prediction 710 based on the adjusted enhanced image 123 and the previous image 623.

[0118] Predictor 650 is coupled to combiner 690 and combiner 692. Predictor 650 provides a predicted image 651 to each of combiner 690 and combiner 692. The predicted image 651 corresponds to a reconstructed image predicted to be generated at a decoder (e.g., device 160) based on encoded low-resolution image data 605. Combiner 690 generates a thinned image 691 based on the difference between high-resolution image 601 and predicted image 651. Scalable video transcoder 600 is configured to perform a transform / quantization 616 on the thinned image 691 to generate quantized data 617. For example, scalable video transcoder 600 applies a transform to the thinned image 691 to generate a transformed thinned image, and applies quantization to the transformed thinned image to generate quantized data 617. Scalable video decoder 600 is configured to provide the quantized data 616 to entropy decoder 618 to generate thinned data 619. For example, entropy decoder 618 applies entropy decoding to quantized data 617 to produce refined data 619. In some implementations, scalable video transcoder 600 initiates the transmission of encoded low-resolution video data 605 and refined data 619 as output data 171 to device 160.

[0119] The scalable video transcoder 600 performs an inverse transform / quantization 620 on the quantized data 617 to produce a reconstructed, thinned image 621. In one particular case, the reconstructed, thinned image 621 corresponds to an estimate of the reconstructed, thinned image to be produced at the decoder (e.g., of device 160) based on the thinned data 619. The combiner 692 combines the predicted image 651 and the reconstructed, thinned image 621 to produce a reconstructed image 693. In one particular case, the reconstructed image 693 corresponds to an estimate of the reconstructed image to be produced at the decoder (e.g., of device 160) based on the encoded low-resolution image data 605 and the thinned data 619. In one particular case, the scalable video transcoder 600 applies an in-loop filter 622 to the reconstructed image 693 to produce a filtered image, and provides the filtered image as a preceding image 623 for inter-frame prediction 610 of subsequent image frames.

[0120] Therefore, the image intensifier 154 at the scalable video transcoder 600 enables the generation of a predicted image 651, which corresponds to an estimate of the reconstructed image to be generated at a decoder (e.g., of the device 160) that includes a copy of the image intensifier 154.

[0121] Referring to FIG. 7, an illustrative configuration of a scalable video decoder 700 is illustrated. In one configuration, the scalable video decoder 700 is integrated into device 160 of FIG. 1. In one configuration, device 102 includes copies of one or more components described with reference to device 160. In one configuration, device 160 includes copies of one or more components described with reference to device 102. In one configuration, each of device 102 and device 160 includes a copy of the scalable video transcoder 600 and a copy of the scalable video decoder 700.

[0122] The scalable video decoder 700 receives encoded low-resolution image data 605 and refinement data 619. For example, the scalable video decoder 700 receives encoded high-resolution image data 605 and refinement data 619 as output data 171 from the device 102 of FIG1.

[0123] The scalable video decoder 700 includes an image reconstructor 750 configured to generate a reconstructed image 751 based on encoded low-resolution image data 605. In one particular configuration, the image reconstructor 750 includes an image enhancement analyzer 740. The image enhancement analyzer 740 includes a decoder 706 coupled to the image enhancer 154. In one particular configuration, the image enhancer 154 corresponds to the image enhancer 154A of FIG. 2A, the image enhancer 154B of FIG. 2B, or the image enhancer 154C of FIG. 2C.

[0124] Decoder 706 decodes the encoded low-resolution image data 605 to produce low-resolution image 107 (e.g., reconstructed low-resolution image). Image intensifier 154 processes low-resolution image 107 to produce adjusted enhanced image 123, as described with reference to FIG1. ​​In some instances, image reconstructor 750 is also configured to perform inter-frame prediction 710 to produce predicted image 711, perform intra-frame prediction 712 to produce predicted image 713, or perform both operations.

[0125] The image reconstructor 750 generates a reconstructed image 751 based on the adjusted enhanced image 123, the predicted image 711, the predicted image 713, or a combination thereof. For example, the image reconstructor 750 generates a predicted image 713 by performing intra-frame prediction 712 based on the adjusted enhanced image 123. As another example, the image reconstructor 750 generates a predicted image 711 by performing inter-frame prediction 710 based on the adjusted enhanced image 123 and the previous image 723.

[0126] The image reconstructor 750 provides the reconstructed image 751 to the combiner 792. The scalable video decoder 700 also includes an entropy decoder 718, which decodes the refined data 619 to produce quantized data 719. The scalable image decoder 700 performs an inverse transform / quantization 720 on the quantized data 719 to produce the reconstructed refined image 721.

[0127] Combiner 792 combines the reconstructed image 751 and the reconstructed refined image 721 to produce a reconstructed image 701. In one particular state, scalable video decoder 700 outputs the reconstructed image 701 to memory, a display device, a network device, or a combination thereof. In one particular state, scalable video decoder 700 applies an in-loop filter 722 to the reconstructed image 701 to produce a filtered image, and provides the filtered image as a preceding image 723 for inter-frame prediction 710 of subsequent image frames.

[0128] Therefore, the image intensifier 154 at the scalable video transcoder 700 enables the generation of an adjusted enhanced image 123 by applying motion compensation to the high-frequency portion of the enhanced image based on the low-resolution image 107, and the generation of a reconstructed image 701 using the adjusted enhanced image 123. The application of motion compensation reduces visible flicker between the reconstructed images 701 output by the scalable video decoder 700.

[0129] Referring to FIG8, a schematic diagram 800 illustrating the illustrative state of operation of the components of the system of FIG1 is shown. The image generator 152 is configured to receive a sequence of low-resolution images 107, such as a sequence of continuously captured image frames of video, which are shown as a first frame (F1) 107A, a second frame (F2) 107B, and one or more additional frames including an Nth frame (FN) 107N (where N is an integer greater than 2).

[0130] Image generator 152 is configured to output a sequence of high-resolution low-frequency images 113, the sequence including a first high-resolution low-frequency image (L1) 113A, a second high-resolution low-frequency image (L2) 113B, and one or more additional high-resolution low-frequency images including an Nth high-resolution low-frequency image (LN) 113N. Image generator 152 is configured to output a sequence of image data sets 820, the sequence including a first image data set (set 1) 820A, a second image data set (set 2) 820B, and one or more additional image data sets including an Nth image data set (set N) 820N.

[0131] In a particular configuration, each image dataset 820 includes an enhanced image 111, a high-resolution low-frequency image 113, a high-frequency image component 115, or a combination thereof. In one instance, an image generator 152 processes a first low-resolution image (F1) 107A to generate a first image dataset (set 1) 820A, the first image dataset 820A including an enhanced image 111A, a high-resolution low-frequency image 113A, a high-frequency image component 115A, or a combination thereof.

[0132] The motion-based adjuster 840 is configured to receive a sequence of image data set 820 and generate a sequence of adjusted high-frequency image components 121, the sequence including a first adjusted high-frequency image component (H1) 121A, a second adjusted high-frequency image component (H2) 121B, and one or more additional adjusted high-frequency image components including an Nth adjusted high-frequency image component (HN) 121N. In a particular case, the motion-based adjuster 840 includes a motion estimator 106 and a motion compensation-based adjuster 108.

[0133] The combiner 114 is configured to combine a sequence of adjusted high-frequency image components 121 and a sequence of high-resolution low-frequency images 113 to produce a sequence of adjusted enhanced images 123, the sequence of adjusted enhanced images 123 including a first adjusted enhanced image (E1) 123A, a second adjusted enhanced image (E2) 123B and one or more additional adjusted enhanced images including an Nth adjusted enhanced image (EN) 123N.

[0134] During operation, image generator 152 processes a first low-resolution image (F1) 107A to generate a first image data set (set 1) 820A, and motion-based adjuster 840 processes the first image data set (set 1) 820A to generate an adjusted high-frequency image component (H1) 121A. Combiner 114 combines the first high-resolution low-frequency image (L1) 113A and the adjusted high-frequency image component (H1) 121A to generate a first adjusted enhanced image 123A. Image generator 152 processes a second low-resolution image (F2) 107B to generate a second image data set (set 2) 820B, and motion-based adjuster 840 processes the second image data set (set 2) 820B to generate a second adjusted high-frequency image component (H2) 121B. Combiner 114 combines the second high-resolution low-frequency image (L2) 113B and the second adjusted high-frequency image component (H2) 121B to produce a second adjusted enhanced image 123B. This processing continues, including image generator 152 processing the Nth low-resolution image (FN) 107N to produce the Nth image data set (set N) 820N, and motion-based adjuster 840 processing the Nth image data set (set N) 820N to produce the Nth adjusted high-frequency image component (HN) 121N. Combiner 114 combines the Nth high-resolution low-frequency image (LN) 113N and the Nth adjusted high-frequency image component (HN) 121N to produce the second adjusted enhanced image 123N. By combining the adjusted high-frequency image component 121 with the high-resolution low-frequency image 113 to produce the adjusted enhanced image 123, visible signal interference between the adjusted enhanced images 123 can be reduced.

[0135] FIG9 illustrates an implementation 900 of device 102 as an integrated circuit 902, which includes one or more processors 190. Components of the one or more processors 190 (including image intensifier 154) are integrated in the integrated circuit 902. The integrated circuit 902 also includes image input 904 (such as one or more bus interfaces) to enable receiving low-resolution image 107 for processing. The integrated circuit 902 also includes image output 906 (such as a bus interface) to enable transmitting adjusted enhanced image 123. The integrated circuit 902 enables motion compensation for neural network-enhanced images to be implemented as a component in a system, such as a mobile phone or tablet device illustrated in FIG10, a wearable electronic device illustrated in FIG11, a camera illustrated in FIG12, a virtual reality, mixed reality or augmented reality headset illustrated in FIG13, or a vehicle illustrated in FIG14 or FIG15.

[0136] Figure 10 illustrates implementation 1000, wherein device 102 includes mobile device 1002 (such as a telephone or tablet device), as an illustrative and non-limiting example. Mobile device 1002 includes camera 126 and display device 104 (e.g., display screen). Components of one or more processors 190 (including image intensifier 154) are integrated into mobile device 1002 and are shown using dashed lines to indicate internal components that are not normally visible to the user of mobile device 1004. In a particular example, image intensifier 154 operates to apply motion compensation to neural network-enhanced images to produce adjusted enhanced images. Device 1002 also executes procedures using the adjusted enhanced images, such as launching a graphical user interface or otherwise displaying the adjusted enhanced images 123 on display device 104.

[0137] Figure 11 illustrates implementation 1100, wherein device 102 includes wearable electronic device 1102 (shown as a "smartwatch"). Image intensifier 154, display device 104 (e.g., display screen), and camera 126 are integrated into wearable electronic device 1102. In a particular example, image intensifier 154 operates to apply motion compensation to neurally enhanced images to produce adjusted enhanced images. Wearable electronic device 1102 also executes procedures using the adjusted enhanced images, such as launching a graphical user interface or otherwise displaying the adjusted enhanced images 12 on display device 104 of wearable electronic device 1102. For example, wearable electronic device 1102 may include a display screen configured to display notifications based on the adjusted enhanced images 123 generated by wearable electronic device 1102. In a particular example, wearable electronic device 1102 includes a haptic device that provides haptic notifications (e.g., vibration) in response to the generation of adjusted enhanced images 123. For example, haptic notifications can cause a user to view the wearable electronic device 1102 to see a notification of an adjusted enhanced image 123 displaying an instruction. Therefore, the wearable electronic device 1102 can alert a user with hearing impairment or a user wearing headphones to the display of the adjusted enhanced image 123.

[0138] FIG12 illustrates implementation 1200, wherein device 102 includes a portable electronic device corresponding to camera device 1202. In one particular case, camera device 1202 corresponds to camera 126 of FIG1. ​​Image intensifier 154 is included in camera device 1202. During operation, in response to a received verbal command, camera device 1202 can perform operations such as capturing low-resolution image 107 and generating adjusted enhanced image 123, as an illustrative example. As another example, camera device 1202 can capture high-resolution image and encode the high-resolution image using scalable video transcoder 600 of FIG6, and generate refinement data based on the adjusted enhanced image 123 in predictor 650. Camera device 1202 can transmit the encoded image and refinement data to a remote device.

[0139] Figure 13 illustrates implementation 1300, wherein device 102 includes a portable electronic device corresponding to a virtual reality, mixed reality, or augmented reality headset 1302. An image intensifier 154, a display device 104, a camera 126, or a combination thereof are integrated into the headset 1302.

[0140] In one particular configuration, a GPU buffer 1304, a camera buffer 1306, or both are integrated into the headset 1302. The camera buffer 1306 is configured to store images captured by the camera 126 (e.g., low-resolution image 107). The GPU buffer 1304 is configured to store images generated by the GPU of the headset 1302 (e.g., low-resolution image 107).

[0141] A visual peripheral device (e.g., display device 104) is located in front of the user's eyes to enable the display of augmented reality, mixed reality, or virtual reality images or scenes to the user when the headset 1302 is worn. In some implementations, the headset 1302 receives relatively low-resolution images (e.g., to save bandwidth, memory usage, processing requirements, or a combination thereof) and performs motion-adjusted neural network-based image enhancement to provide an improved user experience to the wearer of the headset 1302. In a particular instance, the visual peripheral device is configured to generate and display adjusted enhanced images 123.

[0142] Figure 14 illustrates implementation 1400, in which device 102 corresponds to or is integrated within vehicle 1402 (shown as a manned or unmanned aerial device, such as a package delivery drone). Image intensifier 154, display device 104 (e.g., display screen), camera 126, or combinations thereof are integrated into vehicle 1402. In one particular configuration, vehicle 1402 may generate and display adjusted enhanced images 123, such as assembly instructions for the delivered items.

[0143] Figure 15 illustrates another implementation 1500, in which device 102 corresponds to or is integrated into vehicle 1502 (shown as an automobile). Vehicle 1502 includes one or more processors 190, which include image intensifier 154. Vehicle 1502 also includes display device 104, camera 126, or both. In some implementations, user voice activity detection may be performed based on audio signals received from one or more microphones (e.g., for voice commands from authorized passengers). In one particular implementation, in response to receiving a verbal command identified as user voice, a voice-activated system initiates one or more operations of vehicle 1502 based on one or more detected keywords (e.g., "play a movie," "show the weather forecast," or another voice command), such as by generating and providing adjusted enhanced image 123 via display device 104.

[0144] Referring to FIG16, a specific implementation of a method 1600 for motion compensation of neural network-enhanced images is illustrated. In a particular case, one or more operations of method 1600 are performed by at least one of the following: the image generator 152 of FIG1, the motion analyzer 112, the motion estimator 106, the motion compensation-based adjuster 108, the combiner 114, the image intensifier 154, one or more processors 190, the device 102, the system 100, or a combination thereof.

[0145] At 1602, method 1600 includes: applying an enhancement neural network to the first image to generate an enhanced image. For example, the image generator 152 of FIG1 applies neural network 156 to the low-resolution image 107 to generate an enhanced image 111, as described with reference to FIG1.

[0146] At 1604, method 1600 also includes: adjusting at least a portion of the high-frequency components of the enhanced image based on motion compensation operations to generate adjusted high-frequency image components. For example, the motion compensation-based adjuster 108 of FIG1 generates a predicted high-frequency image component 425 by performing a motion compensation operation on a previous high-frequency image component (e.g., adjusted high-frequency image component 121A), and adjusts at least the high-frequency image component 115 of the enhanced image 111 based on the predicted high-frequency image component 425 to generate adjusted high-frequency image component 121, as described with reference to FIG1 and 4-5.

[0147] At 1606, method 1600 also includes combining the low-frequency component of the enhanced image and the adjusted high-frequency image component to produce an adjusted enhanced image. For example, combiner 114 combines the high-resolution low-frequency image 113 of the enhanced image 111 with the adjusted high-frequency image component 121 to produce an adjusted enhanced image 123, as described with reference to FIG1. ​​Method 1600 reduces signal interference by adjusting the neural network-enhanced image based on motion compensation.

[0148] The method 1600 of Figure 16 can be implemented by: a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a controller, another hardware device, a firmware device, or any combination thereof. As an example, the method 1600 of Figure 16 can be executed by a processor that executes instructions, as described with reference to Figure 17.

[0149] Referring to FIG. 17, a block diagram illustrating a particular illustrative implementation of the device is shown, and is designated as 1700 overall. In various implementations, device 1700 may have more or fewer components than shown in FIG. 17. In one illustrative implementation, device 1700 may correspond to device 102. In one illustrative implementation, device 1700 may perform one or more operations described with reference to FIG. 1-16.

[0150] In one particular implementation, device 1700 includes a processor 1706 (e.g., a central processing unit (CPU)). Device 1700 may include one or more additional processors 1710 (e.g., one or more DSPs). In one particular example, one or more processors 190 of FIG. 1 correspond to processor 1706, processor 1710, or a combination thereof. Processor 1710 may include a speech and music transcoder (CODEC) 1708, which includes a speech decoder ("vocoder") encoder 1736, a vocoder decoder 1738, or both. Processor 1710 may include an image intensifier 154.

[0151] Device 1700 may include memory 1786 and CODEC 1734. Memory 1786 may include instructions 1756 that can be executed by one or more additional processors 1710 (or processor 1706) to implement the functions described in reference image intensifier 154. Device 1700 may include a modem 170 coupled to antenna 1752 via transceiver 1750.

[0152] Device 1700 may include a display device 104 coupled to display controller 1726. In one particular embodiment, device 1700 may be coupled to camera 126. Speaker 1792 and microphone 1790 may be coupled to CODEC 1734. CODEC 1734 may include digital-to-analog converter (DAC) 1702, analog-to-digital converter (ADC) 1704, or both. In one particular implementation, CODEC 1734 may receive an analog signal from microphone 1790, convert the analog signal to a digital signal using analog-to-digital converter 1704, and provide the digital signal to voice and music transcoder 1708. Voice and music transcoder 1708 may process the digital signal. In one particular implementation, voice and music transcoder 1708 may provide the digital signal to CODEC 1734. The CODEC 1734 can use a digital-to-analog converter 1702 to convert digital signals into analog signals, and can provide the analog signals to the speaker 1792.

[0153] In one particular implementation, device 1700 may be included in a system-in-package or system-on-a-chip device 1722. In one particular implementation, memory 1786, processor 1706, processor 1710, display controller 1726, CODEC 1734, and modem 170 are included in a system-in-package or system-on-a-chip device 1722. In one particular implementation, input device 1730 and power supply 1744 are coupled to system-on-a-chip device 1722. Furthermore, in one particular implementation, as shown in FIG17, display device 104, camera 126, input device 1730, speaker 1792, microphone 1790, antenna 1752, and power supply 1744 are located external to system-on-a-chip device 1722. In one particular implementation, each of the display device 104, camera 126, input device 1730, speaker 1792, microphone 1790, antenna 1752, and power supply 1744 can be coupled to components of the system-on-chip device 1722, such as an interface or controller.

[0154] Device 1700 may include smart speakers, soundbars, mobile communication devices, smartphones, cellular phones, laptops, computers, tablets, personal digital assistants, display devices, televisions, game consoles, music players, radio units, digital video players, digital video disc (DVD) players, tuners, cameras, navigation devices, vehicles, headsets, augmented reality headsets, virtual reality headsets, virtual reality headsets, aircraft, home automation systems, voice-activated devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, mixed reality (MR) devices, augmented reality (AR) devices, base stations, mobile devices, or any combination thereof.

[0155] In conjunction with the described implementation, an apparatus includes: a unit for applying a neural network to a first image to generate an enhanced image. For example, the unit for applying the neural network may correspond to the image generator 152, image intensifier 154, one or more processors 190, device 102, system 100 of FIG. 1, image generator 152A, image intensifier 154A of FIG. 2A, image intensifier 154B of FIG. 2B, image generator 152B of FIG. 2C, image enhancement predictor 640, predictor 650, scalable video transcoder 600 of FIG. 6, image enhancement analyzer 740, image reconstructor 750, scalable image decoder 700 of FIG. 7, processor 1710, processor 1706, device 1700, one or more other circuits or components configured to apply a neural network to the first image to generate an enhanced image, or any combination thereof.

[0156] The device also includes a unit for adjusting at least a portion of the high-frequency components of the enhanced image based on motion compensation operation to generate adjusted high-frequency image components. For example, a unit for adjusting at least a portion of the high-frequency components of an enhanced image may correspond to the motion-compensated adjuster 108, motion analyzer 112, image intensifier 154, one or more processors 190, device 102, system 100 of FIG. 1, motion analyzer 112A, image intensifier 154A of FIG. 2A, motion analyzer 112B, image intensifier 154B of FIG. 2B, image intensifier 154C of FIG. 2C, motion-compensated adjuster 108A, time filter 412 of FIG. 4, motion-compensated adjuster 108B of FIG. 5, image intensifier predictor 640, predictor 650, scalable video transcoder 600 of FIG. 6, image intensifier analyzer 740, image reconstructor 750, scalable video decoder 700 of FIG. 7, processor 1710, processor 1706, device 1700, one or more other circuits or components configured to adjust at least a portion of the high-frequency components of an enhanced image, or any combination thereof.

[0157] The apparatus also includes a unit for combining the low-frequency component of the enhanced image and the adjusted high-frequency image component to produce the adjusted enhanced image. For example, the unit for combining may correspond to the combiner 114, motion analyzer 112, image intensifier 154, one or more processors 190, device 102, system 100, motion analyzer 112A and image intensifier 154A of FIG. 2A, motion analyzer 112B and image intensifier 154B of FIG. 2B, image intensifier 154C of FIG. 2C, image intensifier predictor 640, predictor 650, scalable video transcoder 600 of FIG. 6, image intensifier analyzer 740, image reconstructor 750, scalable video decoder 700 of FIG. 7, processor 1710, processor 1706, device 1700, one or more other circuits or components configured to combine the low-frequency component of the enhanced image and the adjusted high-frequency image component, or any combination thereof.

[0158] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 1786) includes instructions (e.g., instructions 1756) that, when executed by one or more processors (e.g., one or more processors 1710 or processor 1706), cause one or more processors to apply a neural network (e.g., neural network 156) to a first image (e.g., low-resolution image 107) to generate an enhanced image (e.g., enhanced image 111). When executed by one or more processors, the instructions also cause one or more processors to adjust at least a portion of the high-frequency components of the enhanced image (e.g., high-frequency image component 115) based on motion compensation operations to generate an adjusted high-frequency image component (e.g., adjusted high-frequency image component 121). When executed by one or more processors, the instructions also cause one or more processors to combine the low-frequency components of the enhanced image (e.g., high-resolution low-frequency image 113) and the adjusted high-frequency image component to generate an adjusted enhanced image (e.g., adjusted enhanced image 123).

[0159] The following set of related clauses describes a particular aspect of the case:

[0160] According to Clause 1, an apparatus includes: a memory configured to store instructions; and one or more processors configured to execute the instructions to perform the following operations: applying a neural network to a first image to generate an enhanced image; adjusting at least a portion of a high-frequency component of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component; and combining a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0161] Clause 2 includes the apparatus pursuant to Clause 1, wherein the one or more processors are configured to execute the instructions to perform the following operation: based on a previous image, using a time filter to adjust at least a portion of the high-frequency component of the enhanced image.

[0162] Clause 3 includes the apparatus pursuant to Clause 2, wherein during the processing of the first image, the motion compensation operation and the temporal filter operate based on the recently generated previously adjusted enhanced image.

[0163] Clause 4 includes the device according to any one of Clauses 1 to 3, wherein the one or more processors are also configured to execute the instructions to perform the following operations: generating a low-resolution image of an input image; encoding the low-resolution image to generate image encoding data; and partially decoding the image encoding data to generate the first image, the first image corresponding to the decoded low-resolution image.

[0164] Clause 5 includes the device pursuant to Clause 4, wherein the one or more processors are also configured to execute the instructions to: generate refinement data based on a comparison between the adjusted enhanced image and the input image; and initiate the transmission of the image encoding data and the refinement data to the second device.

[0165] Clause 6 includes the apparatus pursuant to Clause 5, wherein the one or more processors are configured to execute the instructions to perform the following operations: generating a thinned image based on the comparison between the adjusted enhanced image and the input image; applying a transform to the thinned image to generate a transformed thinned image; applying quantization to the transformed thinned image to generate quantized thinned data; and applying entropy decoding to the quantized thinned data to generate the thinned data.

[0166] Clause 7 includes the device pursuant to any one of Clauses 1 to 3, wherein the one or more processors are also configured to execute the instructions to perform the following operations: decoding image encoded data to generate the first image, the first image corresponding to the decoded low-resolution image.

[0167] Clause 8 includes the equipment pursuant to Clause 7, and also includes: a modem configured to receive a bitstream including the image encoded data.

[0168] Clause 9 includes the device pursuant to any one of Clauses 1 to 8, wherein the one or more processors are also configured to execute the instructions to perform the following operations: apply refinement data to the adjusted enhanced image to produce a refined enhanced image.

[0169] Clause 10 includes the equipment pursuant to Clause 9, and also includes: a modem configured to receive a bitstream including the refined data.

[0170] Clause 11 includes the device pursuant to any one of Clauses 1 to 10, and also includes: a display device configured to display an output image based on the adjusted enhanced image.

[0171] Clause 12 includes the device pursuant to Clause 11, wherein the display device includes a mixed reality (MR) display device.

[0172] Clause 13 includes the device described in Clause 11 or Clause 12, wherein the display device includes a virtual reality (VR) display device.

[0173] Clause 14 includes the device pursuant to any one of Clauses 11 to 13, wherein the display device includes an augmented reality (AR) display device.

[0174] Clause 15 includes a device pursuant to any one of Clauses 1 to 6 or any one of Clauses 11 to 14, wherein the one or more processors are also configured to execute the instructions to perform the following operation: receiving the first image from the camera.

[0175] Clause 16 includes a device pursuant to any one of Clauses 1 to 6 or Clauses 11 to 14, wherein the one or more processors are also configured to execute the instructions to perform the following operation: receiving the first image from a graphics processing unit (GPU).

[0176] Clause 17 includes a device pursuant to any one of Clauses 1 to 6 or any one of Clauses 11 to 14, wherein the one or more processors are also configured to execute the instructions to perform the following operation: retrieve the first image from the camera buffer.

[0177] Clause 18 includes a device pursuant to any one of Clauses 1 to 6 or any one of Clauses 11 to 14, wherein the one or more processors are also configured to execute the instructions to perform the following operation: retrieving the first image from a graphics processing unit (GPU) buffer.

[0178] Clause 19 includes a device pursuant to any one of Clauses 1 to 6 or any one of Clauses 11 to 14, wherein the one or more processors are also configured to execute the instructions to retrieve the first image from a camera buffer or a graphics processing unit (GPU) buffer.

[0179] Clause 20 includes a device pursuant to any one of Clauses 1 to 6 or any one of Clauses 11 to 19, wherein the one or more processors are also configured to execute the instructions to: receive an image enhancement request from an application; and, based on the type of the application, retrieve the first image from either a camera buffer or a graphics processing unit (GPU) buffer.

[0180] Clause 21 includes the device pursuant to any one of Clauses 1 to 20, wherein the one or more processors are also configured to execute the instructions to determine whether to generate the adjusted high-frequency image component based on whether the motion associated with the first image is greater than a threshold motion.

[0181] According to Clause 22, a method includes: at a device, applying a neural network to a first image to generate an enhanced image; at the device, adjusting at least a portion of a high-frequency component of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component; and at the device, combining a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0182] Clause 23 includes the method pursuant to Clause 22, and also includes: at the device, using a time filter to adjust at least a portion of the high-frequency component of the enhanced image based on the previous image.

[0183] Clause 24 includes the method according to Clause 23, wherein during the processing of the first image, the motion compensation operation and the temporal filter operate based on the recently generated previously adjusted enhanced image.

[0184] Clause 25 includes the method according to any one of Clauses 22 to 24, and also includes: at the device, generating a low-resolution image of the input image; at the device, encoding the low-resolution image to generate image encoding data; and at the device, partially decoding the image encoding data to generate the first image, the first image corresponding to the decoded low-resolution image.

[0185] Clause 26 includes the method pursuant to Clause 25, and also includes: at the device, generating refinement data based on a comparison between the adjusted enhanced image and the input image; and initiating the transmission of the image encoding data and the refinement data from the device to the second device.

[0186] Clause 27 includes the method pursuant to Clause 26, and also includes: at the device, generating a thinned image based on the comparison between the adjusted enhanced image and the input image; at the device, applying a transformation to the thinned image to generate a transformed thinned image; at the device, applying quantization to the transformed thinned image to generate quantized thinned data; and at the device, applying entropy decoding to the quantized thinned data to generate the thinned data.

[0187] Clause 28 includes the method according to any one of Clauses 22 to 25, and also includes: at the device, decoding image encoding data to generate the first image, the first image corresponding to the decoded low-resolution image.

[0188] Clause 29 includes the method pursuant to any one of Clauses 22 to 28, and also includes: at the device, applying refinement data to the adjusted enhanced image to produce a refined enhanced image.

[0189] Clause 30 includes the method pursuant to any one of Clauses 22 to 27, and also includes: receiving the first image from the camera.

[0190] Clause 31 includes the method pursuant to any one of Clauses 22 to 27, and also includes receiving the first image from a graphics processing unit (GPU).

[0191] Clause 32 includes the method pursuant to any one of Clauses 22 to 27, and also includes retrieving the first image from the camera buffer at the device.

[0192] Clause 33 includes the method according to any one of Clauses 22 to 27, and also includes retrieving the first image from the graphics processing unit (GPU) buffer at the device.

[0193] Clause 34 includes the method pursuant to any one of Clauses 22 to 27, and also includes retrieving the first image from either a camera buffer or a graphics processing unit (GPU) buffer at the device.

[0194] Clause 35 includes the method pursuant to any one of Clauses 22 to 27, and also includes: at the device, receiving an image enhancement request from an application; and retrieving the first image from either a camera buffer or a graphics processing unit (GPU) buffer based on the type of the application.

[0195] Clause 36 includes the method pursuant to any one of Clauses 22 to 35, and also includes: determining whether to generate the adjusted high-frequency image component based on whether the motion associated with the first image is greater than a threshold motion.

[0196] Clause 37 includes the method pursuant to any one of Clauses 22 to 36, and also includes receiving a bitstream comprising the image encoded data via a modem.

[0197] Clause 38 includes the method pursuant to any one of Clauses 22 to 37, and also includes receiving a bitstream including the refined data via a modem.

[0198] Clause 39 includes the method pursuant to any one of Clauses 22 to 38, and also includes providing an output image to a display device, the output image being based on the adjusted enhanced image.

[0199] Clause 40 includes the method pursuant to Clause 39, wherein the display device includes a mixed reality (MR) display device.

[0200] Clause 41 includes the method pursuant to Clause 39 or Clause 40, wherein the display device includes a virtual reality (VR) display device.

[0201] Clause 42 includes the method according to any one of Clauses 39 to 41, wherein the display device includes an augmented reality (AR) display device.

[0202] According to Clause 43, an apparatus includes: a memory configured to store instructions; and a processor configured to execute the instructions to perform a method according to any one of Clauses 22 to 42.

[0203] According to Clause 44, a non-transitory computer-readable media storage instruction, which, when executed by a processor, causes the processor to perform the method according to any one of Clauses 22 to 42.

[0204] According to Clause 45, an apparatus includes: a unit for performing the method according to any one of Clauses 22 to 42.

[0205] According to Clause 46, a non-transitory computer-readable media storage instruction, when executed by one or more processors, causes the one or more processors to perform the following operations: apply a neural network to a first image to generate an enhanced image; adjust at least a portion of the high-frequency components of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component; and combine the low-frequency components of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0206] Clause 47 includes non-transitory computer-readable media pursuant to Clause 46, wherein when such instructions are executed by the one or more processors, the one or more processors also cause the one or more processors to perform the following operation: based on a previous image, using a time filter to adjust at least a portion of the high-frequency component of the enhanced image.

[0207] According to Clause 48, an apparatus includes: a unit for applying a neural network to a first image to generate an enhanced image; a unit for adjusting at least a portion of a high-frequency component of the enhanced image based on motion compensation operations to generate an adjusted high-frequency image component; and a unit for combining a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

[0208] Clause 49 includes the apparatus pursuant to Clause 48, wherein the application unit, the adjustment unit, and the combination unit are integrated into at least one of: portable electronic devices, automobiles, vehicles, cameras, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, mixed reality (MR) devices, augmented reality (AR) devices, or combinations thereof.

[0209] It will also be understood by those skilled in the art that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of both. The various illustrative components, blocks, configurations, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or processor-executable instructions depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art to which this invention pertains can implement the described functionality in varying ways for each specific application, and such implementation decisions shall not be construed as a departure from the scope of this invention.

[0210] The steps of the methods or algorithms described herein can be directly embodied in hardware, in a software module executed by a processor, or a combination of both. The software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compressed optical disc read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art to which this invention pertains. An exemplary storage medium is coupled to a processor, allowing the processor to read information from and write information to the storage medium. Alternatively, the memory device can be integrated into the processor. The processor and storage medium can reside in an application-specific integrated circuit (ASIC). The ASIC can reside in a computing device or a user terminal. Alternatively, the processor and storage medium can reside as separate components in a computing device or a user terminal.

[0211] A prior description of the disclosed implementations is provided to enable those skilled in the art to implement or use the disclosed forms. Various modifications to these forms will readily be apparent to those skilled in the art, and the principles defined herein can be applied to other forms without departing from the scope of this application. Therefore, this application is not intended to be limited to the forms shown herein, but is to be endowed with the broadest possible scope consistent with the principles and novel features defined by the claims below. Figure 1 is a block diagram of a specific illustrative form of a system operable according to some examples of this application to perform motion compensation for neural network-enhanced images. [Simplified Explanation of the Diagram]

[0010] Figure 2A is an illustrative schematic diagram of the image intensifier of the system of Figure 1 according to some examples of the content of this case.

[0011] Figure 2B is a schematic diagram of another illustrative example of the image intensifier of the system of Figure 1 according to some examples of the content of this case.

[0012] Figure 2C is a schematic diagram of another illustrative example of the image intensifier of the system of Figure 1 according to some examples of the content of this case.

[0013] Figure 3A is an illustrative schematic diagram of the image generator of the image intensifier of Figure 2A and the image intensifier of Figure 2B, based on some examples of the contents of this case.

[0014] Figure 3B is a schematic diagram illustrating another example of the image generator of the image intensifier of Figure 2A and the image intensifier of Figure 2B, based on some examples of the content of this case.

[0015] Figure 3C is an illustrative schematic diagram of the image generator of the image intensifier of Figure 2C, based on some examples of the content of this case.

[0016] Figure 4 is a schematic diagram illustrating the motion analyzer of the system of Figure 1, based on some examples of the content of this case.

[0017] Figure 5 is a schematic diagram illustrating another example of the motion analyzer of the system of Figure 1 based on some examples of the content of this case.

[0018] Figure 6 is an illustrative schematic diagram of an encoder operable to perform motion compensation for neural network-enhanced images, based on some examples of the content of this case.

[0019] Figure 7 is an illustrative example of a decoder operable to perform motion compensation for neural network-enhanced images, based on some examples of the content of this case.

[0020] Figure 8 is a schematic diagram illustrating the operation of the components of the system of Figure 1 according to some examples of the content of this case.

[0021] Figure 9 illustrates an example of an integrated circuit operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case.

[0022] Figure 10 is a schematic diagram of a mobile device operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case.

[0023] Figure 11 is a schematic diagram of a wearable electronic device operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case.

[0024] Figure 12 is a schematic diagram of a camera operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case.

[0025] Figure 13 is a schematic diagram of a headset (such as a virtual reality, mixed reality, or augmented reality headset) that is operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case.

[0026] Figure 14 is a schematic diagram of a first example of a vehicle operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case.

[0027] Figure 15 is a schematic diagram of a second example of a vehicle operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case.

[0028] Figure 16 is a schematic diagram of a specific implementation of a method for performing motion compensation on neural network-enhanced images, which can be performed by the device of Figure 1 according to some examples of the contents of this case.

[0029] Figure 17 is a block diagram of a specific illustrative example of a device operable to perform motion compensation for neural network-enhanced images, based on some examples of the contents of this case. [Biomaterial Storage]

[0213] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A device for motion compensation of images enhanced by neural networks, comprising: A memory configured to store an enhanced image; and one or more processors configured to: apply a neural network to a first image to generate the enhanced image; adjust at least a portion of a high-frequency component of the enhanced image based on a motion associated with the first image being greater than a motion threshold to generate an adjusted high-frequency image component; and combine a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

2. The device according to claim 1, wherein the one or more processors are configured to use a time filter based on a previous image to adjust at least a portion of the high-frequency component of the enhanced image.

3. The device according to claim 2, wherein when the one or more processors are configured to process the first image, the motion compensation operation and the temporal filter operate based on a recently generated previously adjusted enhanced image.

4. The apparatus according to claim 1, wherein the one or more processors are configured to: generate a low-resolution image of an input image; encode the low-resolution image to generate image encoding data; and partially decode the image encoding data to generate the first image, the first image corresponding to a decoded low-resolution image.

5. The device according to claim 4, wherein the one or more processors are configured to: generate refinement data based on a comparison between the adjusted enhanced image and the input image; and initiate the transmission of the image encoding data and the refinement data to a second device.

6. The apparatus according to claim 5, wherein the one or more processors are configured to: generate a thinned image based on the comparison between the adjusted enhanced image and the input image; apply a transform to the thinned image to generate a transformed thinned image; apply quantization to the transformed thinned image to generate quantized thinned data; and apply entropy encoding to the quantized thinned data to generate the thinned data.

7. The device according to claim 1, wherein the one or more processors are configured to decode image encoded data to generate the first image, the first image corresponding to a decoded low-resolution image.

8. The equipment according to claim 7 also includes: A data receiver is configured to receive a one-bit stream including the image encoded data.

9. The apparatus according to claim 1, wherein the one or more processors are configured to apply refinement data to the adjusted enhanced image to produce a refined enhanced image.

10. The equipment pursuant to claim 9 also includes: A data machine configured to receive a one-bit stream including the refined data.

11. The device according to claim 1 also includes: A display device configured to display an output image based on the adjusted enhanced image.

12. The device according to claim 11, wherein the display device includes a mixed reality (MR) display device, a virtual reality (VR) display device, an augmented reality (AR) display device, or a combination thereof.

13. The device according to claim 1, wherein the one or more processors are configured to receive the first image from a camera.

14. The device according to claim 1, wherein the one or more processors are configured to execute the instructions to receive the first image from a graphics processing unit (GPU).

15. The device according to claim 1, wherein the one or more processors are configured to execute the instructions to retrieve the first image from one of a camera buffer or a graphics processing unit (GPU) buffer.

16. The device according to request item 1, wherein the one or more processors are configured to: receive an image enhancement request from an application; and, based on a type of the application, retrieve the first image from either a camera buffer or a graphics processing unit (GPU) buffer.

17. A method for motion compensation of an image enhanced by a neural network, comprising the steps of: applying a neural network to a first image at a device to generate an enhanced image; adjusting at least a portion of a high-frequency component of the enhanced image at a device based on a motion associated with the first image being greater than a motion threshold to generate an adjusted high-frequency image component; and combining a low-frequency component of the enhanced image and the adjusted high-frequency image component at the device to generate an adjusted enhanced image.

18. The method according to claim 17 also includes the step of: at the device, adjusting at least a portion of the high-frequency component of the enhanced image based on a time filter of a previous image.

19. The method of claim 18, wherein during the processing of the first image, the motion compensation operation and the temporal filter operate based on a recently generated previously adjusted enhanced image.

20. The method according to claim 17 also includes the following steps: generating a low-resolution image of an input image at the device; encoding the low-resolution image at the device to generate image encoding data; and partially decoding the image encoding data at the device to generate the first image, the first image corresponding to a decoded low-resolution image.

21. The method according to claim 20 also includes the following steps: at the device, generating refinement data based on a comparison between the adjusted enhanced image and the input image; and initiating the transmission of the image encoding data and the refinement data from the device to a second device.

22. The method according to claim 17 also includes the step of: at the device, decoding image encoding data to generate the first image, the first image corresponding to a decoded low-resolution image.

23. The method according to claim 17 also includes the step of applying refinement data to the adjusted enhanced image at the device to produce a refined enhanced image.

24. The method according to claim 17 also includes the step of retrieving the first image from either a camera buffer or a graphics processing unit (GPU) buffer at the device.

25. The method according to claim 17 also includes the following steps: at the device, receiving an image enhancement request from an application; and retrieving the first image from either a camera buffer or a graphics processing unit (GPU) buffer based on a type of the application.

26. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: apply a neural network to a first image to generate an enhanced image; adjust at least a portion of a high-frequency component of the enhanced image based on a motion associated with the first image being greater than a motion threshold to generate an adjusted high-frequency image component; and combine a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

27. The non-transitory computer-readable medium according to claim 26, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to use a time filter based on a previous image to adjust at least a portion of the high-frequency component of the enhanced image.

28. A device for motion compensation of images enhanced by neural networks, comprising: A unit for applying a neural network to a first image to generate an enhanced image; a unit for adjusting at least a portion of a high-frequency component of the enhanced image based on a motion associated with the first image being greater than a motion threshold to generate an adjusted high-frequency image component; and a unit for combining a low-frequency component of the enhanced image and the adjusted high-frequency image component to generate an adjusted enhanced image.

29. The apparatus of claim 28, wherein the application unit, the adjustment unit, and the combination unit are integrated into at least one of: a portable electronic device, an automobile, a vehicle, a camera, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a mixed reality (MR) device, an augmented reality (AR) device, or a combination thereof.

Images