Method and system for accelerated image rendering with motion compensation

Abstract

Various systems and methods for accelerated image rendering with motion compensation are provided. In one embodiment, a method includes calculating motion between a previous image frame and a target image frame to be rendered, rendering a small image of a size smaller than a target size of the target image frame based on the calculated motion, and generating the target image frame at the target size based on the small image, the calculated motion, and a reference image frame. In this way, high quality image frames of a video stream can be produced with a reduced number of renderings per frame, thereby reducing overall processing resources dedicated to rendering and power consumption for image rendering.

Description

TECHNICAL FIELD

[0001] Embodiments of the subject matter disclosed herein relate to image rendering with motion compensation. BACKGROUND

[0002] For many years, increases in computer processing power have enabled real-time video rendering, for example for video games or certain animations, to become increasingly complex. For example, early video games featured pixelated sprites moving on a fixed background, while contemporary video games feature realistic three-dimensional scenes populated with characters. At the same time, miniaturization of processing components has enabled mobile devices such as handheld video game devices and smartphones to effectively support real-time rendering of high frame rate, high resolution video. SUMMARY

[0003] In one embodiment, a method includes calculating motion between a previous image frame and a target image frame to be rendered; rendering a small image of a target size smaller than a target size of the target image frame based on the calculated motion; and generating the target image frame at the target size based on the small image, the calculated motion, and a reference image frame. In this way, high quality image frames of a video can be produced with a reduced number of renderings per frame, thereby reducing overall processing resources dedicated to rendering and power consumption for image rendering.

[0004] It is to be understood that the above general description is intended to be illustrative only and not restrictive on the scope of the claimed subject matter as it is to be defined in the claims below. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0005] The application will be better understood and its numerous objects, features, and advantages will become more apparent to the skilled artisan when the following non-limiting embodiments are read in conjunction with the drawings, in which:

[0006] Figure 1 A high-level block diagram illustrating an example image rendering system for accelerated image rendering with motion compensation is shown in accordance with an embodiment;

[0007] Figure 2 A high-level block diagram illustrating example modules for accelerated image rendering with motion compensation is shown in accordance with an embodiment;

[0008] Figure 3 A high-level flow diagram illustrating an example method for image rendering based on estimated motion between image frames is shown in accordance with an embodiment;

[0009] Figure 4 A high-level flow diagram illustrating an example method for motion compensated image enhancement of rendered images is shown in accordance with an embodiment;

[0010] Figure 5 a block diagram illustrating example motion vectors when using an enhanced image frame as a reference frame is shown; and

[0011] Figure 6 a block diagram illustrating example motion vectors when using a full-size image frame as a reference frame is shown. DETAILED DESCRIPTION

[0012] The following description relates to various embodiments of image rendering. In particular, systems and methods for accelerated image rendering with motion compensation are provided. An image rendering system, such as Figure 1 The illustrated system can include an image processor that renders image frames for display based on rendering commands received from one or more input sources. As image rendering can be complex, particularly for games with increasingly complex environments and for display devices with high resolutions and high frame rates, the processing capabilities of image processors, such as graphics processing units (GPUs), are increased to meet the processing demands. However, the amount of power consumed by a GPU is proportional to the amount of pixels being rendered for display, and the power resources of a GPU can be constrained (e.g., in mobile devices). One approach to reducing the processing demands of a GPU while maintaining high resolutions and high frame rates can include utilizing redundant pixels between adjacent frames to reduce the number of pixels that need to be rendered. For example, as Figure 2 illustrated, motion between frames can be analyzed, and this motion can be used to determine which portions of a target frame to render, and the cumulative motion between frames can further be used to compensate for minor motion between corresponding pixels in a frame, rather than re-rendering the entire frame. One approach for accelerated image rendering, such as Figure 3 illustrated, includes rendering a full frame only when there is a scene change or complex motion in a target frame. Otherwise, as Figure 4 illustrated, a reference frame with motion compensation can be utilized to enhance the rendered partial render to generate a complete target frame. In some examples, as Figure 5 illustrated, both full-size rendered image frames (e.g., original or O-frames) and reconstructed full-size rendered frames (e.g., reconstructed or R-frames, including enhanced partial renders) can be used as reference frames, while in other examples, only full-size rendered image frames can be used as reference frames, as Figure 6 illustrated. By copying over redundant areas in image frames, the overall processing demands of a GPU can be reduced while maintaining high resolution, high frame rate rendered video for games and other rendering-based content for display.

[0013] Figure 1A high-level block diagram illustrating an example image rendering system 100 that processes output from one or more input sources 110 to render an image for display is shown. In some examples, components of the image rendering system 100 may be combined in a shared housing (not shown). In other examples, one or more components of the image rendering system 100 may be configured to be communicatively coupled to form peripheral devices of the image rendering system 100.

[0014] The image rendering system 100 includes an image processor 120 configured to receive the output of one or more input sources 110, render or generate an image based on the received output of the one or more input sources 110, and output the rendered image for display via a display device 130. The image rendering system 100 also includes a memory 124 communicatively coupled to the image processor 120 for providing local temporary and / or non-temporary storage.

[0015] Input source 110 may include one or more components related to the rendering of the game display. For example, input source 110 may include one or more user input devices, including but not limited to a keyboard, mouse, joystick, headset, microphone, touchpad, etc., configured to enable a user to provide input. Input source 110 may also include a central processing unit (CPU), memory controller, system memory, communication bus, etc. As an illustrative and non-limiting example, input source 110 may include user input devices, a bus, a CPU, and system memory, wherein the user input devices are communicatively coupled to the CPU via the bus to provide user input related to the video game, and wherein the CPU generates command packets that are sent to image processor 120 to command the rendering of an image displayed by display device 130. Typically, input source 110 provides input to image processor 120, commanding image rendering for display via display device 130.

[0016] The image processor 120 may include one or more physical devices configured to execute one or more instructions. For example, the image processor 120 may include one or more processors configured to execute software instructions. Alternatively or alternatively, the image processor 120 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The image processor 120 may be single-core or multi-core, and programs executing thereon may be configured for parallel or distributed processing. In some examples, the image processor 120 includes one or more graphics processing units (GPUs). Alternatively or alternatively, in some examples, the image processor 120 may include one or more CPUs configured to render images for display.

[0017] The image rendering system 100 also includes a memory 124, which includes both non-temporary and temporary memory for storing instructions that will be executed by the image processor 120 to perform the methods further described herein. The memory 124 may also provide buffers and / or caches for storing data related to processing and / or rendering image frames from the input source 110 to support the functionality of the image processor 120.

[0018] The image rendering system 100 also includes a display 130 communicatively coupled to the image processor 120 and configured to display one or more image frames output by the image processor 110 on the display 130. That is, the image processor 120 can present a visual representation of data held or output by the image processor 120 and / or the memory 124. The display 130 may include a display device that utilizes virtually any type of technology.

[0019] Figure 2 A high-level block diagram illustrating an example module 200 for accelerated image rendering utilizing motion compensation according to an embodiment is shown. For example, module 200 may be implemented as executable instructions in memory 124 and may be executed by image processor 120.

[0020] Module 200 includes a motion analyzer module 205, an image / motion vector (MV) rendering module 210, an upscaler module 212, and a motion-compensated image enhancement module 230. The motion-compensated image enhancement module 230 includes an uncovered area detector module 235, a pixel enhancement module 240, and a motion compensator module 245. Furthermore, the motion-compensated image enhancement module 230 is communicatively coupled to a memory 250, which, for example, may include a memory 124 for storing reference frames that can be subsequently retrieved from the memory 250 for motion-compensated image enhancement.

[0021] It should be understood that, in some examples, the module 200 described herein may be implemented and provided with instructions having a suitable structure. For example, module 200 may be implemented or created using appropriate software code written in an appropriate language, wherein the code is subsequently compiled to produce an executable program that can run on an appropriate computing system having a processor with instructions stored in memory. Alternatively, or in conjunction with an executable program, module 200 may be implemented using an appropriate mixture of hardware, firmware, or software (including software interpreted at runtime). For example, portions of module 200 may be implemented using an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), a field-programmable gate array (FPGA), or another suitable adaptable or programmable processing device. As an illustrative example, one or more of modules 200 may be implemented using a graphics processing unit (GPU), and / or by placing modules 200 implemented as ASICs and / or SoCs together to form a GPU.

[0022] The motion analyzer module 205 receives rendering commands for a frame to be rendered (e.g., the current frame or the target frame) and one or more previous frames in the image frame sequence, and outputs the calculated motion between the frame to be rendered and at least one previous frame. For example, the motion analyzer module 205 may analyze motion based on the output of one or more input sources 110. Specifically, the motion analyzer module 205 calculates the motion between a previous image frame in the image frame sequence and the current or target image frame to be rendered in that image frame sequence. The motion analyzer module 205 also determines whether the cumulative motion between frames is complex based on the determined motion between the previous and current frames. The motion analyzer module 205 may further determine whether a scene change occurs between the previous and current frames. If the motion is complex and the current frame is a scene-changing frame, the motion analyzer module 205 commands the image / MV rendering module 210 to render a full-size image frame or an original (O) frame for the current frame. That is, the motion analyzer module 205 commands the image / MV rendering module 210 to render the entire current frame. However, if the motion is not too complex and the current frame is not a scene change frame, the motion analyzer module 205 instead instructs the image / MV rendering module 210 to generate a small image with a size and / or resolution smaller than 0 frames. The small image may include partial image rendering corresponding to an uncovered area or a region of the current frame containing complex motion (e.g., only a portion of the full-size target image), a scaled-down image of the entire current frame, or a scaled-down partial image rendering. Furthermore, it should be understood that the smaller resolution of the small image indicates that fewer pixels are used to render the small image than a full-size image frame, rather than reducing the frequency response.

[0023] Image / MV rendering module 210 may include a rendering engine that responds to and generates or renders visualizations or images based on rendering commands (e.g., vertex information). As further described herein, image / MV rendering module 210 generates a full-size image (e.g., O-frame) of the current frame or at least one small image of the current frame based on motion between the current frame or target frame and the previous frame, calculated by motion analyzer module 205. When image / MV rendering module 210 generates one or more small images, image / MV rendering module 201 also generates a motion vector (MV) field of the current frame to accompany the small images. In some examples, image / MV rendering module 210 may generate the MV field at a lower resolution to reduce costs.

[0024] If the image / MV rendering module 210 generates an O-frame for the current frame, the full-size current frame is passed to the motion-compensated image enhancement module 230, which then stores the O-frame as a reference frame in the memory 250. The O-frame is also output as the current frame for display.

[0025] If the image / MV rendering module 210 generates one or more small images at a resolution lower than the desired resolution of the current frame, the amplifier module 212 upscales the one or more small images to the desired resolution. In some examples, the amplifier module 212 may include a conventional two-dimensional scaler configured to interpolate pixels to generate small images with the desired resolution. Therefore, the amplifier module 212 can be implemented as software, firmware, or hardware, configured to receive small images, upscale the small images to a larger image, and output the larger image.

[0026] One or more small images rendered at the desired resolution by the image / MV rendering module 210 and / or one or more small images rendered at a lower resolution than the desired resolution by the image / MV rendering unit 210 and subsequently magnified by the amplifier module 212 are input to the motion-compensated image enhancement module 230. In either example, the small images at the desired resolution include R frames compared to O frames.

[0027] The motion-compensated image enhancement module 230 receives both an O-frame and an R-frame from the image / MV rendering module 210 and / or the amplifier module 212. If the current frame is an O-frame, the motion-compensated image enhancement module 230 writes the O-frame as a reference frame to memory 250. If the current frame is an R-frame, the motion-compensated image enhancement module 230 enhances the R-frame using the preceding enhanced frame and / or the preceding O-frame to generate a full-size current frame. For example, except for uncovered areas, the motion-compensated image enhancement module 230 replaces low-quality pixel areas in the R-frame with high-quality pixels from the reference frame (i.e., the preceding enhanced frame and / or the preceding O-frame). The motion-compensated image enhancement module 230 stores the enhanced R-frame as a reference frame in memory 250.

[0028] To perform motion-compensated image enhancement, the uncovered region detector module 235 of the motion-compensated image enhancement module 230 detects one or more uncovered regions in one or more R frames received from the image / MV rendering module 210 and / or the amplifier module 212. For this purpose, the uncovered region detector module 235 uses the MV field received from the image / MV rendering module 210 to detect the uncovered regions. The uncovered region detector module 235 outputs the detected uncovered regions to the pixel enhancement module 240. Simultaneously, the motion compensator module 245 retrieves reference frames from the memory 250, including those from the previous O frames or previous enhanced R frames. The motion compensator module 245 uses the MV field of the current frame to perform motion compensation on the reference frames. Then, the motion compensator module 245 provides the motion-compensated reference frames to the pixel enhancement module 240.

[0029] Pixel enhancement module 240 performs pixel enhancement on R-frames based on the MV field according to uncovered area detector module 235. If magnified low-quality pixels exist, pixel enhancement module 240 merges or blends these pixels with corresponding motion-compensated pixels from motion-compensated reference frames. For regions of R-frames where there are no uncovered issues, motion-compensated pixels from motion-compensated reference frames can replace corresponding pixels in the R-frame, which may be empty because image / MV rendering module 210 does not render pixels in these regions. If there are brightness variations, pixel enhancement module 240 adjusts motion-compensated pixels to address brightness differences. For uncovered areas in R-frames, since there are no corresponding regions in O-frames, pixel enhancement module 240 retains pixels in the uncovered areas of the current frame because image / MV rendering module 210 renders these pixels in smaller images.

[0030] In addition to using MV field detection to detect uncovered areas, or as an alternative, pixel-level detection can be used to detect uncovered areas. To blend pixels based on pixel-level detection, the motion-compensated pixel enhancement module 230 can calculate the change (Delta) of a given pixel at position (i,j) between frame R and reference frame O':

[0031] Delta(i,j)=Abs(R(i,j)-Median(O′(i+y,j+x),LowPass(O′(i+y,j+x)),R)),

[0032] Wherein, the reference frame O' may contain the original O frame or the enhanced / reconstructed R frame, O'(i+y,j+x) contains the motion-compensated reference frame, and (x,y) contains the motion in the horizontal and vertical directions, respectively. Alternatively, this change can be calculated as:

[0033] Delta(i,j)=Abs(R(i,j)-Median(Max3x3(O′(i+y,j+x)),Min3x3(O′(i+y,j+x)),R(i,j)).

[0034] If the pixel's Delta is large, then pixel R(i,j) from frame R is selected. If the Delta is small, then pixel O'(i+y,j+x) from the motion-compensated reference frame is selected. The blended output may include:

[0035] Output(i,j)=O′(i+y,j+x)+Deltaadjusted,

[0036] In addition, Delta can be summed in a 5x5 pixel window, and if the summed Delta is large, the corresponding pixel of the R frame is selected.

[0037] Furthermore, as a method for upscaling a low-resolution image to full resolution before combining the upscaled image with a full-resolution reference image, the pixel enhancement module 240 can selectively combine the reference image and the upscaled image based on the upscaling method. For example, if the low-resolution image is a subsampled version of the full-resolution image and the upscaled pixels are the original pixels, the upscaled pixels may have a greater weight in the image blending than the corresponding reference pixels.

[0038] Figure 3 A high-level flowchart illustrating an example method 300 for image rendering based on estimated motion between image frames, according to an embodiment, is shown. Specifically, method 300 involves partially rendering an image and enhancing the partially rendered image using a reference frame, thereby reducing the computational cost of rendering the image. Method 300 is about... Figure 1 and Figure 2 The method 300 is described using systems and components described herein, but it should be understood that method 300 can be implemented using other systems and components without departing from the scope of this disclosure. Method 300 can be implemented as instructions in memory 124, for example, it can be executed by image processor 120 to perform the actions described herein.

[0039] Method 300 begins at 305. At 305, method 300 determines the motion between the previous frame and the current frame. For example, method 300 analyzes the motion between the previous frame and the current frame or the target frame by analyzing or evaluating the rendering commands of the target frame.

[0040] At 310, method 300 determines whether the current frame includes scene changes relative to the previous frame. For example, if the rendering command indicates that the entire scene changes relative to the previous frame in the current frame, the number of redundant pixels between frames can be minimized. If the current frame includes scene changes (“Yes”), method 300 proceeds to 320. However, if the current frame does not include scene changes (“No”), method 300 proceeds to 315.

[0041] At 315, method 300 determines whether the determined motion indicates that the current frame includes complex motion relative to the previous frame. If a threshold region across image frames or a threshold amount of motion from one region to an adjacent region is identified, the motion between frames can be determined to be complex. Such a threshold, above the threshold amount of motion, may not be precisely compensated for by motion compensation techniques, but substantial motion across a threshold region across image frames indicates that a considerable portion of the image frame, such as most of the image frame, includes the threshold amount of motion. Furthermore, large-scale motion itself is not a problem if all blocks / regions move in the same direction; however, when this motion changes, it can be considered complex.

[0042] If the current frame includes complex motion (“yes”), method 300 continues to 320. At 320, method 300 renders a full-size frame of the current frame. Therefore, if the motion in the current frame determined at 305 indicates a scene change at 310 or complex motion at 315, method 300 renders a full-size frame of the current frame. For example, as described above, image / MV rendering module 210 renders frame O for the current frame. In this way, a target frame relative to the previous frame, excluding a large number of redundant pixels, can be fully rendered. At 325, method 300 stores the full-size frame in memory, such as memory 250. Therefore, the full-size frame can be retrieved later as a reference frame for accelerating image rendering. At 330, method 300 outputs the full-size frame as the current frame. For example, the current frame can be displayed via display 130. Then, method 300 returns.

[0043] However, returning to 315, if the current frame does not contain complex motion (“No”), method 300 continues to 335. At 335, method 300 renders a small image of the current frame based on the determined motion. For example, method 300 may render a small image that includes a portion of a target frame or the current frame in which substantial motion occurs, and may not render the remaining areas of the target frame or the current frame in which no substantial motion occurs. As another example, method 300 may render a small image that includes the entire target frame but has a lower resolution. As yet another example, method 300 may render a small image that includes a portion of the current frame in which substantial motion occurs, but at a resolution lower than the desired resolution of the current frame. If a small image with a lower resolution is rendered at 335, then at 340, method 300 may optionally enlarge the small image to the desired resolution of the current frame.

[0044] Continuing at 345, method 300 retrieves a reference frame from memory, such as memory 250. The reference frame may include a preceding enhanced partial rendering (e.g., an enhanced R frame) or a preceding full-size frame (e.g., an O frame).

[0045] At 350, method 300 performs motion-compensated image enhancement based on the small image, the reference frame, and the determined motion to generate the current frame. For example, method 300 performs motion compensation on the reference frame based on the determined motion to obtain a motion-compensated reference frame. Furthermore, for example, method 300 fills empty pixels in the R frame containing the small image with corresponding pixels from the motion-compensated reference frame. Alternatively or additionally, method 300 selectively blends low-quality pixels in the R frame with relatively high-quality motion-compensated pixels from the motion-compensated reference frame. Thus, the obtained current frame may at least include the small image enhanced by blending and / or replacing pixels from the motion-compensated reference frame, wherein this enhancement may be further determined based on the determined motion (e.g., based on the MV field).

[0046] At 355, method 300 stores the current frame generated at 350 in memory. Since the current frame generated at 350 includes an enhanced R frame, in some examples, the current frame generated at 350 can be retrieved for enhancing subsequent image frames. At 360, method 300 outputs the current frame generated at 350 for display, for example, via display 130. Then, method 300 returns.

[0047] Figure 4 A high-level flowchart illustrating an example method 400 for motion-compensated image enhancement of a rendered image according to an embodiment is shown. Specifically, method 400 relates to how a motion-compensated image enhancement module, such as motion-compensated image enhancement module 230, processes an input frame to generate a current frame or a target frame. Method 400 is about... Figure 1 and Figure 2The systems and components in and Figure 3 The method described herein is 300; however, it should be understood that method 400 can be implemented using other systems, components, and methods without departing from the scope of this disclosure. Method 400 can be implemented as instructions in memory 124, for example, and can be executed by image processor 120 to perform the actions described herein.

[0048] Method 400 begins at 405. At 405, method 400 receives an input frame. For example, the input frame may include a raw O-frame or an R-frame rendered and output by the image / MV rendering module 210.

[0049] At 410, method 400 determines whether the input frame is an 0-frame or a full-size image frame. If the input frame is an 0-frame (“yes”), method 400 proceeds to 415. At 415, method 400 stores the input frame in memory to be used as a reference frame for subsequent image frames. At 420, method 400 outputs the input frame as the current frame for display. After that, method 400 returns. Therefore, if the image / MV rendering module 210 renders a full-size image frame for the current frame, no additional motion compensation and image enhancement are required, and the output image is obtained.

[0050] However, returning to 410, if the input frame is not an O frame (“No”), method 400 continues to 425. At 425, method 400 receives the motion vector field of the input frame. At 430, method 400 detects one or more uncovered regions based on the motion vector field. The input frame includes an R frame that includes at least a portion of the currently rendered frame. The uncovered regions may correspond to portions of the currently rendered frame.

[0051] At 435, method 400 retrieves a reference frame from memory. The reference frame may include a preceding O-frame or a preceding enhanced R-frame. At 440, method 400 performs motion compensation on the reference frame based on the motion vector field. At 445, method 400 stores the motion-compensated reference frame in memory.

[0052] At 450, method 400 performs pixel enhancement to generate the current frame based on the input frame, detected uncovered areas, and motion-compensated pixels from a reference frame. For example, method 400 selectively blends motion-compensated pixels from the motion-compensated reference frame with pixels from the input frame, such that low-quality or missing pixels in the input frame are blended with or replaced by motion-compensated pixels from the motion-compensated reference frame. For uncovered areas, pixel enhancement is performed to ensure appropriate brightness across the current frame, thereby eliminating potential discontinuities caused by the integration of one or more small images with pixels from the motion-compensated reference image.

[0053] At 455, method 400 stores the current frame in memory so that it can be retrieved later as a reference frame. At 460, method 400 outputs the current frame for display. Then, method 400 returns.

[0054] In some examples, the reference frame in memory may include the most recent image frame, which may include an O-frame or an enhanced R-frame. In such examples, the motion vector field generated for the R-frame indicates the motion of the R-frame relative to the previous frame. Figure 5 A block diagram illustrating example motion vectors for a sequence 500 of image frames 502 when both enhanced R-frames and O-frames are used as reference frames, according to an embodiment, is shown. As shown, the first image frame 510 in the sequence 500 of image frames 502 includes an O-frame, while the second image frame 520 includes an R-frame. Therefore, the motion vector 525 of the second image frame 520 is the motion vector between the first image frame 510 and the second image frame 520. Similarly, the third image frame 530 includes an R-frame, and the motion vector 535 of the third image frame 530 includes the motion vector between the second (enhanced) image frame 520 and the third image frame 530. The fourth image frame 540 also includes an R-frame, so the motion vector 545 of the fourth image frame 540 includes the motion vector between the third (enhanced) image frame 530 and the fourth image frame 540. The fifth image frame 550 includes an O-frame, so motion vectors are not necessary for the fifth image frame 550. However, the sixth image frame 560 includes an R frame, and the motion vector 565 generated for the sixth image frame 560 includes the motion vector between the fifth image frame 550 and the sixth image frame 560.

[0055] In other examples, the reference frame stored in memory is always the O frame. In this example, the motion vector field of the R frame can be determined based on the O frame. Figure 6A block diagram illustrating example motion vectors for a sequence 600 of image frames 602 when only an 0-frame is used as a reference frame, according to an embodiment, is shown. As shown, the first image frame 610 of the sequence 600 of image frames 602 includes an 0-frame. The second image frame 620 includes an R-frame, therefore the motion vector 625 for the second image frame 620 includes the motion vector between the first image frame 610 and the second image frame 620. However, the third image frame 630 is also an R-frame, and since the first image frame 610 is an 0-frame and the second image frame 620 is an R-frame, the motion vector 635 for the third image frame 630 includes the motion vector between the first image frame 610 and the third image frame 630. Similarly, the fourth image frame 640 is an R-frame, and since the first image frame is the most recent 0-frame in the sequence 600 of image frames 602, the motion vector 645 for the fourth image frame 640 includes a motion vector between the first image frame 610 and the fourth image frame 640. The fifth image frame 650 includes an O frame, therefore no motion vector is generated for the fifth image frame 650. The sixth image frame 660 includes an R frame, and the motion vector 665 for the sixth image frame 660 includes the motion vectors between the most recent O frame, the fifth image frame 650, and the sixth image frame 660.

[0056] Therefore, various embodiments of accelerated image rendering utilizing motion compensation are provided. In one embodiment, a method includes: calculating motion between a preceding image frame and a target image frame to be rendered; rendering a small image smaller than the target image frame in size based on the calculated motion; and generating a target image frame at the target size based on the small image, the calculated motion, and a reference image frame.

[0057] In a first example of the method, generating a target image frame at a target size includes performing motion compensation on the reference image frame based on the motion between the reference image frame and the target image frame. In a second example of the method, optionally including the first example, the reference image frame includes a preceding image frame, and the motion between the reference image frame and the target image frame includes the calculated motion. In a third example of the method, optionally including one or more of the first and second examples, generating a target image frame at a target size further includes selecting motion compensation pixels from the motion compensation reference image frame for the motion compensation of the target image frame based on the calculated motion. In a fourth example of the method, optionally including one or more of the first to third examples, generating a target image frame at a target size further includes selectively blending the motion compensation pixels from the motion compensation reference image frame with a small image in the target image frame. In a fifth example of the method, optionally including one or more of the first to fourth examples, the method further includes enlarging the small image to the target resolution of the target image frame. In a sixth example of the method, optionally including one or more of the first to fifth examples, the method further includes: determining that the target image frame includes scene changes or complex motion relative to the preceding image frame, rendering the target image frame at the target size, and not rendering the small image. In a seventh example, which optionally includes one or more of the methods in the first to sixth examples, the method further includes: determining to render a predetermined number of preceding images at a lower resolution, rendering a target image frame at a target size, and not rendering small images.

[0058] In another embodiment, a method includes: in response to a determined motion indicating a scene change or complex motion, determining a motion between a preceding image frame and a target image frame, rendering the target image frame, or otherwise rendering a small image based on the determined motion; and generating the target image frame based on the small image and a motion-compensated reference image frame.

[0059] In a first example of the method, the method further includes generating a motion-compensated reference image by applying motion compensation to the reference image frame based on motion between the target image frame and the reference image frame. In a second example of the method, which optionally includes the first example, the small image comprises a portion of the target image frame, and the method further includes filling the remaining area away from the small image with pixels from the motion-compensated reference image frame. In a third example of the method, which optionally includes one or more of the first and second examples, the method further includes selectively blending pixels of the small image with pixels of the motion-compensated reference image frame. In a fourth example of the method, which optionally includes one or more of the first to third examples, the method further includes adjusting the brightness of pixels of the motion-compensated reference image frame based on the brightness of pixels of the small image. In a fifth example of the method, which optionally includes one or more of the first to fourth examples, the method further includes storing the target image frame in memory to use as a reference image frame for at least one subsequent image frame.

[0060] In another embodiment, a system includes an image processor configured with instructions in a non-temporary memory that, when executed, cause the image processor to: calculate motion between a preceding image frame and a target image frame to be rendered; render a small image smaller than the target image frame in size based on the calculated motion; and generate a target image frame at the target size based on the small image, the calculated motion, and a reference image frame.

[0061] In a first example of the system, generating a target image frame at the target size includes performing motion compensation on the reference image frame based on the motion between the reference image frame and the target image frame. In a second example of the system, which optionally includes the first example, the reference image frame includes a preceding image frame, and the motion between the reference image frame and the target image frame includes the calculated motion. In a third example of the system, which optionally includes one or more of the first and second examples, generating a target image frame at the target size further includes selecting motion compensation pixels from the reference image frame for motion compensation of the target image frame based on the calculated motion. In a fourth example of the system, which optionally includes one or more of the first to third examples, generating a target image frame at the target size further includes selectively blending motion compensation pixels with a small image in the target image frame. In a fifth example of the system, which optionally includes one or more of the first to fourth examples, the image processor is further configured with instructions in non-temporary memory that, when executed, cause the image processor to enlarge the small image to the target resolution of the target image frame. In a sixth example of a system that optionally includes one or more of the first to fifth examples, the image processor is further configured with instructions in non-temporary memory that, when executed, cause the image processor to determine that the target image frame includes scene changes or complex motion relative to previous image frames, render the target image frame at the target size, and not render small images.

[0062] As used herein, elements or steps described in the singular and referred to by the words “a” or “an” should be understood to not exclude a plural of elements or steps unless explicitly stated otherwise. Furthermore, the phrase “one embodiment” in relation to the invention is not intended to be construed as excluding the existence of additional embodiments that also include the described features. Moreover, unless explicitly stated otherwise, embodiments that “comprise,” “contain,” or “have” elements or multiple elements possessing a particular property may include elements that do not possess that particular property. The terms “comprise” and “wherein” are used as explicit equivalents to the corresponding terms “comprise” and “wherein.” Furthermore, the terms “first,” “second,” and “third,” etc., are used merely as labels and are not intended to impose numerical requirements or a particular order of position on their objects.

[0063] This written description uses examples to disclose the invention, including the best mode, and also enables those skilled in the art to practice the invention, including making and using any device or system and implementing any combination of methods. The patentable scope of the invention is defined by the claims and may include other examples that would occur to those skilled in the art. Such other examples should be within the scope of the claims if they have structural elements that are not indistinguishable from the literal language of the claims, or if they include equivalent structural elements that are not substantially different from the literal language of the claims.

Claims

1. A method comprising: Calculate the motion between the target image frame and the previous image frame in the sequence of image frames to be rendered; Render a smaller image of the target image frame, wherein the resolution of the rendered image is lower than the target resolution of the target image frame; The rendered image is enlarged to the target resolution of the target image frame; Detect one or more uncovered regions in the reconstructed frame based on the rendered image; The brightness of pixels in a motion-compensated stored reference image frame is adjusted based on the brightness of pixels in the rendered image. as well as A target image frame is generated at the target size based on the small image, the calculated motion, and the motion-compensated previously stored reference image frame, wherein the motion-compensated previously stored reference image frame is generated by applying motion compensation to previously stored reference image frames in the image frame sequence. Wherein, outside the one or more uncovered areas, the pixels of the target image frame include pixels obtained by mixing the corresponding pixels of the rendered image with the corresponding pixels of the motion-compensated stored reference image frame, or pixels obtained by replacing the corresponding pixels of the rendered image with the corresponding pixels of the motion-compensated stored reference image frame; and Within the one or more uncovered areas, the pixels of the target image frame include the corresponding pixels of the rendered image that are retained as is after rendering.

2. The method according to claim 1, wherein, Generating a target image frame at the target size includes performing motion compensation on the previously referenced image frame based on the motion stored between the previous referenced image frame and the target image frame.

3. The method according to claim 2, wherein, The image already stored in the previous reference frame corresponds to the most recent image frame in the previous frame.

4. The method according to claim 1, wherein, Detecting the one or more uncovered areas includes detecting pixel-level differences between pixels of the rendered image and pixels of a motion-compensated stored reference image frame.

5. The method according to claim 1, wherein, The motion between the preceding image frame and the target image frame is below a motion threshold.

6. A system comprising: An image processor configured with instructions in non-temporary memory, which, when executed, cause the image processor to: Calculate the motion between the target image frame and the previous image frame in the sequence of image frames to be rendered; Render a smaller image of the target image frame, wherein the resolution of the rendered image is lower than the target resolution of the target image frame; The rendered image is enlarged to the target resolution of the target image frame; Detect one or more uncovered regions in the reconstructed frame based on the rendered image; The brightness of pixels in a motion-compensated stored reference image frame is adjusted based on the brightness of pixels in the rendered image. as well as A target image frame is generated at the target size based on the small image, the calculated motion, and the motion-compensated previously stored reference image frame, wherein the motion-compensated previously stored reference image frame is generated by applying motion compensation to previously stored reference image frames in the image frame sequence. Wherein, outside the one or more uncovered areas, the pixels of the target image frame include pixels obtained by mixing the corresponding pixels of the rendered image with the corresponding pixels of the motion-compensated stored reference image frame, or pixels obtained by replacing the corresponding pixels of the rendered image with the corresponding pixels of the motion-compensated stored reference image frame; and Within the one or more uncovered areas, the pixels of the target image frame include the corresponding pixels of the rendered image that are retained as is after rendering.

7. The system according to claim 6, wherein, Generating a target image frame at the target size includes performing motion compensation on the previously referenced image frame based on the motion stored between the previous referenced image frame and the target image frame.

8. The system according to claim 7, wherein, The image already stored in the previous reference frame corresponds to the most recent image frame in the previous frame.

9. The system according to claim 6, wherein, Detecting the one or more uncovered areas includes using motion vector fields.

10. The system according to claim 6, wherein, Detecting the one or more uncovered areas includes detecting pixel-level differences between pixels of the rendered image and pixels of a motion-compensated stored reference image frame.

Images