Method and system for enhancing frame sequence quality

Abstract

The present invention provides a method and system for enhancing the quality of a frame sequence. The method includes: a first-level circuit, upon detecting a specific system resource condition, adjusting the quality of a frame sequence to be output and outputting a frame sequence with degraded quality, wherein the degraded frame sequence quality includes at least one of uneven resolution and uneven frames per second (FPS); and an enhancement engine receiving the frame sequence from the first-level circuit and generating an enhanced frame sequence based on the frame sequence for transmission to a second-level circuit. Using this solution, the quality of the frame sequence can be enhanced.

Description

Technical Field

[0001] Embodiments of the present invention relate to image and video processing for enhancing frame quality. Background Art

[0002] Typical edge electronic devices, such as TVs, smartphones, wearable devices, portable computing devices, gaming devices, etc., have limited computing capabilities due to strict requirements on power consumption and thermal performance. Graphics rendering operations on edge devices usually result in a large amount of graphics processing unit (GPU) workload. In order to maintain the target frame rate for smooth image display, edge devices may cause high power consumption. Sometimes, due to various resource limitations, such as high computing workload limitations and power consumption limitations, the target frame rate cannot be achieved. Therefore, it is necessary to improve image processing technology to minimize the impact of resource limitations on frame quality. Summary of the Invention

[0003] In one embodiment, a method for enhancing the quality of a frame sequence is provided. The method includes a first-level circuit adjusting the quality of a frame sequence to be output and outputting a frame sequence with degraded quality when a specific condition of system resources is detected, wherein the quality degradation of the frame sequence includes at least one of uneven resolution and uneven frames per second (FPS). The specific condition of the system resources may refer to insufficient remaining system resources, wherein the specific condition of the system resources may indicate that the usage of the system resources exceeds a first threshold, or the remaining amount of the system resources is lower than a second threshold. The method also includes a boosting engine receiving a frame sequence from the first-level circuit and generating an enhanced frame sequence based on the frame sequence for transmission to a second-level circuit. The uneven resolution indicates that at least two frames in the frame sequence use different resolutions, and the uneven FPS indicates that the frame sequence includes frames obtained at different FPSs.

[0004] In another embodiment, a system is provided for enhancing the quality of a frame sequence. The system includes a first-level circuit configured to adjust the quality of a frame sequence to be output and output the frame sequence with degraded quality upon detecting a specific condition of system resources, wherein the degraded quality of the frame sequence includes at least one of uneven resolution and uneven frames per second (FPS). The system also includes an enhancement engine configured to receive the frame sequence from the first-level circuit and generate an enhanced frame sequence based on the frame sequence for transmission to a second-level circuit.

[0005] Other aspects and features will become apparent to those of ordinary skill in the art upon reading the following description of specific embodiments in conjunction with the accompanying drawings.

[0006] The present application enables the first-level circuit to reduce the quality of frames with great flexibility, thereby saving resources (eg, bandwidth resources, computing resources, and / or power resources of the first-level circuit). BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present invention is illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals indicate similar elements. It should be noted that different references to "one" or "an" embodiment in the present invention are not necessarily the same embodiment, and such references mean at least one. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it means that such feature, structure, or characteristic can be implemented in conjunction with other embodiments within the knowledge of those skilled in the art, whether or not explicitly described.

[0008] Figure 1 Several frame sequence examples are shown in accordance with some embodiments.

[0009] Figure 2 is a block diagram illustrating a system for improving the quality of a frame sequence according to one embodiment.

[0010] Figure 3 is a block diagram illustrating a system for improving the quality of a frame sequence according to another embodiment.

[0011] Figure 4 is a diagram illustrating example boosting engine operation using a previous enhancement frame as a reference frame, according to one embodiment.

[0012] Figure 5 is a diagram illustrating example upscaling engine operation using a previous frame having a higher-than-standard quality as a reference frame, according to one embodiment.

[0013] Figure 6 is a flow chart illustrating a method for systematically enhancing the quality of a frame sequence according to one embodiment.

[0014] Figure 7 An example device according to one embodiment is shown.

[0015] Figure 8 An example device according to another embodiment is shown.

[0016] Figure 9 An example system according to one embodiment is shown. DETAILED DESCRIPTION

[0017] In the following description, many specific details are set forth. However, it should be understood that embodiments of the present invention can be practiced without these specific details. In other cases, well-known circuits, structures, and techniques are not shown in detail to avoid obscuring an understanding of the present invention. However, those skilled in the art will understand that the present invention can be practiced without these specific details. Those of ordinary skill in the art will be able to implement appropriate functionality with the included description without undue experimentation.

[0018] Figure 1 Multiple frame sequence examples according to some embodiments are shown. The first stage circuit 110 sends the frame sequence to the second stage circuit 120. The quality of the frame sequence may sometimes be unstable. That is, sometimes the frame rate of the frame sequence may be lower than the target frames per second (FPS), and the resolution of some frames in the frame sequence may be lower than other frames. The first stage circuit 110 can output any frame sequence shown in the following non-limiting examples. Row (A) shows a frame sequence with uniform quality, where the first stage circuit 110 generates all frames (I0-I6) at the target FPS and target resolution. Row (B) shows an uneven FPS situation, where frames I1, I2, I4 and I6 are lost in transmission or the first stage circuit 110 does not generate (e.g., does not render) frames I1, I2, I4 and I6. In a frame sequence with an uneven FPS situation, the frame rate (also called FPS) changes dynamically without following a regular pattern.

[0019] Row (C) shows an example case with non-uniform resolution, where frames I0, I4, and I6 have a target resolution (e.g., 2400x1080), frame I3 has a resolution lower than the target resolution (e.g., 1600x720), and frames I1, I2, and I5 have the lowest resolution in the frame sequence (e.g., 600x270). In a frame sequence with non-uniform resolution, the resolutions of multiple frames change dynamically and do not follow a regular pattern. Row (D) shows a combination of non-uniform FPS and non-uniform resolution cases. Rows (B), (C), and (D) provide non-limiting examples of frame sequences with non-uniform quality cases. It will be understood that a frame sequence with non-uniform quality cases can have any combination of non-uniform FPS and non-uniform resolution, including only non-uniform FPS or only non-uniform resolution. Although the examples herein show non-uniform quality conditions, it should be understood that the first stage circuit 110 may also send a sequence of frames with uniform quality degradation, for example, every other frame in the sequence of frames is lost, for example, frames I1, I3, and I5 are lost, or each frame in the sequence of frames has the same resolution lower than the target resolution.

[0020] In the following description, a frame sequence with uneven quality may also be referred to as a frame sequence with degraded quality among a plurality of frames distributed at uneven intervals in time. In the example of row (B), the lost frames I1, I2, I4, and I6 are distributed at uneven intervals in time, where the intervals are one frame interval between I1 and I2, two frame intervals between I2 and I4, and two frame intervals between I4 and I6. Similarly, in the examples of rows (C) and (D), the lost frames and / or low-resolution frames are distributed at uneven intervals in time. Therefore, it should be understood that the term "uneven" hereinafter may be interpreted as "inconsistent across the entire frame sequence."

[0021] Due to resource limitations, first-stage circuitry 110 may determine or be requested to generate a sequence of frames with uneven quality. Non-limiting examples of resource limitations include insufficient transmission bandwidth, high computational workload limitations, power consumption limitations, etc. In one embodiment, first-stage circuitry 110 may reduce the quality of frames with low or slowly varying information content. Alternatively or additionally, first-stage circuitry 110 may reduce the quality of one or more frames when the usage of a limited resource (also referred to as a system resource) exceeds a usage threshold, for example, when power consumption exceeds a threshold. When compared to a predetermined threshold, the amount of the limited resource may be "insufficient," "high," or "low." In an alternative embodiment, the predetermined threshold is a speed threshold. When compared to the speed threshold, the comparison result may be that the transmitted data rate is lower than the speed threshold, indicating a slow transmission rate and insufficient bandwidth resources. In one embodiment, the first-stage circuitry may use a host circuit or a running background thread to monitor the usage of the limited resource. When the host circuitry or thread detects that the usage of the limited resource exceeds a first threshold, or the remaining amount of the limited resource falls below a second threshold, it notifies the first-stage circuitry to adjust the quality of the frame sequence, and the first-stage circuitry outputs a sequence of frames with degraded quality. Such quality degradation of the frame sequence includes one or both of uneven resolution and uneven FPS.Non-limiting examples of limited resources include at least one of computing resources, power resources, and transmission bandwidth.

[0022] First-stage circuit 110 and second-stage circuit 120 can be any two endpoints of a frame sequence transmission network or connection. In one embodiment, first-stage circuit 110 and second-stage circuit 120 can be located in the same electronic device, such as a graphics processing unit (GPU) and a display panel in the same device. In another embodiment, first-stage circuit 110 and second-stage circuit 120 can be located in different devices, such as a transmitter (Tx) device and a receiver (Rx) device connected via a transmission network.

[0023] The first stage circuit 110 can dynamically adjust the frame quality during rendering and / or transmission to generate a frame sequence with uneven quality. The adjustment can include temporal reduction and / or spatial reduction. Temporal reduction refers to a reduction in FPS, for example, reducing the number of frames rendered and / or transmitted per time unit. Spatial reduction refers to a reduction in frame resolution, for example, reducing the number of pixels in a rendered and / or transmitted frame. Figure 1 As shown in rows (B) and (D) of , the time reduction may be non-uniform, for example, the time interval between two directly adjacent frames in the frame sequence on the time axis may vary. Figure 1 As shown in rows (C) and (D), spatial reduction can be non-uniform, where at least two frames in the frame sequence have different resolutions, for example, low-resolution frames appear at fixed intervals or at irregular intervals. Non-uniform quality conditions in the frame sequence, such as FPS reduction and / or resolution reduction, can save a significant amount of computing resources, bandwidth, and / or power resources of the first-stage circuit 110. The first-stage circuit 110 can adjust the quality reduction with great flexibility to achieve a better balance between the compensated frame quality and resource usage.

[0024] As will be described in more detail later, the disclosed system also includes an upscaling engine to recover from a frame sequence with degraded quality. The upscaling engine can be activated on demand. In one embodiment, the upscaling engine receives a frame sequence from a first-level circuit, enhances the frame quality, and sends the quality-enhanced frame sequence to a second-level circuit. The frame sequence may have uneven quality or uniform quality. In an embodiment where the upscaling engine is co-located with a first-level circuit such as a GPU, the upscaling engine can offload rendering operations from the GPU. Offloading rendering operations can enable the system to increase FPS with acceptable power consumption. In another embodiment, the upscaling engine is located at an Rx device of a transmission network, and the upscaling engine can act as a stabilizer to stabilize the frame quality received by the Rx device.

[0025] Figure 2A block diagram of a system 100 for improving the quality of a frame sequence according to one embodiment is shown. Elements indicated by dashed lines represent those elements that may not be present in some alternative embodiments. System 100 includes a first-level circuit 110, which is coupled to a second-level circuit 120 via an electrical connection (e.g., a bus connection or a transmission network). An enhancement engine 250 is coupled to the electrical connection and is used to enhance the quality of the frame sequence sent from the first-level circuit 110 to the second-level circuit 120. The enhancement engine 250 can be a dedicated hardware circuit. Alternatively, the enhancement engine 250 can be software containing instructions executable by a processor, or a combination of hardware circuits and software instructions. The frame quality enhancement performed by the enhancement engine 250 can include super-resolution, frame interpolation and / or extrapolation, inpainting, image alignment, motion synthesis, etc.

[0026] In one embodiment, the first-stage circuit 110 includes a resolution adjustment module 112 and an FPS adjustment module 113. The first-stage circuit 110 uses the resolution adjustment module 112 to dynamically adjust frame quality to generate low-resolution frames and / or uses the FPS adjustment module 113 to change the frame rate. In one embodiment, the system 100 may also include a host processor (not shown) that controls the operation of the first-stage circuit 110. The system 100 monitors system resource usage, such as computing resource usage, power consumption, transmission bandwidth usage, etc. When resource usage reaches a limit, the first-stage circuit 110 or the host processor activates one or both of the resolution adjustment module 112 and the FPS adjustment module 113 to adjust (e.g., reduce) the output frame quality. Adjustments can be made as needed; for example, the resolution and / or frame rate can be reduced for any frame at any time interval. One or both of the resolution adjustment module 112 and the FPS adjustment module 113 can be implemented by dedicated hardware circuitry, software containing instructions executable by a processor, or a combination of hardware circuitry and software instructions. In one embodiment, the resolution adjustment module 112 and the FPS adjustment module 113 may both be part of a GPU rendering pipeline.

[0027] In one embodiment, the first stage circuit 110 may generate additional information including metadata about low-resolution frames and lost frames and send the additional information to the upscaling engine 250. For example, the first stage circuit 110 may render frame (N) but skip rendering frame (N+1) or render frame (N+1) at a low resolution. To assist the upscaling engine 250 in improving frame quality, the first stage circuit 110 may generate metadata describing attributes of frame (N+1) and send the metadata to the upscaling engine 250. The metadata may include information about any of the following for frame (N+1): depth, texture, normal, color, instance segmentation, motion vector information (e.g., optical flow), frame resolution, etc. It should be understood that the upscaling engine 250 may enhance frame quality with or without the additional information from the first stage circuit 110.

[0028] In one embodiment, the first stage circuit 110 may send a help request to the enhancement engine 250 to request enhanced frame quality. The help request may indicate a quality reduction strategy such as FPS reduction and / or resolution reduction. The help request may also provide an index of a frame with reduced quality, such as an index of a frame with reduced resolution, or an index in time of a frame that is not rendered or transmitted. For example, the help request may include a frame insertion request indicating the position of a frame that is not rendered or transmitted in a frame sequence. The enhancement engine 250 may perform motion synthesis and alignment to insert these missing frames.

[0029] In one embodiment, the lifting engine 250 includes an alignment module 252 coupled to the optimization module 253. The alignment module 252 performs temporal and spatial alignment of the images relative to their respective reference frames. The alignment module 252 performs geometric transformations, frame interpolation and / or extrapolation, as well as other post-processing including but not limited to blending. The alignment module 252 can perform the above operations using motion information emitted from the first stage circuit 110 (which can be included in the metadata). Alternatively, the alignment module 252 can include a motion synthesizer 251 to generate the motion information. The motion synthesizer 251 can extract motion information from the frames; for example, generating motion vectors and optical flow between the current frame (i.e., the frame currently being processed by the lifting engine 250) and the reference frame. The motion information can include acceleration estimates of objects in the frame.

[0030] The optimization module 253 performs frame optimization operations, including but not limited to super-resolution (SR), repair, blending, sharpening, and other image processing operations. In one embodiment, the optimization module 253 may include an artificial intelligence (AI) model that has been trained to perform the optimization operation. For example, the optimization module 253 may include an AI SR model for amplifying a low-resolution image to a higher-resolution image. The optimization module 253 may also include an AI repair model to repair an image with multiple missing pixels (e.g., a hole in the image). The output of the optimization module 253 (e.g., the quality optimizer 253) may be sent to the second stage circuit 120.

[0031] One or more of the motion synthesizer 251, alignment module 252, and optimization module 253 in the lifting engine 250 may be implemented by dedicated hardware circuits, software containing instructions executable by a processor, or a combination of both. Based on the information sent by the first-stage circuit 110, platform capabilities, and / or target output quality, the lifting engine 250 may activate one or more of the motion synthesizer 251, alignment module 252, and optimization module 253 to improve frame quality.

[0032] Figure 3 is a block diagram illustrating a system 101 for improving the quality of a frame sequence according to another embodiment. Elements indicated by dashed lines represent those elements that may not be present in some alternative embodiments. In one embodiment, improving engine 350 includes a quality detector 310 that detects the quality of frames in a frame sequence sent from first-stage circuitry 110 to second-stage circuitry 120. When quality detector 310 detects uneven quality conditions in the frame sequence, such as uneven FPS and / or uneven resolution, improving engine 350 activates alignment module 252 and optimization module 253 to improve frame quality. Upon detecting a current frame with degraded quality, improving engine 350 performs image alignment between the current frame and a reference frame, as well as frame quality optimization including at least super-resolution and / or inpainting. Using quality detector 310, requests for assistance from first-stage circuitry 110 may no longer be required.

[0033] The alignment module 252 may include a motion synthesizer 251 to generate motion information when the first stage circuit 110 does not provide motion information. Figure 2The operations of the motion synthesizer 251, the alignment module 252, and the optimization module 253 are described. In this embodiment, the output of the optimization module 253 is monitored by a quality checker 320, which compares the quality of the frame output by the optimization module 253 to a quality threshold. If the frame quality does not meet the quality threshold, the output falls back to the original frame received by the enhancement engine 350. For example, the original frame may have lost too much information, making the restored frame performed by the optimization module 253 of unacceptable quality.

[0034] In an alternative embodiment, the boost engine 250 ( Figure 2 ) and / or 350( Figure 3 ) can operate without additional information (e.g., metadata) from the first stage circuit 110 to improve frame quality. For example, the upscaling engine 250 / 350 can calculate any of the following from the received frame sequence: depth, texture, normal, color, instance segmentation, motion vector information (e.g., optical flow), frame resolution, etc. to perform motion estimation and compensation, frame interpolation / extrapolation, alignment, super-resolution, inpainting, etc.

[0035] In embodiments where the first stage circuit 110 does not provide additional information or the provided additional information does not include motion information, the motion synthesizer 510 may use information in one or more previous frames to generate optical flow for motion estimation and compensation.

[0036] Figure 4 is a diagram illustrating an example of operation of a boosting engine using a previous enhanced frame as a reference frame according to one embodiment. The previous enhanced frame is the frame immediately preceding the current frame being processed by the boosting engine. Figure 4 The top dashed box shows the Figure 2 Boost engine 250 or Figure 3 350 in ). Each of rows (A), (B) and (C) shows that the motion synthesizer 251 uses one or more previous frames in the optical flow calculation. The alignment module 252 uses the optical flow and uses the previous frame (e.g., frame (N-1)) as a reference frame to generate the current frame (e.g., frame (N)). In the example of row (A), the resolution of the low-resolution frame (e.g., I1) is improved by super-resolution. In the example of row (B), the reference frame is the frame (e.g., I1) obtained by improving the resolution of the original low-resolution frame (e.g., I1) using super-resolution. In the example of row (C), the reference frame is the missing frame (e.g., I2) obtained by extrapolating and repairing from the previous frames (e.g., I0 and I1).

[0037] Figure 5FIG2 is a diagram illustrating operation of an example boosting engine using a previous frame having a quality better than a standard quality as a reference frame according to one embodiment. The previous frame may be a frame with the best quality among all frames preceding the current frame in a frame sequence, a frame with the best quality in a segment of a frame sequence, or a frame with a quality better than a quality threshold. Figure 5 The top dashed box shows the Figure 2 Boost engine 250 or Figure 3 350). Each of rows (A), (B), and (C) illustrates the use of one or more previous frames by motion synthesizer 251 in optical flow calculations. Alignment module 252 generates a current frame (e.g., frame (N)) using the optical flow and the previous best quality frame as a reference frame. In this non-limiting example, frame I0 is the best quality frame in the sequence of frames I0-I3. Therefore, I0 is used as a reference frame to generate I1-I3 with target quality.

[0038] Figure 6 6 is a flow chart illustrating a method 600 for enhancing the quality of a frame sequence according to one embodiment. The system includes a first stage circuit and an enhancement engine, such as system 100 ( Figure 2 ) or System 101( Figure 3 ) in the first stage circuit 110 and the boost engine 250 or 350.

[0039] Method 600 begins at step 610, where the first-stage circuitry, upon detecting a specific condition of system resources, adjusts the quality of a frame sequence to be output and outputs a frame sequence with degraded quality. The degraded quality of the frame sequence may include at least one of uneven frames per second (FPS) and uneven resolution. At step 620, a boosting engine receives the frame sequence from the first-stage circuitry. At step 630, the boosting engine generates an enhanced frame sequence based on the frame sequence for transmission to the second-stage circuitry.

[0040] In one embodiment, the lifting engine can use the frame before the current frame as a reference frame to calculate the optical flow of the current frame. Alternatively, the lifting engine can use the previous frame with the best quality as a reference frame to calculate the optical flow of the current frame.

[0041] The first-stage circuitry can dynamically change frame quality during frame sequence generation in response to system resource constraints. In one embodiment, the first-stage circuitry can activate the upscaling engine by sending a help request to the upscaling engine, where the help request indicates information about frames with degraded quality. The degraded frames can include frames with a resolution lower than a target resolution or frames that were not rendered or transmitted in the frame sequence. The degraded frame information can include the index of the frame with a resolution lower than the target resolution or the position of the frame in the frame sequence that was not rendered or transmitted. In response to the help request, the upscaling engine performs image alignment between the current frame and a reference frame, as well as frame quality optimization including at least super-resolution and / or inpainting. In another embodiment, the upscaling engine can detect a current frame with degraded quality. In response to this detection, the upscaling engine performs image alignment between the current frame and a reference frame, as well as frame quality optimization including at least super-resolution and / or inpainting. After optimizing the quality of the received frame, the upscaling engine compares the optimized quality with a threshold and, if the optimized quality falls below the threshold, reverts to the quality of the received frame.

[0042] In one embodiment, the first stage circuit, the boost engine, and the second stage circuit are located in the same electronic device. In another embodiment, the first stage circuit and the second stage circuit are located in two electronic devices coupled to each other via a transmission network.

[0043] Figure 7 An example of a device 700 according to one embodiment is shown. The device 700 may be Figure 2 System 100 or Figure 3 . An example of device 700 is an electronic device with graphics processing capabilities. For example, device 700 can be a smartphone, a computing device, a network-connected device, a gaming device, an entertainment device, an Internet of Things (IoT) device, or any device capable of processing and displaying images and / or video.

[0044] The device 700 includes processing hardware 770. In one embodiment, the processing hardware 770 includes a central processing unit (CPU) 760, a GPU 710, and one or more of the following: a digital processing unit (DSP), an artificial intelligence (AI) processor, a multimedia processor, and other general and / or specialized processing circuits. In one embodiment, the GPU 710 may be the aforementioned first stage circuit 110 ( Figure 1 、 2 and 3), and the CPU 760 is the host processor of the GPU 710.

[0045] The device 700 also includes a display subsystem 780 coupled to the processing hardware 770 via a display interface circuit 740. In one embodiment, the display subsystem 780 includes a boost engine 750 and a display panel 720. The display panel 720 displays information such as images, videos, messages, games, text, graphics, etc. In one embodiment, the display panel 720 may be the aforementioned second stage circuit 120 ( Figure 1 、 2 and 3), and the boost engine 750 may be the boost engine 250 ( Figure 2 ) or 350( Figure 3 ). Upscaling engine 750 may be part of display subsystem 780. Upscaling engine 750 may be implemented in hardware or in program code running on general-purpose or dedicated processing circuitry. In one embodiment, upscaling engine 750 may include a hardware accelerator, such as an AI accelerator, to perform frame quality optimizations such as AI-based super-resolution, restoration, and other image processing operations.

[0046] The apparatus 700 also includes a memory 730 coupled to the processing hardware 770 and the display subsystem 780. The memory 730 may include memory devices such as dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, and other non-transitory machine-readable storage media, such as volatile or non-volatile storage devices. The memory 730 includes one or more buffers 735, such as a color buffer, a metadata buffer, a frame buffer, and the like. The GPU 710 may store rendered frames in a color buffer or a frame buffer, and metadata in a metadata buffer, wherein the metadata includes information about frames that have not been rendered and frames with degraded quality (e.g., low resolution). In some embodiments, the memory 730 may store instructions that, when executed by the processing hardware 770, cause the processing hardware 770 to perform Figure 6 Method 600.

[0047] In one embodiment, the CPU 760 can set the target refresh rate of the display interface circuit 740 to control the rate at which images are output from the buffer 735 to the display panel 720, and can dynamically perform refresh rate adjustments when necessary. Figure 7 The embodiments are simplified for illustrative purposes. Additional hardware components may be included. For example, device 700 may also include a network interface to connect to a wired and / or wireless network for sending and / or receiving voice, digital data, and / or media signals.

[0048] Figure 8 An example of a device 701 according to another embodiment is shown. The device 701 may be Figure 2 System 100 or Figure 3 A non-limiting example of system 101 is provided in FIG. Figure 7 and Figure 8 7. In the present invention, like reference numerals are used to refer to like components. Device 701 includes processing hardware 771 and display subsystem 781. In this embodiment, boost engine 750 is part of processing hardware 771. For example, boost engine 750 can be a coprocessor of GPU 710, or can be implemented by program code running on general-purpose or dedicated processing circuits of processing hardware 771.

[0049] exist Figure 7 and Figure 8 In an embodiment, GPU 710 may offload some of its rendering tasks to boost engine 750 to achieve smooth image display on display panel 720. Offloading may occur in response to resource constraints, such as insufficient computing power or high power consumption. Offloading enables the display refresh rate to be maintained at a default refresh rate (e.g., 60 FPS) or exceed the default refresh rate (e.g., 120 FPS).

[0050] Figure 9 An example of a system 900 according to one embodiment is shown. The system 900 may be Figure 2 System 100 or Figure 3 The system 900 includes a transmitter (Tx) device 910 and a receiver (Rx) device 920 coupled to each other via a transmission network 980. The transmission network 980 can be a wired and / or wireless network, such as the Internet. Figure 1 、 2 and 3, the Tx device 910 includes a first stage circuit 110, and the Rx device 920 includes a second stage circuit 120 and a boost engine 250 or 350. In one embodiment, the Tx device 910 and the Rx device 920 can be any two endpoints in the transmission path of the frame sequence. As a non-limiting example, the Tx device 910 can be a cloud server, and the Rx device 920 can download the frame sequence (e.g., video) from the cloud server. The quality of the frame sequence generated by the Tx device 910 and received by the Rx device 920 may be uneven because the FPS and resolution of the frame may change dynamically due to unstable transmission bandwidth. The boost engine 950 can be configured to perform the following operations according to the above-mentioned Figure 2-6 The combined operations stabilize the frame quality at the receiving end.

[0051] Already referenced Figure 2 、 3and 7-9 illustrate exemplary embodiments of Figure 6 However, it should be understood that Figure 6 The operations of the flowchart can be represented by Figure 2 、 3 and 7-9 embodiments of the present invention other than the embodiments, and Figure 2 、 3 The embodiments of and 7-9 may perform operations different from those discussed with reference to the flowcharts. Figure 6 The flowcharts illustrate a particular order of operations performed by certain embodiments of the invention, but it should be understood that such order is exemplary (e.g., alternative embodiments may perform operations in a different order, combine certain operations, perform certain operations in parallel in time, etc.).

[0052] Various functional components or blocks have been described herein. As will be understood by those skilled in the art, the functional blocks are preferably implemented by circuits (either dedicated circuits or general-purpose circuits, operating under the control of one or more processors and coded instructions), which typically include transistors configured to control the operation of the circuits according to the functions and operations described herein. For example, the first-stage circuit 110 and the boost engine 250 may be functional blocks implemented by dedicated circuits or general-purpose circuits.

[0053] Although the present invention has been described in terms of several embodiments, those skilled in the art will recognize that the present invention is not limited to the embodiments described and can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is therefore to be regarded as illustrative rather than restrictive.

Claims

1. A method for enhancing the quality of a frame sequence, characterized in that: include: The first-stage circuit adjusts the quality of a frame sequence to be output during rendering to generate a frame sequence with degraded quality when a specific condition of system resources is detected, and outputs the frame sequence with degraded quality; wherein the system resources include one or more of computing resources, power resources, and transmission bandwidth; The upscaling engine receives a frame sequence with quality degradation from the first stage circuit; the quality degradation of the frame sequence includes at least one of uneven resolution and uneven frames per second (FPS); and The boosting engine generates an enhanced frame sequence based on the frame sequence for transmission to a second level circuit.

2. The method according to claim 1, characterized in that Generating the enhanced frame sequence includes calculating an optical flow for the current frame using a frame preceding the current frame as a reference frame.

3. The method according to claim 1, characterized in that Generating the enhanced frame sequence includes calculating an optical flow for a current frame using a previous frame having a quality better than a quality threshold as a reference frame.

4. The method according to claim 1, wherein Adjusting the quality of the output frame sequence during rendering includes: Dynamically change frame quality during rendering in response to system resource limitations.

5. The method according to claim 1, wherein Also includes: activating the boosting engine by sending a help request from the first stage circuit to the boosting engine, the help request indicating information of frames with degraded quality; as well as In response to the help request, the upscaling engine performs image alignment between the current frame and a reference frame, and frame quality optimization including at least super-resolution and / or inpainting.

6. The method according to claim 1, characterized in that Also includes: The upscaling engine detects a current frame with degraded quality; as well as The upscaling engine performs image alignment between the current frame and a reference frame, and frame quality optimization including at least super-resolution and / or inpainting.

7. The method according to claim 1, characterized in that Also includes: The boosting engine optimizes the quality of received frames in the sequence of frames; The improvement engine compares the optimized quality with a threshold; as well as When the optimized quality is lower than the threshold, the quality is returned to the quality of the received frame.

8. The method according to claim 1, characterized in that The first stage circuit, the boost engine and the second stage circuit are located in the same electronic device, or, The first-stage circuit and the second-stage circuit are in two electronic devices coupled to each other through a transmission network.

9. A system for enhancing the quality of a frame sequence, characterized in that include: The first stage circuit is configured to, upon detecting a specific condition of system resources, adjust the quality of a frame sequence to be output during rendering to generate a frame sequence with degraded quality, and output the frame sequence with degraded quality, wherein the quality degradation of the frame sequence includes at least one of uneven resolution and uneven frames per second (FPS); wherein the system resources include one or more of computing resources, power resources, and transmission bandwidth; as well as The enhancement engine is configured to receive the frame sequence from the first-level circuit and generate an enhanced frame sequence based on the frame sequence for transmission to the second-level circuit.

10. The system according to claim 9, characterized in that The lifting engine is further configured to calculate an optical flow for the current frame using a frame preceding the current frame as a reference frame.

11. The system according to claim 9, wherein: The lifting engine is further configured to calculate an optical flow for a current frame using a previous frame having a quality better than a quality threshold as a reference frame.

12. The system according to claim 9, wherein: The first stage circuit is further configured to dynamically change frame quality during rendering in response to system resource limitations.

13. The system according to claim 9, wherein: The first stage circuit further operates to activate the upscaling engine by sending a help request indicating information of a frame with degraded quality, and in response to the help request, the upscaling engine performs image alignment between a current frame and a reference frame, and frame quality optimization including at least super-resolution and / or inpainting.

14. The system according to claim 9, wherein: The upscaling engine is further configured to detect a current frame with degraded quality and perform image alignment between the current frame and a reference frame and frame quality optimization including at least super-resolution and / or restoration.

15. The system according to claim 9, wherein: The boosting engine is further configured to optimize the quality of received frames in the sequence of frames; comparing the optimized quality to a threshold; and When the optimized quality is lower than the threshold, the quality is returned to the quality of the received frame.

16. The system according to claim 9, wherein: The first-stage circuit, the boost engine, and the second-stage circuit are located in the same electronic device; or, the first-stage circuit and the second-stage circuit are in two electronic devices coupled to each other via a transmission network.

Images