Method and system for enhancing the quality of frame sequences

Abstract

The present invention provides a method and system for enhancing the quality of a frame sequence. In one embodiment, the method provided by the present invention for enhancing the quality of a frame sequence by an enhancement engine may include: receiving a frame sequence having at least one frame with degraded quality from a first-level circuit, wherein the degraded quality includes at least one of uneven resolution and uneven frames per second (FPS); using a query input based on the frame with degraded quality to query an information library to obtain reference information of the frame with degraded quality as a query output; applying a neural network to the query input and the query output to generate an optimized frame; and sending the enhanced frame sequence including the optimized frame to a second-level circuit. This embodiment of the present invention optimizes frames with degraded quality, thereby enhancing the quality of the frames.

Description

Technical Field

[0001] Embodiments of the present invention relate to image processing techniques, and more particularly, to image and video processing to perform frame quality enhancement. Background Art

[0002] Typical edge electronic devices, such as TVs, smartphones, wearable devices, portable computing devices, gaming devices, etc., have limited computing capabilities due to strict requirements on power consumption and thermal performance. Graphics rendering operations on edge devices usually generate a large amount of graphics processing unit (GPU) workload. In order to maintain the target frame rate for smooth image display, edge devices may suffer from high power consumption. Sometimes, due to various resource limitations, such as high computational workload and power consumption limitations, the target frame rate cannot be achieved. Therefore, it is necessary to improve image processing technology to minimize the impact of resource limitations on frame quality. Summary of the Invention

[0003] The present invention provides a method and system for enhancing the quality of a frame sequence.

[0004] In one embodiment, the method provided by the present invention for enhancing the quality of a frame sequence by an enhancement engine may include: receiving a frame sequence having at least one frame with degraded quality from a first-level circuit, wherein the quality degradation includes at least one of uneven resolution and uneven frames per second (FPS); using a query input based on the frame with degraded quality to query an information library to obtain reference information of the frame with degraded quality as a query output; applying a neural network to the query input and the query output to generate an optimized frame; and sending the enhanced frame sequence including the optimized frame to a second-level circuit.

[0005] In one embodiment, the system for enhancing the quality of a frame sequence provided by the present invention may include: a first-level circuit for sending a frame sequence having at least one frame with degraded quality, wherein the quality degradation includes at least one of uneven resolution and uneven frames per second (FPS); and an enhancement engine for receiving the frame sequence, querying an information library using a query input based on the frame with degraded quality to obtain reference information of the frame with degraded quality as a query output; applying a neural network to the query input and the query output to generate an optimized frame; and sending the enhanced frame sequence including the optimized frame to a second-level circuit.

[0006] As described above, the embodiments of the present invention optimize frames with degraded quality, thereby enhancing the quality of the frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Examples of multiple frame sequences are illustrated according to some embodiments.

[0008] Figure 2 is a block diagram illustrating a system 100 for improving the quality of a frame sequence, according to one embodiment.

[0009] Figure 3 is a diagram illustrating an example of enhancement engine operation using information provided by the cloud server 320 according to one embodiment.

[0010] Figure 4 is an example diagram illustrating the operation of an enhancement engine using information provided by database 420 according to one embodiment.

[0011] Figure 5 A query operation on the database 420 is shown according to one embodiment.

[0012] Figure 6 is a diagram illustrating the calculation of new (key element, value) pairs according to one embodiment.

[0013] Figure 7 is a diagram illustrating calculation of new (key element, value) pairs according to another embodiment.

[0014] Figure 8 is a diagram illustrating an operation of updating a database according to one embodiment.

[0015] Figure 9 is a diagram illustrating database expansion based on a portion of a frame according to one embodiment.

[0016] Figure 10 is a diagram illustrating repairing using an expanded database according to one embodiment.

[0017] Figure 11 is a diagram illustrating super-resolution (SR) performed using an extended database according to one embodiment.

[0018] Figure 12 is a flow chart illustrating a method 1200 for enhancing the quality of a frame sequence, according to one embodiment.

[0019] Figure 13 An example diagram of device 1300 is shown according to one embodiment.

[0020] Figure 14 An example diagram of a device 1400 is shown according to another embodiment.

[0021] Figure 15 An example of system 1500 is shown according to one embodiment. DETAILED DESCRIPTION

[0022] Certain terms are used throughout the specification and claims to refer to specific components. Those skilled in the art will understand that hardware manufacturers may use different terms to refer to the same component. This specification and claims do not distinguish components based on differences in name, but rather on differences in their functionality. Throughout the specification and claims, the terms "including" and "comprising" are open-ended and should be interpreted as meaning "including, but not limited to." "Substantially" or "approximately" means that within an acceptable range of error, a person skilled in the art can solve the technical problem and achieve the desired technical effect. Furthermore, the term "coupled" or "coupled" encompasses any direct and indirect electrical connection means. Therefore, if a first device is described as coupled to a second device, this means that the first device can be directly electrically connected to the second device or indirectly electrically connected to the second device through other devices or connection means.

[0023] The following description is made for the purpose of illustrating the general principles of the invention and is not to be construed in a limiting sense. The scope of the invention is best determined by reference to the appended claims.

[0024] Figure 1 Examples of multiple frame sequences are illustrated according to some embodiments. The first level circuit 110 sends the frame sequence to the second level circuit 120. The quality of the frame sequence may sometimes be unstable; that is, sometimes the frame rate of the frame sequence may be lower than the target frame per second (FPS), and the resolution of some frames in the frame sequence may be lower than the resolution of other frames. The first level circuit 110 can output any frame sequence shown in the following non-limiting examples. Row (A) shows a frame sequence with even quality, where all frames (I0-I6) are generated by the first level circuit 110 at the target FPS and target resolution. Row (B) shows an uneven FPS condition, where frames I1, I2, I4, and I6 are lost in transmission or are not generated by the first level circuit 110 (e.g., are not rendered). In a frame sequence with an uneven FPS condition, the frame rate (also called FPS) changes dynamically without following a regular pattern.

[0025] Row (C) shows a non-uniform resolution condition, where frames I0, I4, and I6 have a target resolution (e.g., 2400x1080), frame I3 has a resolution lower than the target resolution (e.g., 1600x720), and frames I1, I2, and I5 have the lowest resolution in the frame sequence (e.g., 600x270). In a frame sequence with non-uniform resolution, the resolution of the frames changes dynamically without following a regular pattern. Row (D) shows a combination of non-uniform FPS and non-uniform resolution conditions. Rows (B), (C), and (D) provide non-limiting examples of frame sequences with non-uniform quality conditions. It will be understood that a frame sequence with non-uniform quality conditions can have any combination of non-uniform FPS and non-uniform resolution, including only non-uniform FPS or only non-uniform resolution. Although the examples herein show uneven quality conditions, it should be understood that the first stage circuit 110 may also transmit a sequence of frames with even quality degradation, for example, a sequence of frames with every other frame missing or every other frame having a lower resolution.

[0026] In the following description, a frame sequence with an uneven quality condition may also be referred to as a frame sequence with reduced quality in a plurality of frames that are unevenly spaced in time. In the example of row (B), the lost frames I1, I2, I4, and I6 are unevenly spaced in time, where the interval between I1 and I2 is one frame interval (flame interval), the interval between I2 and I4 is two frame intervals, and the interval between I4 and I6 is two frame intervals. Similarly, in the examples of rows (C) and (D), the lost and / or low-resolution frames are unevenly spaced in time. Therefore, it should be understood that the term "uneven" hereinafter may be interpreted as "uneven frame sequence."

[0027] Due to resource limitations, first-stage circuitry 110 may determine or be requested to generate a sequence of frames with uneven quality conditions. Non-limiting examples of resource limitations include insufficient transmission bandwidth, high computational workload, power consumption limitations, etc. In one embodiment, first-stage circuitry 110 may reduce the quality of frames with low or slowly varying information content. Alternatively or additionally, first-stage circuitry 110 may reduce the quality of one or more frames when a limited resource has exceeded its usage threshold, for example, when power consumption exceeds a threshold. When compared to a predetermined threshold, the quantized value may be "insufficient," "high," "slow," or "low." In one embodiment, the first-stage circuitry may monitor the usage of the limited resource using a host circuit or running a background thread. When the host circuitry or thread detects that the usage of the limited resource exceeds the threshold, it notifies the first-stage circuitry to adjust the quality of the frame sequence. This quality reduction may include at least one of uneven resolution and uneven FPS. Non-limiting examples of limited resources include one or more of computing resources, power resources, and transmission bandwidth.

[0028] First-stage circuit 110 and second-stage circuit 120 can be any two endpoints of a frame sequence transmission network or connection. In one embodiment, first-stage circuit 110 and second-stage circuit 120 can be located in the same electronic device, such as a graphics processing unit (GPU) and a display panel located in the same device. In another embodiment, first-stage circuit 110 and second-stage circuit 120 can be located in different devices, such as a transmitter (Tx) device and a receiver (Rx) device connected via a transmission network.

[0029] The first stage circuit 110 can dynamically adjust the frame quality during rendering and / or transmission to produce a sequence of frames with uneven quality conditions. The adjustment can include temporal reduction and / or spatial reduction. Temporal reduction refers to a reduction in FPS; for example, reducing the number of rendered frames and / or transmitted frames per time unit. Spatial reduction refers to a reduction in frame resolution; for example, reducing the number of pixels in a rendered and / or transmitted frame. Figure 1 As shown in rows (B) and (D) of , the temporal reduction may be non-uniform; for example, the time interval between two directly adjacent frames may vary from time to time. Figure 1As shown in rows (C) and (D) of FIG, the spatial reduction may be non-uniform; for example, low-resolution frames may not appear at regular intervals. Non-uniform quality conditions in a frame sequence, such as FPS reduction and / or resolution reduction, can save significant computational, bandwidth, and / or power resources of the first-stage circuitry 110. The first-stage circuitry 110 can adjust the quality reduction with great flexibility to achieve a balanced and smooth trade-off between frame quality and resource usage.

[0030] As will be described in more detail later, the disclosed system also includes a booster engine to recover from quality degradation of a frame sequence. The boost engine can be activated on demand. In one embodiment, the boost engine receives a frame sequence from the first-level circuit 110, enhances the frame quality, and sends the enhanced frame sequence to the second-level circuit 120. The frame sequence can have an uneven quality condition or a uniform quality condition. In an embodiment where the boost engine coexists with the first-level circuit 110, such as a GPU, the boost engine can offload rendering operations from the GPU. The offloading of rendering operations can enable the system to increase FPS with acceptable power consumption. In another embodiment where the boost engine is located in an Rx device of the transmission network, the boost engine can be used as a stabilizer to stabilize the frame quality received by the Rx device.

[0031] Figure 2 is a block diagram illustrating a system 100 for improving the quality of a sequence of frames according to one embodiment. Components indicated by dashed lines represent components that may not be present in some alternative embodiments. System 100 includes a first-level circuit 110 and a second-level circuit 120 coupled to the first-level circuit 110 via a connection (e.g., a bus connection or a transmission network connection). An enhancement engine 250 is coupled to the connection and is used to improve the quality of a sequence of frames transmitted from the first-level circuit 110 to the second-level circuit 120. The enhancement engine 250 can be a dedicated hardware circuit, or a general-purpose hardware circuit that executes software containing instructions for enhancing frame quality, software executed by a processor, or a combination of hardware circuits and software instructions. Therefore, in some embodiments, the enhancement engine 250 is also referred to as an enhancement engine circuit. The frame quality enhancement performed by the enhancement engine 250 may include super-resolution, frame interpolation and / or extrapolation, restoration, image alignment, motion synthesis, and the like.

[0032] In one embodiment, the first-stage circuit 110 includes a resolution adjustment module 112 and an FPS adjustment module 113. The first-stage circuit 110 uses the resolution adjustment module 112 to dynamically adjust frame quality to generate low-resolution frames, and / or the FPS adjustment module 113 to change the frame rate. In one embodiment, the system 100 may also include a main processor (not shown) that controls the operation of the first-stage circuit 110. The system 100 monitors system resource usage, such as computing resource usage, power consumption, and transmission bandwidth usage. When resource usage reaches a limit, the first-stage circuit 110 or the main processor activates one or both of the resolution adjustment module 112 and the FPS adjustment module 113 to adjust (e.g., reduce) the output frame quality. Adjustments can be made as needed; for example, the resolution and / or frame rate can be reduced for any frame at any time interval. One or both of the resolution adjustment module 112 and the FPS adjustment module 113 can be implemented by dedicated hardware circuitry, software containing instructions executable by a processor, or a combination of hardware circuitry and software instructions. In one embodiment, the resolution adjustment module 112 and the FPS adjustment module 113 may both be part of a GPU rendering pipeline.

[0033] In one embodiment, the first stage circuit 110 may generate additional information including metadata about the low-resolution frame and the lost frame, and send the additional information to the enhancement engine 250. For example, the first stage circuit 110 may render frame (N) but not render the low-resolution frame (N+1) or render frame (N+1) at a low resolution. To help the enhancement engine 250 improve the frame quality, the first stage circuit 110 may generate metadata describing the attributes of frame (N+1) and send the metadata to the enhancement engine 250. The metadata may include any of the following information about frame (N+1): depth, texture, normal, color, instance segmentation, motion vector information (e.g., optical flow), frame resolution, etc. It will be appreciated that the enhancement engine 250 may enhance the quality of the frame with or without using the additional information from the first stage circuit 110.

[0034] In one embodiment, the first stage circuit 110 may send a help request to the enhancement engine 250 to request frame quality enhancement. The help request may indicate a quality reduction strategy, such as FPS reduction and / or resolution reduction. The help request may also provide an index of a frame with reduced quality. For example, the help request may include a frame insertion request indicating the location of a frame in a frame sequence that was not rendered or transmitted. The enhancement engine 250 may perform motion synthesis and alignment to insert these missing frames.

[0035] In one embodiment, the enhancement engine 250 includes an alignment module 252, which is coupled to the optimization module 253. The alignment module 252 performs temporal and spatial alignment on multiple frames based on their respective reference frames. The alignment module 252 performs geometric transformations, frame interpolation and / or extrapolation, and other post-processing (including but not limited to blending). The alignment module 252 can perform the above operations using motion information sent by the first stage circuit 110 (which can be included in the metadata). Alternatively, the alignment module 252 can include a motion synthesizer 251 to generate motion information. The motion synthesizer 251 can extract motion information from the frame; for example, by generating motion vectors and optical flow between the current frame (i.e., the frame currently being processed by the enhancement engine 250) and the reference frame. The motion information can include acceleration estimates of objects in the frame.

[0036] The optimization module 253 performs frame optimization operations, including but not limited to super-resolution (SR), repair, blending, sharpening, and other image processing operations. In one embodiment, the optimization module 253 may include an artificial intelligence (AI) model trained to perform the optimization operation. For example, the optimization module 253 may include an AI SR model for upscaling a low-resolution image to a higher-resolution image. The optimization module 253 may also include an AI repair model for repairing images with many missing pixels (e.g., holes in the image). The output of the optimization module 253 may be sent to the second-stage circuit 120.

[0037] One or more of the motion synthesizer 251, alignment module 252, and optimization module 253 in the enhancement engine 250 may be implemented by dedicated hardware circuits, software containing instructions executable by a processor, or a combination of both. Based on information transmitted from the first-stage circuit 110, platform capabilities, and / or target output quality, the enhancement engine 250 may activate one or more of the motion synthesizer 251, alignment module 252, and optimization module 253 to improve frame quality.

[0038] In one embodiment, enhancement engine 250 includes a quality detector 210 for detecting frame quality in a frame sequence transmitted from first-stage circuitry 110 to second-stage circuitry 120. When quality detector 210 detects uneven quality in the frame sequence, such as uneven FPS and / or uneven resolution, enhancement engine 250 activates alignment module 252 and optimizer module 253 to improve frame quality. Using quality detector 210 eliminates the need for assistance from first-stage circuitry 110.

[0039] In this embodiment, the output of the optimization module 253 is checked by the quality checker 220, which compares the quality of the frame output from the optimization module 253 to a quality threshold. If the frame quality of the output frame does not meet the quality threshold, the output falls back to the original frame received by the enhancement engine 250. For example, the original frame may have lost so much information that the restoration performed by the optimization module 253 is of unacceptable quality.

[0040] In an alternative embodiment, the enhancement engine 250 may improve frame quality without additional information (e.g., metadata) from the first stage circuit 110. For example, the enhancement engine 250 may calculate any of the following from the received frame sequence: depth, texture, normal, color, instance segmentation, motion vector information (e.g., optical flow), frame resolution, etc. to perform motion estimation and compensation, frame interpolation / extrapolation, alignment, super-resolution, inpainting, etc.

[0041] In embodiments where the first stage circuit 110 does not provide additional information or the provided additional information does not include motion information, the motion synthesizer 251 may use information in one or more previous frames to generate optical flow for motion estimation and compensation.

[0042] As an alternative to or in addition to the additional information provided by the first-stage circuitry, enhancement engine 250 may utilize information provided by other sources. In one embodiment, enhancement engine 250 may query (i.e., search) information repository 260 for reference information using a query input that includes content from a given frame as an index. The output of information repository 260 is then used to enhance the quality of the given frame. In one embodiment, information repository 260 may include any information accessible via the Internet; for example, information provided by the World Wide Web (i.e., the World Wide Web). Enhancement engine 250 may use proprietary or public search engines to search for information. Alternatively, information repository 260 may include a database that is locally or remotely accessible by enhancement engine 250 via a public or proprietary connection, whether wired or wireless.

[0043] Figure 3 is a diagram illustrating an example of an enhancement engine operation using information provided by a cloud server 320 according to one embodiment. In this embodiment, the enhancement engine 350 includes a neural network 310 to process the query output and generate an optimized image 370. The enhancement engine 350 may be Figure 2. The enhancement engine 350 may send a search request to the cloud server 320 at runtime, wherein the search request includes the input image 300. The search request may be sent via the Internet. In one embodiment, the cloud server 320 manages image information stored in the cloud storage 321. The input image 300 may be one of the following: a frame in a frame sequence, an interpolated frame, an extrapolated frame, or a portion of a frame. The input image 300 may have degraded image quality, such as low resolution or missing pixel data. The output image 360 ​​generated by the search may improve the quality of the input image 300 through restoration, super-resolution, or other image processing techniques. The input image 300 and the output image 360 ​​are fed to the neural network 310, which performs an SR operation or a restoration operation to produce an optimized image 370. The optimized image 370 is part of the enhanced frame sequence that is sent to the second-stage circuit 120.

[0044] Figure 4 4 is an example diagram illustrating the operation of an enhancement engine using information provided by a database 420 according to one embodiment. The database 420 stores a set of N (key, value) pairs, where each key is of length C and each value is of length C′, where C and C′ may be different. That is, the length of each key is independent of the length of its corresponding value. The key and value are trainable parameters. The enhancement engine 250 may send a search request to the database 420 at runtime, the search request including a feature map 455 generated by the neural network 430 from the input image 400, which may be compared to the input image 400. Figure 3 The input image 300 in is the same.

[0045] The feature map 455 has dimensions H (height) × W (width) × C (channels), also expressed as (H, W, C). The feature map 455 includes H × W feature map elements 456, each of which has dimensions (1, 1, C). For each feature map element 456, the enhancement engine 250 queries the database 420 to obtain an output element 466 of size (1, 1, C′). Therefore, when the query input is the entire feature map 455, the database 420 produces a query output 460 of size (H, W, C′). The query output 460 and the feature map 455 are sent to the neural network 410, which performs an SR operation or inpainting operation to produce an optimized image 470. The optimized image 470 is part of the enhanced frame sequence that is sent to the second stage circuit 120.

[0046] Figure 5-Figure 8According to some embodiments, some operations of querying, extending, and updating database 420 are described. In an embodiment where database 420 is managed by enhancement engine 450, these database operations may be performed by enhancement engine 450. In another embodiment where database 420 is managed by an entity other than enhancement engine 450 (e.g., a database server), these database operations may be performed by that entity in response to requests submitted by enhancement engine 450. Although the following disclosure references Figure 5-Figure 8 While an enhancement engine is described as performing database operations, it should be understood that the enhancement engine may perform or cause another entity (e.g., a database server) to perform the operations described in the accompanying reference. Figure 5-Figure 8 The exposed database operations.

[0047] Figure 5 According to one embodiment, a query operation on the database 420 is shown. Figure 5 In an embodiment, the database 420 ( Figure 4 ) includes N pairs of key elements 521 and values ​​522 (i.e., N (key element, value) pairs). Each of the feature graph elements 456 is used to query the N key elements 521 to generate N blending weights 510; for example, cross-correlation can be applied to each feature graph element 456 and the N key elements 521. Each blending weight 510 is an indication of the similarity between the query input (i.e., feature graph element 456) and the corresponding key element. The blending weight 510 is applied to the value 522 (e.g., by convolution or matrix product) to generate a weighted sum of the value 522. The weighted sum is an element of the query output 460 (e.g., Figure 4 The calculation of the mixing weights and weighted sums is repeated to generate query output 460.

[0048] In one embodiment, the enhancement engine 450 may extend and / or update the database 420 at runtime, or cause the database 420 to be extended and / or updated at runtime. The extension and update of the database may be based on frames received by the enhancement engine 450 at runtime.

[0049] Figure 6 is a diagram illustrating the calculation of new (key element, value) pairs according to one embodiment. Figure 4, in one embodiment, this calculation may be performed by the enhancement engine 450 and the new (key element, value) pairs are submitted to the database 420. Initially, the enhancement engine 450 computes an image embedding 610 from a frame or a portion of a frame using a neural network such as the neural network 430. The image embedding 610 may be a feature map 455 or a portion of a feature map 455. The image embedding 610 has dimensions (H″, W″, C″), which may be the same as or different from the dimensions (H, W, C) of the feature map. The enhancement engine 450 performs N operations 650 on the image embedding 610 to produce new key elements 641 and new values ​​642. According to the operation 650, the image embedding 610 is convolved with a convolution kernel weight 621 of dimensions (H″, W″, C″). The stride size of the convolution is 1. Each convolution produces a pixel of dimension (1, 1, 1). The convolution is repeated C times to obtain a new key element 631 of size (1, 1, C). Similarly, the image embedding 610 is convolved with a convolution kernel weight 622 of size (H″, W″, C″) with a stride size of 1. The convolution is repeated C′ times to obtain a new value 632 of size (1, 1, C′). Operation 650 is repeated N times, each time using a different set of convolution kernel weights 621 and a different set of convolution kernel weights 622. The convolution kernel weights 621 and 622 are trainable parameters. The N repetitions produce N new key elements 641 and N new values ​​642.

[0050] Figure 7 is a diagram illustrating the calculation of new (key element, value) pairs according to another embodiment. Figure 4 In one embodiment, this calculation may be performed by the enhancement engine 450 and the new (key element, value) pair is submitted to the database 420. The enhancement engine 450 performs operation 770 on the image embedding 610 and then performs operation 780 N times to produce a new key element 761 and a new value 762. According to operation 770, the image embedding 610 is convolved with a convolution kernel weight 721 of size (k, k, C″). The stride size of the convolution is s. The convolution is repeated C times to produce a size of The middle key element 731 of A set of multi-layer perception (MLP) weights 741 are convolved with the intermediate key element 731 to produce a new key element 751 of size (1, 1, C). The convolution is of stride = 1 Convolution, and the same kernel weights are applied to each of the C channels.

[0051] Furthermore, according to operation 770, the image embedding 710 is convolved with a convolution kernel weight 722 of size (k, k, C″). The stride of the convolution is s. The convolution is repeated C′ times to produce a convolution kernel weight of size The intermediate value element 732 of A set of MLP weights 742 of is convolved with the intermediate value element 732 to produce a new value element 752 (i.e., a new value) of size (1, 1, C). This convolution is of stride = 1. Convolution is performed, and the same kernel weights are applied to each of the C′ channels. Operation 780 is repeated N times, generating N new key elements 761 and N new values ​​762. Each repetition uses a different set of MLP weights 741 and a different set of MLP weights 742. Convolution kernel weights 721 and 722, as well as MLP weights 741 and 742, are trainable parameters.

[0052] according to Figure 6 and Figure 7 The (key element, value) pairs calculated by the operations in can be used to expand and / or update the database 420. For example, the enhancement engine 450 can expand the database 420 by adding new (key element, value) pairs to existing pairs. The enhancement engine 450 can also update the database 420 by replacing existing (key element, value) pairs with new (key element, value) pairs to control database storage space. The existing pairs to be replaced may be the oldest or least used pairs. Alternatively or additionally, the enhancement engine 450 can update the database 420 through parameterized updates, where the update speed is controlled by parameters α and β.

[0053] Figure 8 821 and the updated value 822 are calculated as follows: (updated key element, updated value) = (α·new key element + (1-α)·old key element, β·new value + (1-β)·old key element), where α and β are parameters in the range of [0, 1]. When α=β=0, the N existing (key element, value) pairs in the database 420 remain unchanged. When α=β=1, the N existing (key element, value) pairs in the database 420 are completely replaced by the new (key element, value) pairs. In another embodiment, updates can be performed on one or more (key element, value) pairs at a time.

[0054] Figures 9-11As an example, frames received by the enhancement engine 450 at times T=0, 1, and 2 are used. It will be understood that the three frames at T=0, 1, and 2 are not three consecutive frames in the frame sequence received by the enhancement engine 450; the frame sequence has multiple intermediate frames between the times shown in the figure. The three frames show that the second object 940 moves from the left to the right of the first object 930. At T=0, area 931 of the first object 930 is visible, and the rest of the first object 930 is occluded by the second object 940. At T=1, the first object 930 is completely occluded by the second object 940. At T=2, area 932 of the first object 930 is not covered (i.e., visible), while area 932 was at least partially occluded at T=0.

[0055] Figure 9 is a diagram illustrating database expansion based on a portion of a frame according to one embodiment. Figure 4 , the enhancement engine 450 can expand the database 420 by adding M (key element, value) pairs to the existing N (key element, value) pairs to generate an expanded database 920. The enhancement engine 450 can calculate one or more of the M (key element, value) pairs based on a portion of a frame (e.g., a frame before T=2) (e.g., region 931 of the first object 930 at time T=0). The enhancement engine 450 can predict that region 931 of the first object 930 will be occluded and region 932 will become visible in a subsequent frame (e.g., a frame at T=2). This prediction can be based on motion synthesis of multiple frames between time 0 and 1. Although region 931 and region 932 are different regions (in an optional embodiment, region 931 and region 932 may also partially overlap), they belong to the same object. Therefore, compared to other values ​​in the expanded database 920, the information (value) derived from region 931 may be more useful for optimizing region 932. When there is quality degradation in a subsequent frame (e.g., at T=2), the enhancement engine 450 can query the expanded database 920 and benefit from the (key element, value) pairs with information related to the first object 930 (e.g., the (key element, value) pairs related to region 931) to predict the (key element, value) pairs of region 932, thereby optimizing region 932 (e.g., repairing lost pixels in 932 or enhancing the resolution of region 932). Therefore, the enhancement engine 450 can expand the database based on image regions that are currently in the scene (in-scene) and may be at least partially in the scene in subsequent frames but have quality degradation (e.g., missing some / all pixels) in subsequent frames. In addition, the enhancement engine 450 can expand the database based on high resolution (HR) frames, and the database query output can be used to sharpen the image in subsequent low resolution (LR) frames.

[0056] exist Figure 9 In the example of FIG. 1 , the enhancement engine 450 uses the region 931 to calculate (key element, value) pairs (e.g., one or more of the aforementioned M (key element, value) pairs) by applying the neural network 910 to the region 931, where the neural network 910 can be used according to Figure 6 or Figure 7 The operations shown are performed. Encoding (e.g., by applying neural network 910 to) a region of a frame rather than the entire frame can save database storage space and reduce database search time. Because the database stores information of interest per frame rather than the entire frame, the same amount of database storage space can store information for more frames. Enhancement engine 450 then adds the calculated (key element, value) pair(s) to the expanded database 920.

[0057] It should be noted that in Figure 9 Only two times, T=0 and T=1, are shown before T=2. In practice, for region 932 with missing pixels at time T=2 (e.g., region 932 may be completely missing or partially missing), enhancement engine 450 may generate the M (key element, value) pairs using multiple regions from multiple frames at multiple times before T=2, rather than solely based on region 931 of object 930 at time T=0. That is, a portion of the M (key element, value) pairs may be derived from (key element, value) pairs associated with region 931 at time T=0 that partially overlapped with region 932, while the remainder of the M (key element, value) pairs may be derived from (key element, value) pairs associated with other visible regions that overlapped with region 932 at other times before T=2. Enhancement engine 450 may then utilize these M (key element, value) pairs for optimization (e.g., performing super-resolution processing on the frame at time T=2 or repairing region 932 at time T=2). That is, the (key element, value) pairs serving as reference (key element, value) pairs for region 932 may originate from multiple regions of multiple frames at multiple time points before time T=2. Furthermore, in an alternative embodiment, K existing (key element, value) pairs in a database may be used as reference (key element, value) pairs to optimize visible region 932 at time T=2. These K (key element, value) pairs may be independent of the time sequence (i.e., not originate from any frame before T=2). These K (key element, value) pairs may be a portion of the existing N (key element, value) pairs.

[0058] Figure 10 FIG is a diagram illustrating the use of an expanded database to perform repair according to an embodiment. Figure 4, the enhancement engine 450 may detect that the first object 930 is moving compared to the second object 940 (in practice, the moving object may be the first object or the second object). Based on the motion synthesis of multiple frames between time 0 and 1, the enhancement engine 450 may predict the position of the first object 930 at T=2 and the visible area of ​​the first object 930 at T=2 (i.e., area 932). If the frame is lost at T=2 or the pixel data of area 932 is lost or damaged, the enhancement engine 450 may construct a frame for T=2, where the constructed frame includes area 932. In one embodiment, the enhancement engine 450 uses the neural network 1010 to generate a feature map 1055 as a query input to query the expanded database 920. In this example, the enhancement engine 450 may apply the neural network 1010 to a portion of the frame predicted to appear at T=2 (e.g., area 932 with missing pixels (here, missing pixels may include area 932 being completely missing or only partially missing pixels)) to generate a feature map 1055. According Figure 5 In the query operation shown, the database 420 outputs one element of the query output 1060 (having size (1, 1, C')) for each element of the feature map 1055 (having size (1, 1, C)). The enhancement engine 450 then applies the neural network 1020 to the query output 1060 and the feature map 1055 to generate a restored frame 1070 at time T = 2, wherein the restored frame 1070 includes the restored region 932 of the first object 930.

[0059] Figure 11 is a diagram illustrating super-resolution (SR) using an extended database according to one embodiment. In this example, the frame at T=2 has a low resolution (LR), while the frames at T=0 and T=1 have a high resolution (HR). Figure 4 , the enhancement engine 450 uses a portion of the LR frame including region 932 as input to the neural network 1110 to generate a feature map 1155. The enhancement engine 450 then uses the feature map 1155 as a query input to query the expanded database 920. Figure 5 In the query operation shown, the database 420 outputs one element of the query output 1160 (with size (1, 1, C)) for each element of the feature map 1155 (with size (1, 1, C)). The enhancement engine 450 then applies the neural network 1120 to the query output 1160 and the feature map 1155 to generate an SR version of the region 932. The enhancement engine 450 can also use the frame at T=I as a reference frame and use the alignment module 252 ( Figure 2 ) aligns the second object 940 in the frame at T = 2 with the second object 940 in the frame at T = 1. The aligned output is added to the SR version of region 932 to produce the SR frame 1170 for T = 2.

[0060] Figure 12 1 is a flow chart illustrating a method 1200 for enhancing the quality of a frame sequence according to one embodiment. The method 1200 may be performed by an enhancement engine, such as the enhancement engine 250 ( Figure 2 ), Enhanced Engine 350( Figure 3 ) and / or enhancement engine 450 ( Figure 4 ). The enhancement engine is used to enhance the transmission from the first level circuit to the second level circuit (for example Figure 1 and Figure 2 Method 1200 begins at step 1210, where an enhancement engine receives a frame sequence having at least one frame with degraded quality from the first-stage circuit 110 and the second-stage circuit 120. The degraded quality includes at least one of uneven resolution and uneven frames per second (FPS). At step 1220, the enhancement engine uses a query input based on the degraded frame to query an information repository to obtain reference information for the degraded frame as a query output. At step 1230, the enhancement engine applies a neural network to the query input and the query output to generate an optimized frame. At step 1240, the enhancement engine sends the enhanced frame sequence including the optimized frame to the second-stage circuit.

[0061] In one embodiment, a neural network performs at least one of a super-resolution (SR) operation and a restoration operation on a degraded frame. The information repository may be accessible via the internet; alternatively, the information repository may be a database readable and writable by the enhancement engine. The database stores multiple sets of key elements and corresponding values. In one embodiment, the enhancement engine may use a given neural network and generate a feature map based on the frame (with degraded quality), use elements of the feature map to query the key elements to generate a set of blending weights, and apply the set of blending weights to corresponding values ​​or cause the set of blending weights to be applied to corresponding values ​​to generate elements of the query output. In one embodiment, the neural network is applied to the region of the degraded frame having missing pixels to generate the feature map.

[0062] In one embodiment, the enhancement engine may calculate an image embedding based on a first region of an object in a given frame in a sequence of frames. The enhancement engine may further apply convolution kernel weights to the image embedding or cause convolution weights to be applied to the image embedding to obtain new (key element, value) pairs, expand a database or cause the database to be expanded by adding the new (key element, value) pairs to the database, and query the database (i.e., the expanded database) to obtain information on a second region of the object in a subsequent frame (e.g., a frame of degraded quality). The second region is visible in the given frame and is at least predicted to be visible in the subsequent frame.

[0063] In one embodiment, the enhancement engine may calculate an image embedding based on a first region of an object in a given frame in the sequence of frames. The enhancement engine may further apply a convolution kernel weight to the image embedding or cause the convolution weight to be applied to the image embedding to obtain a new (key element, value) pair, and update a database or cause the database to be updated by replacing an existing (key element, value) pair with the new (key element, value) pair.

[0064] In one embodiment, the first-stage circuit, the enhancement engine, and the second-stage circuit can be located within the same electronic device. Alternatively, the first-stage circuit and the second-stage circuit can be located within two electronic devices coupled to each other via a transmission network. Furthermore, the neural network disclosed herein is characterized by having trainable parameters.

[0065] Figure 13 An example diagram of a device 1300 is shown according to one embodiment. The device 1300 may be Figure 2 1. An example of a device 1300 is an electronic device with graphics processing capabilities. For example, the device 1300 may be a smartphone, a computing device, a networking device, a gaming device, an entertainment device, an Internet of Things (IoT) device, or any device capable of processing and displaying images and / or video.

[0066] Device 1300 includes processing hardware 1370. In one embodiment, processing hardware 1370 includes a central processing unit (CPU) 1360, a GPU 1310, and one or more of the following: a digital processing unit (DSP), an artificial intelligence (AI) processor, a multimedia processor, and other general and / or specialized processing circuits. In one embodiment, GPU 1310 may be the aforementioned first stage circuit 110 ( Figure 1 、 2 and 3), and CPU 1360 is the main processor of GPU 1310.

[0067] Device 1300 also includes a display subsystem 1380 coupled to processing hardware 1370 via display interface circuitry 1340. In one embodiment, display subsystem 1380 includes enhancement engine 1350 and display panel 1320. Display panel 1320 displays information such as images, videos, messages, games, text, graphics, etc. In one embodiment, display panel 1320 may be the second stage circuitry 120 ( Figure 1 and Figure 2 ), and the enhancement engine 1350 may be the enhancement engine 250 ( Figure 2 ), Enhanced Engine 350( Figure 3 ) and / or enhancement engine 450 ( Figure 4 ). Enhancement engine 1350 can be part of display subsystem 1380. Enhancement engine 1350 can be implemented in hardware or can be implemented by program code running on general-purpose or dedicated processing circuits. In one embodiment, enhancement engine 1350 can include a hardware accelerator, such as an AI accelerator, to perform frame quality optimization (such as AI-based super-resolution, restoration, and other image processing operations).

[0068] The device 1300 also includes a memory 1330 coupled to the processing hardware 1370 and the display subsystem 1380. The memory 1330 may include memory such as dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, and other non-transitory machine-readable storage media (e.g., volatile or non-volatile memory devices). The memory 1330 includes one or more buffers 1335, such as a color buffer, a metadata buffer, a frame buffer, etc. The GPU 1310 may store rendered frames in the color buffer or the frame buffer, and store metadata in the metadata buffer, where the metadata includes information about frames that are not rendered, and frames that are of reduced quality (e.g., low resolution). In some embodiments, the memory 1330 may store instructions that, when executed by the processing hardware 1370, cause the processing hardware 1370 to perform Figure 12 Method 1200.

[0069] In one embodiment, the CPU 1360 can set the target refresh rate of the display interface circuit 1340 to control the rate at which images are output from the buffer 1335 to the display panel 1320, and can dynamically perform refresh rate adjustments when necessary. Figure 13 The embodiments are simplified for illustrative purposes. The device of the present invention may include additional hardware components. For example, the device 1300 may also include a network interface to connect to a wired and / or wireless network for transmitting and / or receiving voice, digital data, and / or media signals.

[0070] Figure 14 An example diagram of a device 1400 is shown according to another embodiment. The device 1400 may be Figure 2 A non-limiting example of the system 100 is shown in FIG. Figure 13 and Figure 1414. Device 1400 includes processing hardware 1470 and a display subsystem 1480. In this embodiment, enhancement engine 1350 is part of processing hardware 1470; for example, enhancement engine 1350 may be a coprocessor of GPU 1310, or enhancement engine 1350 may be implemented by program code running on general-purpose or dedicated processing circuitry within processing hardware 1470.

[0071] exist Figure 13 and Figure 14 In an embodiment, GPU 1310 may offload some of its rendering tasks to enhancement engine 1350 to achieve smooth image display on display panel 1320. Offloading may occur in response to resource limitations (e.g., insufficient computing power or high power consumption). Offloading enables the refresh rate of the display to be maintained at a default refresh rate (e.g., 60 FPS) or exceed the default refresh rate (e.g., 120 FPS).

[0072] Figure 15 An example of a system 1500 is shown according to one embodiment. The system 1500 may be Figure 2 The system 1500 includes a transmitter (Tx) device 1510 and a receiver (Rx) device 1520 coupled to each other via a transmission network 1580. The transmission network 1580 can be a wired and / or wireless network; for example, the Internet. Figure 1 and Figure 2 , the Tx device 1510 includes a first stage circuit 110, and the Rx device 1520 includes a second stage circuit 120 and an enhancement engine 1550. The enhancement engine 1550 may be an enhancement engine 250 ( Figure 2 ), Enhanced Engine 350( Figure 3 ) and / or enhancement engine 450 ( Figure 4 In one embodiment, the Tx device 1510 and the Rx device 1520 may be any two endpoints in the transmission path of the frame sequence. As a non-limiting example, the Tx device 1510 may be a cloud server, and the Rx device 1520 may download the frame sequence (e.g., video) from the cloud server. The quality of the frame sequence generated by the Tx device 1510 and received by the Rx device 1520 may be uneven because the FPS and resolution of the frame may change dynamically due to unstable transmission bandwidth. The enhancement engine 1550 may be based on the above combination of Figure 2-12 operations to stabilize the frame quality at the receiving end.

[0073] Already referenced Figure 2-Figure 4 The exemplary embodiments describe Figure 12 However, it should be understood that Figure 12 The operation of the flowchart of the present invention can be Figure 2-4 and Figure 2-4 Embodiments of the invention may perform operations different from those discussed above. Referring to the flowchart. Figure 12 The flowcharts illustrate a particular order of operations performed by certain embodiments of the present invention, but it should be understood that such order is exemplary (e.g., alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc.).

[0074] Although the present invention has been described by way of example and according to preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements (which are obvious to those skilled in the art). Therefore, the scope of the appended claims should be given the broadest interpretation so as to cover all such modifications and similar arrangements.

Claims

1. A method for enhancing the quality of a frame sequence by an enhancement engine, characterized in that: include: receiving a sequence of frames having at least one frame with degraded quality from a first stage circuit, wherein the degraded quality comprises at least one of non-uniform resolution and non-uniform frames per second; querying an information base using a query input based on the frame with degraded quality to obtain reference information of the frame with degraded quality as a query output; applying a neural network to the query input and the query output to generate an optimized frame; sending the enhanced frame sequence including the optimized frame to the second stage circuit; The information database is a database read and written by the enhancement engine, and the database stores multiple sets of key elements and corresponding values. Querying the information database using a query input based on the quality-degraded frame to obtain reference information of the quality-degraded frame as a query output includes: Generate a feature map based on the degraded frame using a given neural network; Using the elements of the feature graph to query the database for key elements to generate a set of hybrid weights, wherein each hybrid weight is used to indicate the similarity between each feature graph element and the corresponding key element; and Applying the set of blending weights to the corresponding value or causing the set of blending weights to be applied to the corresponding value to generate an element of the query output.

2. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 1, wherein: The neural network performs at least one of a super-resolution operation and a restoration operation on the degraded frame.

3. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 1, wherein: The information repository is accessed via the Internet.

4. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 1, wherein: The method of generating a feature map based on the degraded frame using a given neural network further includes: The given neural network is applied to the region with missing pixels in the degraded frame to generate the feature map.

5. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 1, wherein: The database stores a plurality of (key element, value) pairs, wherein the length of each key element is independent of the length of each corresponding value.

6. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 1, wherein: Also includes: computing an image embedding based on a first region of the object in a given frame in the sequence of frames; applying a convolution kernel weight to the image embedding or causing the convolution kernel weight to be applied to the image embedding to obtain new (key element, value) pairs; The database is extended or caused to be expanded by adding the new (key element, value) pair to the database.

7. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 6, wherein: Also includes: The frame with degraded quality includes a subsequent frame of the given frame, and querying an information base using a query input based on the frame with degraded quality to obtain reference information of the frame with degraded quality as a query output comprises: The expanded database is queried using a query input based on the subsequent frame to obtain information on a second region of the object in the subsequent frame, wherein the second region is visible in the given frame and is at least predicted to be visible in the subsequent frame.

8. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 1, wherein: Also includes: computing an image embedding based on a first region of the object in a given frame in the sequence of frames; applying a convolution kernel weight to the image embedding or causing the convolution kernel weight to be applied to the image embedding to obtain new (key element, value) pairs; Updates the database, or causes the database to be updated, by replacing an existing (key element, value) pair with the new (key element, value) pair.

9. The method for enhancing the quality of a frame sequence by an enhancement engine according to claim 6 or 8, characterized in that: Applying or causing the convolution kernel weights to be applied to the image embedding to obtain new (key element, value) pairs further comprises: Applying the convolution kernel weight to the image embedding or causing the convolution kernel weight to be applied to the image embedding to obtain an intermediate (key element, value) pair, and applying a multi-layer perceptron weight to the intermediate (key element, value) pair to obtain the new (key element, value) pair.

10. A system for enhancing the quality of a frame sequence, characterized in that include: a first stage circuit for transmitting a sequence of frames having at least one frame with degraded quality, wherein the degraded quality comprises at least one of non-uniform resolution and non-uniform frames per second; and an enhancement engine, configured to receive the frame sequence, and query an information base using a query input based on the frame with degraded quality to obtain reference information of the frame with degraded quality as a query output; applying a neural network to the query input and the query output to generate an optimized frame; sending the enhanced frame sequence including the optimized frame to the second stage circuit; The information database is a database read and written by the enhancement engine, and the database stores multiple sets of key elements and corresponding values. When a query input based on the quality-degraded frame is used to query the information database to obtain reference information of the quality-degraded frame as a query output, the enhancement engine executes: Generate a feature map based on the degraded frame using a given neural network; using elements of the feature map to query key elements in the database to generate a set of hybrid weights; and Applying the set of blending weights to the corresponding value or causing the set of blending weights to be applied to the corresponding value to generate an element of the query output.

11. The system for enhancing the quality of a frame sequence according to claim 10, wherein: The neural network performs at least one of a super-resolution operation and a restoration operation on the degraded frame.

12. The system for enhancing the quality of a frame sequence according to claim 10, wherein: The information repository is accessed via the Internet.

13. The system for enhancing the quality of a frame sequence according to claim 10, wherein: The method of generating a feature map based on the degraded frame using a given neural network further includes: The given neural network is applied to the region with missing pixels in the degraded frame to generate the feature map.

14. The system for enhancing the quality of a frame sequence according to claim 10, wherein: The database stores a plurality of (key element, value) pairs, wherein the length of each key element is independent of the length of each corresponding value.

15. The system for enhancing the quality of a frame sequence according to claim 10, wherein: The enhancement engine also performs: computing an image embedding based on a first region of the object in a given frame in the sequence of frames; applying a convolution kernel weight to the image embedding or causing the convolution kernel weight to be applied to the image embedding to obtain new (key element, value) pairs; The database is extended or caused to be expanded by adding the new (key element, value) pair to the database.

16. The system for enhancing the quality of a frame sequence according to claim 15, wherein: The frame with degraded quality includes a subsequent frame of the given frame, and when the query input based on the frame with degraded quality is used to query the information base to obtain reference information of the frame with degraded quality as the query output, the enhancement engine further performs: The expanded database is queried using a query input based on the subsequent frame to obtain information on a second region of the object in the subsequent frame, wherein the second region is visible in the given frame and is at least predicted to be visible in the subsequent frame.

17. The system for enhancing the quality of a frame sequence according to claim 10, wherein: The enhancement engine also performs: computing an image embedding based on a first region of the object in a given frame in the sequence of frames; applying a convolution kernel weight to the image embedding or causing the convolution kernel weight to be applied to the image embedding to obtain new (key element, value) pairs; Updates the database, or causes the database to be updated, by replacing an existing (key element, value) pair with the new (key element, value) pair.

18. The system for enhancing the quality of a frame sequence according to claim 10, wherein: The first-stage circuit, the enhancement engine, and the second-stage circuit are located in the same electronic device; or the first-stage circuit and the second-stage circuit are located in two electronic devices coupled to each other via a transmission network.

Images