Video processing circuit and method for performing SR operation

Abstract

The video processing circuit includes an input buffer, an online adaptation circuit, and an artificial intelligence (AI) super-resolution (SR) circuit. The input buffer receives input low-resolution (LR) frames and high-resolution (HR) frames from a video source through a network. The online adaptation circuit forms a training pair and uses the training pair to calculate an update of a representative feature, which has the characteristics of the input LR frame. Each training pair consists of one of the input LR frames and one of the HR frames. The AI ​​SR circuit receives the input LR frame from the input buffer and receives the representative feature from the online adaptation circuit. While calculating the update of the representative feature, the AI ​​SR circuit generates an SR frame for display from the input LR frame based on the representative feature. Each SR frame has a higher resolution than a corresponding one of the input LR frames.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to a video playback technology, and more particularly, to a video processing circuit and a method for performing super-resolution (SR) operations. Background Art

[0002] Today's image display devices are able to stream video over a network and enhance the streamed content before displaying it. Some devices are able to perform super-resolution (SR) operations on streamed content using image enhancement techniques. Super-resolution operations refer to upscaling a low resolution (LR) image to a high resolution image, for example, from an input image of (720x480) pixels to an output image of (3840x2160) pixels. Conventional image resizing techniques based on up / down-sampling will reduce image quality in terms of amount of blurring, noise, distortion, color condition, sharpness, contrast, etc.

[0003] Typical edge devices (such as TVs or smartphones) have limited computing power due to strict requirements on power consumption and thermal performance. Therefore, image enhancement operations on edge devices are usually based on algorithms and parameters pre-configured by the manufacturer of the device. This results in limited flexibility in adjusting the pre-configuration after the consumer uses the device. When the input image contains a variety of content and quality, the limited flexibility may have an adverse effect on the output image quality. Therefore, there is a need to improve the image enhancement operation to minimize the upsampling effect on the output image quality. Summary of the invention

[0004] In view of the above, one of the objectives of the present invention is to provide a video processing circuit and a method for performing a super-resolution (SR) operation to have image enhancement capability.

[0005] In one embodiment, a video processing circuit is provided, which includes an input buffer, an online adaptation circuit and an artificial intelligence (AI) super-resolution (SR) circuit. The input buffer receives input low-resolution (LR) frames and high-resolution (HR) frames from a video source via a network. The online adaptation circuit forms a training pair and uses the training pair to calculate an update of a representative feature, wherein the representative feature has a feature of the input low-resolution (LR) frame. Each training pair consists of one of the input low-resolution (LR) frames and one of the HR frames. The AI ​​super-resolution (SR) circuit receives the input low-resolution (LR) frame from the input buffer and receives the representative feature from the online adaptation circuit. While calculating the update of the representative feature, the AI ​​super-resolution (SR) circuit generates an SR frame for display from the input low-resolution (LR) frame based on the representative feature. Each SR frame has a higher resolution than a corresponding one of the input low-resolution (LR) frames.

[0006] In one embodiment, the representative feature includes one or more of the following: scene type, degradation type, degradation level and color condition.

[0007] In one embodiment, the representative features include information for the AI ​​SR circuit to update characteristics of the AI ​​model used to generate the SR frame.

[0008] In one embodiment, the incoming HR frames are received through the network at a lower frequency than the incoming LR frames.

[0009] In an embodiment, one or more of the training pairs include HR frames and LR frames having different content and different resolutions.

[0010] In one embodiment, the online adaptation circuit is operable to: identify the representative feature using a convolutional neural network (CNN).

[0011] In one embodiment, the AI ​​SR circuit is operable to generate the SR frame using a convolutional neural network (CNN).

[0012] In one embodiment, the online adaptation circuit is operable to: periodically receive the input HR frames; and pair each input HR frame with a plurality of LR frames of the input LR frames to form a plurality of training pairs.

[0013] In one embodiment, the online adaptation circuit is operable to: receive one or more of the input HR frames when a predetermined event is detected, wherein the predetermined event includes a scene change or an available network bandwidth exceeding a threshold.

[0014] In one embodiment, when a predetermined event is detected, the AI ​​SR circuit can be operated to: update the AI ​​model used to generate the SR frame, wherein the predetermined event includes one of the following: unstable network bandwidth, scene change, fixed time period and duration of each frame.

[0015] In another embodiment, a method for performing an SR operation is provided. An input low resolution (LR) frame and an HR frame are received from a video source via a network. Training pairs are formed, wherein each training pair consists of one of the input low resolution (LR) frames and one of the HR frames. An update of a representative feature is calculated using the training pair, the representative feature having a feature of the input low resolution (LR) frame. While calculating the update to the representative feature, an SR frame for display is generated from the input low resolution (LR) frame based on the representative feature. Each SR frame has a higher resolution than a corresponding one of the input low resolution (LR) frames.

[0016] In one embodiment, the representative feature includes one or more of the following: scene type, degradation type, degradation level and color condition.

[0017] In one embodiment, the representative features include information for updating characteristics of an AI model used to generate the SR frame.

[0018] In one embodiment, the incoming HR frames are received through the network at a lower frequency than the incoming LR frames.

[0019] In an embodiment, one or more of the training pairs include HR frames and LR frames having different content and different resolutions.

[0020] In one embodiment, the representative features are identified using a convolutional neural network (CNN).

[0021] In one embodiment, the SR frame is generated using a convolutional neural network (CNN).

[0022] In one embodiment, forming the training pairs further comprises: periodically receiving the input HR frames; and pairing each input HR frame with a plurality of LR frames of the input LR frames to form a plurality of training pairs.

[0023] In one embodiment, forming the training pair further comprises: receiving one or more of the input HR frames when a predetermined event is detected, wherein the predetermined event comprises a scene change or an available network bandwidth exceeds a threshold.

[0024] In one embodiment, the method further includes: updating the AI ​​model used to generate the SR frame when a predetermined event is detected, wherein the predetermined event includes one of the following: unstable network bandwidth, scene change, fixed time period and duration of each frame.

[0025] By reading the following description of specific embodiments in conjunction with the accompanying drawings, other aspects and features will become apparent to those of ordinary skill in the art. This disclosure is not intended to limit the present invention. The present invention is defined by the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are included to provide a further understanding of the embodiments of the present disclosure, and the accompanying drawings are incorporated into and constitute a part of the embodiments of the present disclosure. The accompanying drawings illustrate implementations of the embodiments of the present disclosure and, together with the description, are used to explain the principles of the embodiments of the present disclosure. It is understood that the drawings are not necessarily drawn to scale, as some components may be shown not to scale with the actual implementation size to clearly illustrate the concepts of the embodiments of the present disclosure.

[0027] Figure 1 FIG. 4 is a block diagram of a video processing circuit for performing SR operation according to an embodiment of the present invention.

[0028] Figure 2 FIG. 4 is a block diagram of a video processing circuit for performing SR operation according to another embodiment of the present invention.

[0029] Figure 3A and Figure 3B A device for forming and using online training pairs according to an embodiment of the present invention is shown.

[0030] Figure 4 FIG. 4 is a flowchart of a method for performing an SR operation according to an embodiment of the present invention.

[0031] Figure 5 An example of an apparatus operable to perform an SR operation according to an embodiment of the present invention is shown.

[0032] In the following detailed description, for the purpose of illustration, many specific details are set forth so that those skilled in the art can more thoroughly understand the embodiments of the present invention. However, it is apparent that one or more embodiments may be implemented without these specific details, and different embodiments may be combined as needed, and should not be limited to the embodiments listed in the accompanying drawings. DETAILED DESCRIPTION

[0033] The following description is a preferred embodiment of the present invention, which is only used to illustrate the technical features of the present invention, and is not used to limit the scope of the present invention. Certain words are used throughout the specification and claims to refer to specific components. It should be understood by those skilled in the art that manufacturers may use different names to refer to the same components. Therefore, the present specification and claims do not use the difference in name as a way to distinguish components, but use the difference in function of the components as a basis for distinction. The terms "component", "system" and "device" used in the present invention can be entities related to computers, wherein the computer can be hardware, software, or a combination of hardware and software. The terms "including" and "comprising" mentioned in the following description and claims are open-ended terms, so they should be interpreted as "including, but not limited to...". In addition, the term "coupled" means an indirect or direct electrical connection. Therefore, if a device is described as coupled to another device, it means that the device can be directly electrically connected to the other device, or indirectly electrically connected to the other device through other devices or connection means.

[0034] Corresponding numbers and symbols in the various figures of the drawings generally refer to corresponding parts unless otherwise indicated. The drawings are drawn to clearly illustrate the relevant parts of the embodiments and are not necessarily drawn to scale.

[0035] The term "substantially" or "approximately" as used herein means that within an acceptable range, a person skilled in the art can solve the technical problem to be solved and basically achieve the technical effect to be achieved. For example, "approximately equal to" means that a certain error from "completely equal to" is acceptable to a person skilled in the art without affecting the correctness of the result.

[0036] Embodiments of the present invention provide a video processing circuit that performs super-resolution (SR) operations on input frames based on online training. The video processing circuit includes a super-resolution (SR) engine that operates on low-resolution (LR) input frames using an artificial intelligence (AI) model and generates output frames (called SR frames) with a desired resolution (required resolution, or target resolution). The SR engine (also known as an AI super-resolution (SR) circuit) can reduce or eliminate artifacts in the input low-resolution (LR) frames. Each AI model is defined by characteristics such as parameters, structures, and operators. These features can be updated during operation based on online training pairs, where each training pair includes a low-resolution (LR) frame and a high-resolution (HR) frame. In some embodiments, embodiments of the present invention provide an online AI super-resolution engine and related operations, and in other embodiments, the present invention relates to an edge device with image enhancement capabilities to perform super-resolution operations.

[0037] Conventional super-resolution (SR) circuits usually rely only on pre-trained parameters, or in other words, offline training parameters, which are usually pre-stored in the device. The number of pre-trained parameters is limited due to limited storage capacity. Therefore, conventional super-resolution (SR) circuits cannot correctly process different types of input images with different image qualities.

[0038] The video processing circuit described herein also includes an online adaptation circuit that identifies representative features (or key features) from the online training pairs and provides the identified representative features to the AI ​​super-resolution (SR) circuit. Via the same communication network, the video processing circuit can receive online training pairs and input low-resolution (LR) frames in parallel from the same video source. In one embodiment, the online training pairs are received at a lower frequency than the input low-resolution (LR) frames to reduce the use of network bandwidth. In other words, the input low-resolution (LR) frames are received more frequently than the online training pairs.

[0039] In addition, the low-resolution (LR) frames in the online training pair can be a subset of the input low-resolution (LR) frames; therefore, compared with the pre-trained parameters, the online training pair generally has content and quality information that is more relevant to the input low-resolution (LR) frames. When the content and / or quality of the input frame changes, the content and / or quality of the online training pair also changes accordingly. Therefore, the representative features extracted from the online training pair can provide hints on how to enhance the input low-resolution (LR) frames. For edge devices (e.g., smart TVs, smart phones, IoT devices, etc.) with limited processing resources and storage capacity, the method described in this article can provide great flexibility based on real-time training with minimized computational overhead.

[0040] As used herein, the terms "LR" and "HR" are relative; that is, for the same display size (e.g., N square inches), a low-resolution (LR) frame has fewer pixels than a high-resolution (HR) frame. For example, for the same display size, a low-resolution (LR) frame may have (720x480) pixels and a high-resolution (HR) frame may have (3840x2160) pixels. It should be understood that for the same display size, the low-resolution (LR) frame and the high-resolution (HR) frame may have any number of pixels as long as the low-resolution (LR) frame has fewer pixels than the high-resolution (HR) frame. The resolution of a super-resolution (SR) frame is higher than the resolution of a low-resolution (LR) frame, and may be equal to or lower than the resolution of a high-resolution (HR) frame.

[0041] Figure 1 1 is a block diagram of a video processing circuit 100 according to an embodiment of the present invention. The video processing circuit 100 may be part of a device (also referred to as an edge device). Examples of the device may include: a television, a smartphone, a computing device, a network-connected device, a gaming device, an entertainment device, an Internet-of-things (IoT) device, or any device capable of processing and displaying images and / or videos.

[0042] In one embodiment, the video processing circuit 100 includes, among other components, an AI super-resolution (SR) circuit 120 coupled to an online adaptation circuit 140. The AI ​​super-resolution (SR) circuit 120 is also coupled to an input port 110 to receive a low-resolution input frame. (i.e., an input low-resolution (LR) frame). The AI ​​super-resolution (SR) circuit 120 may perform an SR operation on the input low-resolution (LR) frame according to one or more AI models. An example of an AI model is an artificial neural network, such as a convolutional neural network (CNN) or other machine learning or deep learning network. Examples of SR operations performed by the AI ​​super-resolution (SR) circuit 120 include, but are not limited to: CNN operations, machine learning operations, or deep learning operations. For each input low-resolution (LR) frame, the AI ​​super-resolution (SR) circuit 120 generates an output frame of higher resolution, referred to as a super-resolution (SR) frame. The AI ​​super-resolution (SR) circuit 120 outputs the SR frame to an output port 130. The SR frame is sent to a display for viewing by a user.

[0043] In one embodiment, the online adaptive circuit 140 is coupled to another input port 115 to receive an online training pair. Each online training pair includes a low-resolution (LR) frame and a corresponding high-resolution (HR) frame. The low-resolution (LR) frame included in the online training pair may be one of the input low-resolution (LR) frames processed by the AI ​​super-resolution (SR) circuit 120 for generating an SR frame. The corresponding HR frame included in the online training pair may be a high-resolution (HR) frame or a plurality of close high-resolution (HR) frames, for example, if the high-resolution frame HR1 and the high-resolution frame HR2 correspond to the same scene type respectively, they may be considered close. In one embodiment, the low-resolution (LR) frames in the multiple online training pairs may be a subset of the input low-resolution (LR) frames. In one embodiment, the high-resolution (HR) frames in the multiple online training pairs may be a subset of the input high-resolution (HR) frames, for example, in Figure 3A In the example of , if HR1 and HR2 are close, HR1 can also be used to assist the online training of the LR frame in the online training pair corresponding to HR2. In one embodiment, the AI ​​super-resolution (SR) circuit 120 and the online adaptation circuit 140 can receive their corresponding low-resolution (LR) frames in parallel.

[0044] The online adaptation circuit 140 performs online training using the online training pair. In one embodiment, the online adaptation circuit 140 can identify (e.g., detect or extract) representative features from the online training pair and provide the representative features to the AI ​​super-resolution (SR) circuit 120 to improve the performance of the SR operation. The online adaptation circuit 140 performs online training, and in parallel, the AI ​​super-resolution (SR) circuit performs the SR operation. The online adaptation circuit 140 can perform non-AI calculations to detect some representative features and / or perform AI operations (e.g., CNN operations, machine learning operations, or deep learning operations) to detect some representative features.

[0045] In one embodiment, the representative features may indicate characteristics of the input low-resolution (LR) frame, wherein the features may include, but are not limited to, scene type, degradation type, degradation level, color condition, and other indications of image content and / or quality. For example, scene type may include natural scene, computer-generated (CG) scene, etc.; degradation type and level may include image noise type and level, video compression parameters, blur, texture aliasing, edge / contour jaggy, etc.; color condition may include color saturation, contrast, sharpness, etc. The representative features may include global features or local features of a frame or frame sequence, and / or high-level features (e.g., scene type) or low-level features (e.g., noise level) of a frame or frame sequence.

[0046] Additionally or alternatively, the representative features may be updated to indicate updated parameters and / or structures of the AI ​​model, wherein the AI ​​super-resolution (SR) circuit 120 uses the AI ​​model to generate SR frames. The AI ​​super-resolution (SR) circuit 120 may update the internal layers and / or output characteristics of the neural network based on the representative features. For example, the update may be applied to the structure and / or parameters of feature maps, activation layers, filter kernels, etc. The update is performed periodically, for example, per frame or at a fixed period. Alternatively, the update may be performed when a predetermined condition is detected. Examples of the predetermined condition may include, but are not limited to, scene changes, unstable internet bandwidth, etc.

[0047] Therefore, the AI ​​super-resolution (SR) circuit 120 can reduce or eliminate artifacts in the input low-resolution (LR) frame using the identified representative features, wherein the representative features characterize the features in the input low-resolution (LR) frame. In addition, the AI ​​super-resolution (SR) circuit 120 can reduce or remove artifacts in the input low-resolution (LR) frame using one or more AI models updated online based on the representative features obtained from the online training pair.

[0048] Figure 2 FIG. 2 is a block diagram of a video processing circuit 200 according to another embodiment of the present invention. The video processing circuit 200 includes Figure 1 The AI ​​super-resolution (SR) circuit 120 and the online adaptation circuit 140 in the embodiment. In this embodiment, the output of the AI ​​super-resolution (SR) circuit 120 is coupled to the picture quality (PQ) engine 250 to further enhance the quality of the super-resolution (SR) frame generated by the AI ​​super-resolution (SR) circuit 120. In one embodiment, the picture quality (PQ) engine 250 performs image enhancement operations, including but not limited to: focus peaking, sharpness enhancement, saturation tone mapping, etc. The output of the picture quality (PQ) engine 250 is coupled to the output port 130, which provides the enhanced SR frame to the display for viewing by the user.

[0049] Figure 3AFIG. 3 is a block diagram of an apparatus 310 including a video processing circuit 300 according to an embodiment of the present invention. The video processing circuit 300 may be Figure 1 The video processing circuit 100 or Figure 2 3. The video processing circuit 300 is coupled to an input buffer 320 and a display 330. The device 310 receives input video from a video server 350 via a communication network 340. The video server 350 provides a video streaming service for videos of multiple resolutions that can be selected by the device 310 (e.g., a user of the device 310). Figure 3A In the example of , the device 310 selects a video stream with two resolutions, namely a low resolution (LR) frame and a high resolution (HR) frame. The device 310 can configure the stream operation so that the high resolution (HR) frame is received less frequently than the low resolution (LR) frame, that is, the low resolution (LR) frame is received more frequently. In one embodiment, the device 310 can receive HR frames periodically, for example, periodically receiving HR frames at a predetermined fixed time period. Alternatively, the device 310 can receive HR frames when a specific trigger event (such as a scene change) is detected, when the available network bandwidth exceeds a threshold or another trigger event. The device 310 receives one or more HR frames at a fixed period or when the aforementioned event is detected. Optionally, the device 310 can receive a predetermined number of consecutive HR frames at a fixed period or when the aforementioned event is detected.

[0050] exist Figure 3AIn the example of , two HR frames (HR1 and HR2) and four low resolution (LR) frames (LR1, LR2, LR3 and LR4) are received in the same time period. HR1 and LR1 can have the same content but different resolutions. The term "content" in this article refers to scenes, image objects and backgrounds, etc. HR1 and LR2 can have similar or different content (for example, HR1 contains images of cats and LR2 contains images of houses) and have different resolutions. Similarly, HR1 and LR3 can have similar or different content and different resolutions. In this example, HR1 can be paired with three low resolution (LR) frames to form three training pairs. For example (HR1, LR1), (HR1, LR2) and (HR1, LR3), regardless of the degree of similarity or difference in content between the high resolution (HR) frame and the low resolution (LR) frame in each training pair. The same applies to HR2, which can be paired with one or more low resolution (LR) frames to form one or more training pairs, for example, (HR2, LR4). In one embodiment, if the high-resolution frame HR2 is close to HR1 (wherein, the closeness of HR1 and HR2 generally refers to the same scene type involved, for example, having the same living room background but different shooting angles), the high-resolution frame HR1 can also be used to assist the online training of the LR frame associated with HR2, such as Figure 3B As shown, for example, an online training pair (HR1, LR4) is formed using a high-resolution frame HR1 that is close to the previous one. That is, if the previous HR frame is close to the current HR frame, the previous HR frame can be further used to assist the online training of the current LR frame. Since the high-resolution (HR) frame contains higher quality and more detailed information than the low-resolution (LR) frame, the training pair can be effectively trained regardless of the corresponding content in the HR frame and the low-resolution (LR) frame that constitute the training pair.

[0051] The training pair includes a corresponding pair of high-resolution (HR) frames and low-resolution (LR) frames. The training pair disclosed in the present specification is an "online training pair" because the training is performed simultaneously with SR generation, wherein the training uses an input low-resolution (LR) frame and a corresponding HR frame (e.g., one of a plurality of HR frames), and the SR generation produces an SR frame having the same content as the input LR frame but a higher resolution. More specifically, when the AI ​​super-resolution (SR) circuit 120 processes an input low-resolution (LR) frame (e.g., LR1-LR4) for display, at the same time, the online adaptation circuit 140 performs training (e.g., identifying or updating representative features) using the same input low-resolution (LR) frame and the HR frame paired therewith. During video stream processing, the online adaptation circuit 140 can continuously calculate updates to the representative features and output the updated representative features to the AI ​​super-resolution (SR) circuit 120. In some embodiments, the representative features calculated by the online adaptation circuit 140 based on the input frames (e.g., LR1-LR4 and HR1-HR2) are received by the AI ​​super-resolution (SR) circuit 120 after LR1-LR4 are processed into SR frames. That is, the AI ​​super-resolution (SR) circuit 120 can generate SR frames from LR1-LR4 based on the representative features calculated based on those input frames before LR1-LR4 and HR1-HR2.

[0052] In one embodiment, the device 310 uses an input buffer 320 to buffer frames received from the network 340. The input buffer 320 can buffer the received low-resolution (LR) frames and high-resolution (HR) frames. The corresponding low-resolution (LR) frames and high-resolution (HR) frames form an online training pair, which is sent from the input buffer 320 to the online adaptation circuit 140. The input buffer 320 also sends low-resolution (LR) frames (including those LR frames in the online training pair) as input LR frames to the AI ​​super-resolution (SR) circuit 120. In this example, the low-resolution (LR) frames in the multiple online training pairs are a subset of the input low-resolution (LR) frames.

[0053] Since device 310 obtains the online training pairs and the input low-resolution (LR) frames from the same video source via the same communication network path, the representative features identified from the online training pairs provide a strong indication of the features in the input low-resolution (LR) frames, as well as the structure / parameters of the AI ​​model that is most suitable for the SR operation.

[0054] In some embodiments, the video processing circuit 300 may be an AI processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or another general or special processing circuit. In one embodiment, the AI ​​processor may be used to perform CNN calculations to detect representative features and process the current frame. In one embodiment, the video processing circuit 300 may be implemented as a system-on-a-chip (SoC). In some embodiments, the video processing circuit 300 may be implemented in more than one chip in the same electronic device.

[0055] In one embodiment, the AI ​​super-resolution (SR) circuit 120 includes a CNN accelerator to perform CNN operations on input low-resolution (LR) frames. The CNN accelerator includes hardware components dedicated to accelerating neural network operations through convolution operations, fully connected operations, activation, pooling, normalization, element-by-element mathematical calculations, etc. In some embodiments, the CNN accelerator includes multiple computing units and memories (e.g., static random access memory (SRAM), wherein each computing unit also includes multiplier and adder circuits for performing mathematical operations, such as multiply-and-accumulate (MAC) operations to speed up convolution, activation, pooling, normalization, and other neural network operations. The CNN accelerator can perform fixed-point and floating-point neural network operations. The CNN accelerator can perform SR operations to output SR frames with enhanced image quality.

[0056] In one embodiment, the AI ​​super-resolution (SR) circuit 120 performs a CNN operation according to a CNN model (which is an example of an AI model). In addition, the CNN operation includes convolution of an input feature map with a kernel filter. For example, an input feature map from a previous layer of the CNN operation can be convolved with a kernel filter to generate an output feature map to the next layer. The characteristics of the AI ​​model (such as the hierarchical structure of the neural network and the parameters of the kernel filter) can be updated by the results of online training generated by the online adaptive circuit 120.

[0057] Figure 4 FIG. 4 is a flowchart of a method 400 for performing an SR operation according to an embodiment of the present invention. For example, the method 400 may be combined with Figure 1 , Figure 2 , Figures 3A, 3B and Figure 5It should be understood that this embodiment is only an example, and other devices or circuits with video processing capabilities can perform the method 400.

[0058] The method 400 starts at step 410 when the video processing circuit receives input low resolution (LR) frames and HR frames from a video source via a network. At step 420, the video processing circuit forms training pairs, each training pair formed by one of the input low resolution (LR) frames and one of the HR frames. At step 430, the video processing circuit uses the training pairs to calculate updates to representative features characterizing the input low resolution (LR) frames. While calculating updates to the representative features, the video processing circuit generates SR frames for display from the input low resolution (LR) frames based on the representative features at step 440. Each SR frame has a higher resolution than a corresponding one of the input low resolution (LR) frames.

[0059] In one embodiment, the representative features include one or more of: scene type, degradation type, degradation level, and color condition. In one embodiment, the representative features include information for an AI super-resolution (SR) circuit to update characteristics of an AI model used to generate SR frames.

[0060] In one embodiment, HR frames are received less frequently than input low resolution (LR) frames.In one embodiment, one or more training pairs include HR frames and low resolution (LR) frames, which may have different content in addition to having different resolutions.

[0061] In one embodiment, the video processing circuit includes an AI super-resolution (SR) circuit and an online adaptation circuit. The online adaptation circuit is used to identify representative features using CNN. The AI ​​super-resolution (SR) circuit is used to generate SR frames using CNN. The online adaptation circuit can periodically receive HR frames, and can pair each HR frame with multiple LR frames of the input low-resolution (LR) frame to form multiple training pairs. When a predetermined trigger event is detected (including a scene change or the available network bandwidth exceeds a threshold), the online adaptation circuit can receive the HR frame. When a predetermined trigger event is detected, the AI ​​super-resolution (SR) circuit can update the AI ​​model for generating SR frames. The event includes one of the following: unstable network bandwidth, scene change, fixed time periods, and per frame time.

[0062] Figure 5An example of an apparatus 500 is shown according to an embodiment of the present invention. An example of apparatus 500 is a television that receives a video of a low resolution (e.g., 720x480 pixels) and performs an SR operation to scale the video to a higher resolution (e.g., 3840x2160 pixels) for display on a television screen. For example, apparatus 500 receives an input low resolution (LR) frame and an input high resolution (HR) frame from a network entity 570 via a network 580. Alternatively, apparatus 500 may be a smart phone, a computing device, a network-connected device, a gaming device, an entertainment device, an Internet of Things (IoT) device, or any device capable of processing and displaying images and / or videos.

[0063] The apparatus 500 includes processing hardware 510, which may include Figure 1 , Figure 2 and any one of the video processing circuits 100, 200, and 300 in FIGS. 3A and 3B. In one embodiment, the processing hardware 510 may include one or more processors, such as one or more of the following: a central processing unit (CPU), a GPU, a digital processing unit (DSP), an AI processor, a multimedia processor, other general and / or special processing circuits. In one embodiment, the processing hardware 510 may include a hardware accelerator, such as a CNN accelerator. In one embodiment, in the above embodiment, the processing hardware 510 includes an AI super-resolution (SR) circuit 120 and an online adaptation circuit 140.

[0064] The apparatus 500 also includes a memory and a buffer 520 coupled to the processing hardware 510. In one embodiment, the memory and the buffer 520 may include Figure 3A 320 in the input buffer. The memory and buffer 520 may include storage devices such as dynamic random access memory (DRAM), SRAM, flash memory and other non-transitory machine-readable storage media; for example, volatile or non-volatile storage devices. The memory and buffer 520 may further include storage devices, for example, any type of solid-state or magnetic storage device. In some embodiments, the memory and buffer 520 may store instructions that, when executed by the processing hardware 510, cause the processing hardware 510 to perform the aforementioned operations for generating an SR frame, such as Figure 4 Method 400.

[0065] The device 500 may also include a display panel 530 to display information such as images, videos, messages, web pages, games, text, and other types of text, image and video data. The device 500 may also include audio hardware 540, such as a microphone and a speaker, for receiving and generating sound.

[0066] In some embodiments, the device 500 may also include a network interface 550 to connect to a wired and / or wireless network for sending and / or receiving voice, digital data and / or media signals. It will be appreciated that the present invention is simplified for the purpose of illustration. Figure 5 In another embodiment, other additional hardware components may be included.

[0067] Already referenced Figure 1 , Figure 2 , Figure 3A , Figure 3B and Figure 5 The exemplary embodiments describe Figure 4 However, it should be understood that Figure 4 The operation of the flowchart can be performed by Figure 1 , Figure 2 , Figure 3A , Figure 3B and Figure 5 Embodiments of the present invention other than the embodiments of the present invention are performed, and Figure 1 , Figure 2 , Figure 3A , Figure 3B and Figure 5 Embodiments of the invention may perform operations different from those discussed with reference to the flowcharts. Figure 4 The flowcharts illustrate a particular order of operations performed by certain embodiments of the present invention, but it should be understood that such order is exemplary (e.g., alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc.).

[0068] Although the present invention has been described by way of example and in terms of preferred embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. Instead, it is intended to cover various modifications and similar structures (as will be apparent to those skilled in the art), for example, combinations or replacements of different features in different embodiments. Therefore, the scope of the appended claims should be given the broadest interpretation to cover all such modifications and similar structures.

Claims

1. A video processing circuit, characterized in that: The video processing circuit includes an input buffer, an online adaptation circuit and an artificial intelligence AI super-resolution SR circuit; The input buffer is used to receive input low-resolution LR frames and input high-resolution HR frames from a video source via a network, wherein the input HR frames are received at a lower frequency than the input LR frames; The online adaptation circuit is used to: form online training pairs, each online training pair consisting of one of the input LR frames and one of the input HR frames; and, using the online training pairs, calculate updates of representative features characterizing the input LR frames; and, The AI ​​SR circuit is used to: receive the input LR frame from the input buffer to generate an SR frame for display using an AI model; and update the AI ​​model based on the update of the representative feature, wherein each SR frame has a higher resolution than a corresponding one of the input LR frames.

2. The video processing circuit according to claim 1, characterized in that: The representative feature includes one or more of the following: scene type, degradation type, degradation level and color condition.

3. The video processing circuit according to claim 1, characterized in that: The representative features include information for the AI ​​SR circuit to update characteristics of the AI ​​model used to generate the SR frame.

4. The video processing circuit according to claim 1, characterized in that: One or more of the training pairs include HR frames and LR frames with different contents and different resolutions.

5. The video processing circuit according to claim 1, characterized in that: The online adaptation circuit can be operated as follows: The convolutional neural network (CNN) is used to identify the representative features.

6. The video processing circuit according to claim 1, characterized in that: The AI ​​SR circuit can be operated as: The SR frame is generated using a convolutional neural network (CNN).

7. The video processing circuit according to claim 1, characterized in that: The online adaptation circuit can be operated as follows: Periodically receiving the input HR frame; as well as, Each input HR frame is paired with multiple LR frames of the input LR frame to form multiple training pairs.

8. The video processing circuit according to claim 1, characterized in that: The online adaptation circuit can be operated as follows: One or more of the input HR frames are received when a predetermined event is detected, wherein the predetermined event includes a scene change or an available network bandwidth exceeding a threshold.

9. The video processing circuit according to claim 1, characterized in that: When a predetermined event is detected, the AI ​​SR circuit can be operated to: update the AI ​​model used to generate the SR frame, wherein the predetermined event includes one of the following: unstable network bandwidth, scene change, fixed time period and duration of each frame.

10. A method for performing a super-resolution (SR) operation, characterized in that: include: Receiving input low-resolution LR frames and input high-resolution HR frames from a video source over a network, wherein the input HR frames are received at a lower frequency than the input LR frames; forming online training pairs, each online training pair consisting of one of the input LR frames and one of the input HR frames; Using the online training to calculate and update representative features of the input LR frame; and, An SR frame for display is generated from the input LR frame using an AI model, and the AI ​​model is updated based on the update of the representative feature, wherein each SR frame has a higher resolution than a corresponding one of the input LR frames.

11. The method according to claim 10, characterized in that The representative feature includes one or more of the following: scene type, degradation type, degradation level and color condition.

12. The method according to claim 10, characterized in that The representative features include information for updating characteristics of the AI ​​model used to generate the SR frame.

13. The method according to claim 10, characterized in that One or more of the training pairs include HR frames and LR frames with different contents and different resolutions.

14. The method according to claim 10, characterized in that The representative features are identified using a convolutional neural network (CNN).

15. The method according to claim 10, characterized in that The SR frame is generated using a convolutional neural network (CNN).

16. The method according to claim 10, characterized in that Forming the training pair further includes: periodically receiving the input HR frame; and, Each input HR frame is paired with multiple LR frames of the input LR frame to form multiple training pairs.

17. The method according to claim 10, characterized in that Forming the training pair also includes: One or more of the input HR frames are received when a predetermined event is detected, wherein the predetermined event includes a scene change or an available network bandwidth exceeding a threshold.

18. The method according to claim 10, characterized in that The method further includes: When a predetermined event is detected, the AI ​​model used to generate the SR frame is updated, wherein the predetermined event includes one of the following: unstable network bandwidth, scene change, fixed time period and duration of each frame.

Images