Image-guided adjustment for super-resolution operations
By using image processing circuits in edge devices, detecting representative features and dynamically adjusting AI model parameters, the problem of image quality degradation in edge devices in super-resolution operations is solved, and higher quality high-resolution image generation and flexibility are achieved sex.
Patent Information
- Application Number
- CN202110260590.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-12-10
- Filing Date
- 2021-03-10
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-03-10
AI Technical Summary
When existing edge devices perform super-resolution operations, image quality is prone to decline, and due to limited computing power, they cannot be flexibly adjusted to adapt to the content and quality of different input images, resulting in poor output image quality.
An image processing circuit is provided, including storing a set of pre-trained AI model parameters, an image guidance module for detecting representative features, a parameter determination module for adjusting AI model parameters, and an SR engine for generating high-resolution images. The AI model parameters are dynamically adjusted to suit different input images by detecting representative features in the image sequence.
This method can automatically adapt to the content and quality of the input image, generate higher quality high-resolution images, reduce artifacts, and maintain flexibility and low computing overhead with limited processing resources and storage capacity.
Smart Images

Figure CN113393375B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to an edge device having image processing capabilities for performing super-resolution operations. Background Art
[0002] Modern image display devices often enhance images before displaying them. Some devices can perform super-resolution through image enhancement. Super-resolution refers to the task of upscaling a low-resolution (LR) image to a high-resolution image (called a super-resolution (SR) image); for example, from an input image of (720x480) pixels to an output image of (3840x2160) pixels. Super-resolution operations may cause image quality degradation, including the amount of blur, noise, distortion, color condition, clarity, contrast, etc.
[0003] Typical edge devices, such as TVs or smartphones, have limited computing power due to stringent requirements on power consumption and thermal performance. Therefore, image enhancement operations on edge devices are usually based on algorithms and parameters pre-configured by the device manufacturer. Consumers have limited flexibility in adjusting the pre-configuration after using the device. When the input image contains a variety of content and quality, the limited flexibility may have an adverse impact on the output image quality. Therefore, there is a need to improve the image enhancement operation to minimize the impact on the output image quality. Summary of the invention
[0004] In one embodiment, an image processing circuit is provided for performing a super-resolution (SR) operation. The image processing circuit includes: a memory for storing multiple parameter sets of multiple artificial intelligence (AI) models; an image guidance module for detecting representative features in an image sequence, the time series including a current frame and past frames within a time window; a parameter determination module for adjusting the parameters of one or more AI models based on the measurement results of the representative features; and an SR engine for processing the current frame using one or more AI models with adjusted parameters to generate a high-resolution image for display.
[0005] In another embodiment, a method for performing an SR operation is provided. The method includes: detecting representative features in an image sequence including a current frame and past frames within a time window; adjusting parameters of one or more AI models based on the measurement results of the representative features; and processing the current frame using the one or more AI models with the adjusted parameters to generate a high-resolution image for display.
[0006] Other aspects and features will become apparent to those of ordinary skill in the art by reading the following description of specific embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The present invention is illustrated by way of example and not limitation in the accompanying drawings, in which like reference numerals represent similar elements. It should be noted that different references to "one" embodiment in the present disclosure are not necessarily the same embodiment, and such references mean at least one. In addition, when specific features, structures, or characteristics are described in conjunction with an embodiment, it is proposed that such features, structures, or characteristics may be implemented in conjunction with other embodiments within the knowledge of a person of ordinary skill in the art, whether or not explicitly described.
[0008] Figure 1 A block diagram illustrating an adjustable super-resolution (SR) operation according to one embodiment.
[0009] Figure 2 A block diagram of an image processing circuit performing an SR operation according to one embodiment is shown.
[0010] Figure 3A A parameter fusion module according to one embodiment is shown.
[0011] Figure 3B A parameter fusion module according to another embodiment is shown.
[0012] Figure 3C A parameter fusion module according to yet another embodiment is shown.
[0013] Figure 4 An example of SR operations performed by a SR engine according to one embodiment is shown.
[0014] Figure 5 An example of SR operations performed by a SR engine according to another embodiment is shown.
[0015] Figure 6 is a flow chart illustrating a method for SR operation according to one embodiment.
[0016] Figure 7 An example of a device according to one embodiment is shown. DETAILED DESCRIPTION
[0017] In the following description, many specific details are set forth. However, it should be understood that embodiments of the present invention can be implemented without these specific details. In other cases, well-known circuits, structures, and techniques are not shown in detail so as not to obscure the understanding of this specification. However, it will be understood by those of ordinary skill in the art that the present invention can be implemented without these specific details. With the included description, those of ordinary skill in the art will be able to implement appropriate functions without excessive experimentation.
[0018] Embodiments of the present invention provide an image processing circuit that can automatically adapt its operation to the content and quality of the input image. The image processing circuit includes a super-resolution (SR) engine that operates on a low-resolution (LR) input frame using an AI model with dynamically adjustable parameters and outputs an SR frame with the required high resolution. The SR engine can remove artifacts in the input image. The parameters of the AI model are pre-trained and stored in a memory or register of the image processing circuit. Pre-training can be performed using a training set including LR images and corresponding high-resolution (HR) images. These parameters can be adjusted at runtime based on representative features detected in the input image. The adjustability of these parameters enables the SR engine to adapt to various input image content and quality and generate higher quality output images.
[0019] The terms "LR" and "HR" are relative; that is, for the same display size (e.g., N square inches), an LR image has fewer pixels than an HR image. For example, for the same display size, an LR image may have (720x480) pixels and an HR image may have (3840x2160) pixels. It should be understood that for the same display size, the LR image and the HR image may have any number of pixels as long as the LR image has fewer pixels than the HR image. The resolution of an SR image is higher than that of an LR image, and may be the same or lower than that of an HR image. In the following description, the terms "image" and "frame" are used interchangeably.
[0020] Typically, pre-trained parameters are obtained from an offline training process using a limited number and types of training images. Therefore, the pre-trained parameters usually do not cover input images with different content and quality. Due to the limited storage capacity in the image processing circuit, the number of pre-trained parameters may be limited. Conventional SR circuits that rely solely on pre-trained parameters may not be able to process different types of input images with different image quality levels. As a result, conventional SR circuits may not be able to remove artifacts present in the input image.
[0021] According to the embodiments described herein, the SR engine uses one or more AI models in performing SR operations. Each AI model is characterized or defined by a set of AI operations and pre-trained parameters associated with the AI operations. In one embodiment, two or more pre-trained parameter sets are weighted and combined to form an adjusted parameter set for use by the SR engine. The weighting of the parameters may be based on the measurement results of representative features detected in the input image sequence. In one embodiment, the input image sequence may include a predetermined or configurable number of frames in the video. The input image sequence may include the current frame and the past frames within the time window, such as frame (tn) to frame (t), where t is a time unit and n is a positive integer. The time window may be a continuously sliding time window, so that the parameters can be adjusted for each time t input frame with reference to frame t to (tn). As the content and / or quality of the input image changes, the parameters of the AI model can be adjusted accordingly without having to retrain the AI model. For edge devices (e.g., smart TVs, smart phones, IoT devices, etc.) with limited processing resources and storage capacity, the method described herein has great flexibility in runtime and minimal computational overhead.
[0022] Figure 1 A block diagram of an adjustable super-resolution (SR) operation according to one embodiment is shown. The AI training module 180 uses a set of training sets in the training database 181 to train the AI model. Each training set contains an LR image and its corresponding HR image. The training database 181 includes multiple (e.g., N) training sets of different training set types. The training set type may indicate or scene type, degradation type, color condition, compression type, resolution or another indicator of image content or quality. In addition, the training set type may also indicate a level, an index, a score or a measurement result of another indicator of scene type, degradation type, color condition, compression type, resolution or image content or quality (such as peak signal to noise ratio, structural similarity). The output of the AI training module 180 is a set of parameter sets. For example, N parameter sets, where N can be any positive integer greater than 1. Each parameter set contains parameters of an AI model for SR operation. Examples of AI models include, but are not limited to, convolutional neural networks (CNNs), machine learning networks, deep learning networks, and the like. Each parameter set may be labeled with the type of training set it was trained on; for example, when parameter set P1 is generated from corresponding training sets of LR and HR images containing a CG scene, parameter set P1 may be labeled as having a computer-generated (CG) scene.
[0023] In one embodiment, the training is performed in an offline training phase. In one embodiment, the offline training can be performed by a server, a cloud computing system, a computer, or another computing platform. The N parameter sets can be stored in the memory 120 of the edge device performing the SR operation or downloaded to the memory 120 of the edge device performing the SR operation. The memory 120 may include one or more of a volatile and / or non-volatile random access memory (RAM) device, a read-only memory (ROM) device, or any combination of the above.
[0024] Examples of edge devices (also referred to as devices) may include: televisions, smartphones, computing devices, network-connected devices, gaming devices, entertainment devices, Internet-of-things (IoT) devices, or any device capable of processing and displaying images and / or video.
[0025] The online processing phase begins when the device receives a series of input frames through the input port. The input frames are LR images. The sequence of input frames includes the current frame (e.g., frame (t)) and the past frames within a time window. For example, frame (tn) to frame (t), where t is a time unit and n is a configurable positive integer. The time window can be a continuously sliding time window so that parameters can be adjusted for each time t input frame with reference to t frame to (tn) frame. Optionally, the time window can be applied to the input stream every n frames with a configurable period or based on detected events.
[0026] The online processing stage is performed on the device. During the online processing stage, the image guidance module 110 detects and / or measures representative features from the content and / or quality of the input frame sequence within the time window. In one embodiment, the image guidance module 110 may calculate a linear combination or a statistical combination of the input frames to obtain information about the frames, including image content, image quality, etc. Therefore, any change in image content, image quality and other image indicators, types, scores, levels, etc. within the time window can be detected and measured. The image guidance module 110 then performs detection and / or measurement of representative features on the input frame or a combination of input frames. In one embodiment, the image guidance module 110 performs statistical operations and / or AI operations to detect and / or measure representative features. Examples of AI operations include, but are not limited to, convolutional neural network (CNN) operations, machine learning operations, deep learning operations, etc.
[0027] In one embodiment, the representative features detected and / or measured by the image guidance module 110 may include at least a classification of image content, such as a scene type, and an indication of image quality, such as an image quality index. Other representative features may include, but are not limited to, predefined scene types (e.g., CG images or real images, street scenes or sky scenes, etc.), image quality indicators (such as Peak signal to Noise Ratio, structural similarity), degradation type, degradation level, color condition, image resolution, noise level, noise type, video compression parameters, codec artifact level, color saturation, clarity, contrast, etc.
[0028] The parameter determination module 130 receives the detected representative features as input. In one embodiment, the parameter determination module 130 identifies one or more parameter sets in the memory 120 based on the representative features. For example, the representative feature can be an image quality measurement result, which is an image quality indicator (such as Peak signal to Noise Ratio, structural similarity), such as image quality indicator, degradation type, degradation level, color condition, image resolution, noise level, noise type, etc. When the representative feature is the image quality indicator q, the parameter determination module 130 selects a parameter set from the memory 120, which is pre-trained for images with image quality index equal to q or close to q (i.e., within a predetermined threshold of q). If the memory 120 has two parameter sets pre-trained with image quality indexes q1 and q2, where q1<q<q2, the parameter determination module 130 can output a weighted combination of the two parameter sets as the adjusted parameters. When the representative feature includes When the image quality measurement results and the classification of the image content (e.g., the image quality index q of the CG scene) are determined, the pre-trained parameter set P1 is identified as having a CG scene, and another pre-trained parameter set P2 is identified as having an image quality index q. Then, the parameter determination module 130 generates adjusted parameters as output by adjusting the parameters from sets P1 and P2 based on the measurement results of the representative features (e.g., the image quality index q in the CG scene). The adjusted parameters can be used as new parameters for one or more AI models of the SR operation. In some embodiments, a weighted combination of two or more parameter sets can be calculated and used as new parameters for the SR operation.
[0029] During the online processing stage, the SR engine 140 in the device receives the adjusted parameters and the low-resolution current frame (e.g., frame (t)). Using one or more AI models and the adjusted parameters generated by the parameter determination module 130, the SR engine 140 performs an SR operation on the current frame. In some embodiments, the SR engine 140 can remove artifacts in the current frame using one or more AI models and parameters adjusted based on the scene type and image quality indicators measured from the input image sequence. In some embodiments, the SR engine 140 can remove artifacts in the current frame using one or more AI models and parameters adjusted or selected based on representative features detected or measured in the input image sequence.
[0030] An example of an SR operation performed by the SR engine 140 is a CNN operation. The SR engine 140 may also perform other machine learning operations and deep learning operations. The SR engine 140 generates a high-resolution output frame. The output frame may be sent to a display through an output port.
[0031] Figure 2 FIG. 2 is a block diagram of an image processing circuit 200 that performs an SR operation according to an embodiment. The image processing circuit 200 includes an image guiding module 110, a parameter determination module 130, an SR engine 140, and a memory 120 that stores a set of parameters, such as Figure 1 As described in the example. In some embodiments, the image processing circuit 200 may be an AI processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or other general or special processing circuit. In one embodiment, the image guidance module 110 and the SR module 140 are part of an AI processor that is operable to perform CNN calculations for detecting representative features and processing the current frame. In one embodiment, the image processing circuit 200 may be implemented as a system-on-a-chip (SoC). In some embodiments, the image processing circuit 200 may be implemented in more than one chip in the same electronic device.
[0032] exist Figure 2In an embodiment of the present invention, the parameter determination module 130 further includes a parameter weight generator 210 and a parameter fusion module 220. The parameter weight generator 210 receives representative features detected and / or measured by the image guidance module 110. In one example, the image guidance module 110 may output representative features with measurement results, for example, a noise level σ, a blur level γ, and a JPEG compression level α, where σ, γ, and α are numerical values. In another example, the image guidance module 110 may output representative features with scene classification scores for a first scene type and a second scene type. In yet another example, the image guidance module 110 may output representative features with image condition measurement results of brightness, saturation, and contrast. The parameter weight generator 210 generates weights for representative features based on the measurement results.
[0033] The parameter determination module 130 also includes a parameter fusion module 220, which applies weights to parameter sets associated with representative features. In one embodiment, the parameter fusion module 220 uses weights to calculate weighted sums. For example, the parameter weight generator 210 may generate weights w1, w2, w3, ..., wN for N different parameter sets, respectively, and the parameter fusion module 220 may calculate the weighted sum of the N parameter sets to generate a new parameter set Pnew. For example, the parameter weight generator 210 may generate a weight w1 for the noise level φ, and a weight w2 for the blur level φ. The parameter fusion module 220 may calculate the weighted sum Pnew = (w1×Pi+w2×Pj), where Pi is a parameter set for the noise level and Pj is a parameter set for the blur. The new parameter set Pnew is used by the SR engine for one or more AI models to perform SR operations on the current frame.
[0034] Figure 3A FIG. 3 shows a parameter fusion module 310 according to an embodiment. The parameter fusion module 310 may be Figure 2 . In this example, the parameter fusion module 310 includes N multipliers to multiply the weights w1, w2, w3, ..., wN with their corresponding parameter sets. Some weights may be zero, and the sum of the weights is equal to 1. The parameter fusion module 310 also includes an adder 315 to calculate the weighted sum of the parameter set as an output.
[0035] Figure 3B FIG. 3 shows a parameter fusion module 320 according to another embodiment. The parameter fusion module 320 may be Figure 2. In this example, the parameter fusion module 320 includes N multipliers to multiply the weights w1, w2, w3, ..., wN with their corresponding parameter sets. Some weights may be zero, and the sum of the weights is equal to 1. The parameter fusion module 320 also includes a multiplexer 325 for selecting a weighted parameter set as an output. In one embodiment, only one of the weights (e.g., wi) is equal to one, and all other weights are zero. Therefore, the multiplexer 325 selects the parameter set corresponding to the weight wi as an output. The selected parameter set is used by the SR engine 140 to perform AI-based SR operations.
[0036] Figure 3C FIG. 3 shows a parameter fusion module 330 according to another embodiment. The parameter fusion module 330 may be Figure 2 . In this example, the parameter fusion module 330 pairs the weights w1, w2, w3, ..., wN with their corresponding parameter sets. Some weights may be zero, and the sum of the weights is equal to 1. The parameter fusion module 330 may output the (non-zero weight, parameter set) pair to the SR engine 140 for further processing. This embodiment allows the SR engine 140 to perform the combination Figure 5 Describes the weighted AI operation.
[0037] Depending on the content and quality of the input image sequence, Figure 2 The parameter fusion module 220 in the Figure 3A , any fusion operation shown in 3B and 3C. Therefore, in one embodiment, the parameter fusion module 220 may include Figure 3A , a combination of the circuits shown in 3B and 3C to achieve various selection and fusion operations.
[0038] Figure 4 An example of SR operations performed by the SR engine 140 according to one embodiment is shown. In this example, the SR engine 140 includes a CNN accelerator 410 to perform CNN operations on an input frame.
[0039] In one embodiment, the CNN accelerator 410 includes hardware elements that specifically accelerate neural network operations through convolution operations, fully connected operations, activation, pooling, normalization, element-by-element mathematical calculations, etc. In some embodiments, the CNN accelerator 410 includes multiple computing units and memories (e.g., static random access memory (SRAM)), wherein each computing unit also includes multiplier and adder circuits for performing mathematical operations such as multiply-and-accumulate (MAC) operations to accelerate convolution, activation, pooling, normalization, and other neural network operations. The CNN accelerator 410 can perform fixed-point and floating-point neural network operations. The CNN accelerator 410 can perform SR operations to generate high-resolution output frames with enhanced image quality.
[0040] CNN accelerator 410 performs a convolution operation by computing the convolution of an input feature map with a kernel. For example, an input feature map 430 from a previous layer of a CNN operation can be convolved with kernel 420 to generate an output feature map 440 to the next layer. Kernel 420 includes a set of parameters, such as a set of 3x3 parameters. In one embodiment, the parameters in kernel 420 are from Figure 1 The parameter determination module 130 in Figure 2 The new parameters (also called adjusted parameters) generated by the parameter fusion module 220 in Figure 4 As shown in the example of , the parameter fusion module 220 calculates the weighted sum of multiple parameter sets, such as Figure 3A , where each of these parameter sets defines a different kernel. In this example, kernel 420 (θ f )=w1θ1+w2θ2+…+w N θ N The kernel 420 is passed from the parameter fusion module 220 to the SR engine 140 to perform the CNN operation.
[0041] Figure 5 FIG. 1 shows an example of an SR operation performed by the SR engine 140 according to another embodiment. In this example, the SR engine 140 calculates a weighted sum of AI operations (AI_OP1, AI_OP2, etc.) such as CNN operations or other deep learning or machine learning operations. In one embodiment, the SR engine 140 calculates the weighted sum of AI operations (AI_OP1, AI_OP2, etc.) such as CNN operations or other deep learning or machine learning operations. Figure 3C The parameter fusion module 330 in receives the (non-zero weight, parameter set) pair. Each AI_OP is executed using one of the parameter sets. The output of each AI_OP is multiplied by the corresponding weight. The weighted sum can be sent to the next layer of AI processing.
[0042] Figure 6FIG. 6 is a flowchart of a method 600 for performing an SR operation according to an embodiment. For example, the method 600 may be performed by Figure 2 The image processing circuit 200 in Figure 7 The device 700 described in and / or in combination Figure 1 , 3A , 3B and 3C. It is understood that these embodiments are for illustration purposes only. Other devices with image processing capabilities can perform method 600.
[0043] Method 600 begins when the device detects a representative feature in an image sequence, the image sequence including a current frame and past frames within a time window. At step 620, the device adjusts parameters of one or more AI models based on the measurement results of the representative feature. At step 630, the device processes the current frame using the one or more AI models with the adjusted parameters to generate a high-resolution image for display.
[0044] To adjust the parameters, the device may generate weights of two or more pre-trained parameter sets of the AI model based on the measurement results of the representative features, and apply the weights to the two or more pre-trained parameter sets to generate adjusted parameters. In some embodiments, the device may calculate a weighted sum based on the weights and the two or more pre-trained parameter sets to generate the adjusted parameters. In some embodiments, the device may calculate a weighted sum of the kernels of the CNN to obtain a new kernel for processing the current frame.
[0045] Figure 7 An example of a device 700 according to an embodiment is shown. The device 700 may include Figure 2 The image processing circuit 200 performs the aforementioned image enhancement operations including the SR operation. An example of the device 700 is a television, which receives a video with a low resolution (e.g., 720 x 480 pixels) and performs an SR operation to scale the video up to a high resolution (e.g., 3840 x 2160 pixels) for display on a television screen. Optionally, the device 700 may be a smart phone, a computing device, a network connection device, a gaming device, an entertainment device, an Internet of Things (IoT) device, or any device capable of processing and displaying images and / or videos.
[0046] The device 700 includes processing hardware 710. In one embodiment, the processing hardware 710 may include one or more processors, such as a central processing unit (CPU), a GPU, a digital processing unit (DSP), an AI processor, a multimedia processor, and one or more of other general and / or special processing circuits. In one embodiment, the processing hardware 710 may include a hardware accelerator, such as a CNN accelerator. In one embodiment, the processing hardware 710 includes the image guidance module 110, the parameter determination module 130, and the SR engine 140 in the above-mentioned embodiments.
[0047] The device 700 also includes memory and storage hardware 720 coupled to the processing hardware 710. The memory and storage hardware 720 may include memory devices such as dynamic random access memory (DRAM), SRAM, flash memory and other non-random access memory, and other temporary machine-readable storage media, such as volatile or non-volatile storage devices. The memory and storage hardware 720 may further include storage devices, such as any type of solid-state or magnetic storage device. In one embodiment, the memory and storage hardware 720 may store such as in combination with Figure 1 and Figure 2 The described pre-trained parameter set 725. In some embodiments, the memory and storage hardware 720 may store instructions that, when executed by the processing hardware 710, cause the processing hardware 710 to perform the above-described operations for generating a high-resolution image, e.g. Figure 6 Method 600.
[0048] Device 700 may also include a display panel 730 to display information such as images, videos, messages, web pages, games, text, and other types of text, image, and video data. Device 700 may also include audio hardware 740 for receiving and generating sound, such as a microphone and a speaker.
[0049] In some embodiments, the device 700 may also include a network interface 750 to connect to a wired and / or wireless network for sending and / or receiving voice, digital data and / or media signals. It will be appreciated that for purposes of illustration, Figure 7 The embodiments are simplified. Other hardware elements may be included.
[0050] Figure 6 The operation of the flowchart has been referenced Figure 2 and Figure 7 However, it should be understood that Figure 6 The operations of the flowchart can be performed by Figure 2 and Figure 7 Embodiments other than the embodiments of the present invention are performed, and Figure 2 and Figure 7 Embodiments of the invention may perform operations different from those discussed with reference to the flowcharts. Figure 6 The flowcharts show a specific order of operations performed by a specific embodiment of the present invention, but it should be understood that this order is exemplary (for example, alternative embodiments may perform operations in a different order, may combine specific operations, overlap specific operations, etc.).
[0051] Various functional components or blocks have been described herein. As will be appreciated by those of ordinary skill in the art, the functional blocks will preferably be implemented by circuits (special purpose or general purpose circuits operating under the control of one or more processors and codec instructions), which typically include transistors configured to control the operation of the circuits according to the functions and operations described herein. For example, Figure 1 The image guiding module 110, the parameter determining module 130 and the SR engine 140 may be functional blocks implemented by dedicated circuits or general-purpose circuits.
[0052] Although the present invention has been described according to several embodiments, those skilled in the art will recognize that the present invention is not limited to the described embodiments and that the present invention can be implemented with modifications and variations within the spirit and scope of the appended claims. Therefore, the description should be considered illustrative rather than restrictive.
Claims
1. An image processing circuit for performing a super-resolution operation, comprising: A memory for storing multiple parameter sets of multiple artificial intelligence models; An image guidance module for detecting representative features in an image sequence, the image sequence comprising a current frame and past frames within a time window; a parameter determination module for adjusting parameters of one or more artificial intelligence models based on the measurement results of the representative features; as well as A super-resolution engine for processing the current frame using one or more artificial intelligence models with multiple adjusted parameters to generate a high-resolution image for display, Among them, the parameter determination module is also used to: generate multiple weights of pre-trained parameter sets of two or more artificial intelligence models based on the measurement results of the representative features; and apply the multiple weights to two or more pre-trained parameter sets to generate the multiple adjusted parameters.
2. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that The parameter determination module further includes an adder, which can be used to: A weighted sum is calculated from the plurality of weights and the two or more pre-trained parameter sets to generate the plurality of adjusted parameters.
3. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that The parameter determination module further includes an adder, which can be used to: A weighted sum of multiple kernels of a convolutional neural network is calculated to obtain a new kernel of the super-resolution engine to process the current frame.
4. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that The representative feature comprises an image quality measurement q, and the two or more pre-trained parameter sets are pre-trained using a plurality of images having image quality measurements q1 and q2, where q1<q<q2.
5. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that: The representative features include at least image quality measurement results and image content classification.
6. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that The parameter determination module further comprises a multiplexer, configured to: Based on the measurement result of the representative feature, select a parameter set from a plurality of pre-trained parameter sets as a new parameter set; as well as The selected parameter set is output to the super-resolution engine to process the current frame.
7. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that: Also included is a convolutional neural network accelerator, which can be used to perform multiple convolutional neural network calculations when processing the current frame.
8. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that: The representative features include one or more of the following: image quality indicators, scene types, degradation types, degradation levels, color conditions, image resolutions, noise levels, noise types, video compression parameters, codec artifact levels, color saturation, sharpness, and contrast.
9. The image processing circuit for performing super-resolution operation according to claim 1, characterized in that: The super-resolution engine can also be used to: Using one or more artificial intelligence models and the plurality of parameters adjusted based on the scene type and image quality indicators measured from the image sequence, a plurality of artifacts in the previous frame are removed.
10. A method for performing a super-resolution operation, comprising: detecting representative features in an image sequence, the image sequence comprising a current frame and past frames within a time window; adjusting a plurality of parameters of one or more artificial intelligence models based on the measurement results of the representative features; as well as processing the current frame using one or more artificial intelligence models having a plurality of adjusted parameters to generate a high-resolution image for display, Among them, the step of adjusting the multiple parameters also includes: generating multiple weights of pre-trained parameter sets of two or more artificial intelligence models based on the measurement results of the representative features; and applying the multiple weights to two or more pre-trained parameter sets to generate the multiple adjusted parameters.
11. The method for performing super-resolution operation according to claim 10, characterized in that The step of applying the plurality of weights further comprises: A weighted sum is calculated from the plurality of weights and two or more pre-trained parameter sets to generate the plurality of adjusted parameters.
12. The method for performing super-resolution operation according to claim 10, characterized in that Also includes: A weighted sum of multiple kernels of a convolutional neural network is calculated to obtain a new kernel for processing the current frame.
13. The method for performing super-resolution operation according to claim 10, characterized in that The representative feature comprises an image quality measurement q, and the two or more pre-trained parameter sets are pre-trained using a plurality of images having image quality measurements q1 and q2, where q1<q<q2.
14. The method for performing super-resolution operation according to claim 10, characterized in that: The representative features include at least image quality measurement results and image content classification.
15. The method for performing super-resolution operation according to claim 10, characterized in that: The step of adjusting multiple parameters also includes: Selecting a parameter set from a plurality of pre-trained parameter sets based on the measurement result of the representative feature; and The current frame is processed using one or more artificial intelligence models having the selected set of parameters.
16. The method for performing super-resolution operation according to claim 10, characterized in that: The step of processing the current frame also includes: A convolutional neural network calculation is performed using the plurality of adjusted parameters to process the current frame.
17. The method for performing super-resolution operation according to claim 10, characterized in that: The representative features include one or more of the following: image quality indicators, scene types, degradation types, degradation levels, color conditions, image resolutions, noise levels, noise types, video compression parameters, codec artifact levels, color saturation, sharpness, and contrast.
18. The method for performing super-resolution operation according to claim 10, characterized in that: The step of processing the current frame further comprises: Using one or more artificial intelligence models and the plurality of parameters adjusted based on the scene type and image quality indicators measured from the image sequence, a plurality of artifacts in the previous frame are removed.
