Image enhancement using implicit tensor-product b-spline modeling

Abstract

Methods and apparatus for image enhancement using implicit tensor-product B-spline (TPB) modeling. According to an example embodiment, a method for image enhancement includes partitioning an input image into a plurality of first patches; applying TPB modeling to each of the plurality of first patches to generate a respective plurality of implicit TPB models; generating a plurality of second patches using the respective pluralities of implicit TPB models, each of the second patches representing a respective one of the first patches; and combining the plurality of second patches to form an output image representing the input image, the output image having a higher resolution or a better noise metric than the input image.

Description

IMAGE ENHANCEMENT USING IMPLICIT TENSOR-PRODUCT B-SPLINE MODELING 1. Cross-Reference to Related Applications

[0001] This application claims the benefit of priority from U.S. Provisional Patent Application Ser. No. 63 / 520,686, filed on 21 August 2023, and European Application No. 23197314.0 filed on 14 September 2023, each of which is incorporated by reference herein in its entirety. 2. Field of the Disclosure

[0002] Various example embodiments relate to image processing and, more specifically but not exclusively, to operations directed at improvement of a selected image characteristic. 3. Background

[0003] Downsampling and upsampling are two widely used image-processing tasks, with applications in image display, compression, and progressive transmission. Downsampling reduces the spatial resolution while substantially maintaining perceptually important features of the image. Downsampling is used, for example, to reduce the storage and / or transmission requirements of images. Upsampling increases the spatial resolution while substantially maintaining perceptually important features of the image. Upsampling is used, for example, for zooming in on a part of the image or for eliminating the pixelation effect that arises when a low- resolution image is displayed on a relatively large frame. In some cases, downsampling and upsampling are used in combination, e.g., in lossy compression, multiresolution lossless compression, and progressive transmission of images.

[0004] Image denoising is an image-processing task that involves removing at least some noise from an image. Noise can be introduced into images during acquisition or processing and typically reduces the perceptual image quality and / or makes it more difficult to interpret the image. Image denoising techniques aim to restore an image to its original or intended perceptual quality by reducing or removing the noise while preserving perceptually important features of the image. BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS

[0005] Disclosed herein are various embodiments of methods and apparatus for image enhancement using implicit tensor-product B-spline (TPB) modeling. In embodiments directed at generating super-resolved images, implicit TPB models are fitted to overlapping patches derived from an input image. The processed overlapping patches are subsequently fused using an intricate mechanism that incorporates information from multiple sources, effectively amplifying the resolution of a final image. In various examples, the fusion process is configured to ensure that the final image is not only coherent and visually pleasing but also preserves perceptually important features of the input image. In embodiments directed at generating denoised images, two differently configured TPB optimization procedures are applied to non- overlapping patches of the input image. By integrating these two optimization procedures, the corresponding embodiments operate to effectively diminish the noise component while maintaining perceptually important details and textures within the denoised image.

[0006] According to an example embodiment, provided is a method of image enhancement comprising: partitioning an input image into a plurality of first patches; applying TPB modeling to each of the plurality of first patches to generate a respective plurality of implicit TPB models; generating a plurality of second patches using the respective pluralities of implicit TPB models, each of the second patches representing a respective one of the first patches; and combining the plurality of second patches to form an output image representing the input image, the output image having a higher resolution or a better noise metric than the input image.

[0007] According to another example embodiment, provided is a non-transitory computer- readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the above method.

[0008] According to yet another example embodiment, provided is an apparatus for image enhancement, the apparatus comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: partition an input image into a plurality of first patches; apply TPB modeling to each of the plurality of first patches to generate a respective plurality of implicit TPB models; generate a plurality of second patches using the respective pluralities of implicit TPB models, each of the second patches representing a respective one of the first patches; and combine the plurality of second patches to form an output image representing the input image, the output image having a higher resolution or a better noise metric than the input image.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Other aspects, features, and benefits of various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings, in which:

[0010] FIG. 1 is a block diagram illustrating a method of generating an upsampled image according to some embodiments.

[0011] FIG. 2 is a block diagram illustrating an example layout of overlapping patches constructed in the method of FIG. 1 according to some embodiments.

[0012] FIG. 3 is a block diagram illustrating Multilayer TPB Optimization operations of the method of FIG. 1 according to some embodiments.

[0013] FIG. 4 is a flowchart illustrating operations of a first layer of the Multilayer TPB Optimization block illustrated in FIG. 3, according to some embodiments.

[0014] FIG. 5 is a flowchart illustrating operations of an even-numbered layer of the Multilayer TPB Optimization block illustrated in FIG. 3, according to some embodiments.

[0015] FIG. 6 is a flowchart illustrating operations of an odd-numbered layer of the Multilayer TPB Optimization block illustrated in FIG. 3, according to some embodiments.

[0016] FIG. 7 graphically illustrates PSNR dependencies on the number N of the layers in the Multilayer TPB Optimization block illustrated in FIG. 3 according to various examples.

[0017] FIG. 8 is a diagram illustrating a weighting matrix used in the RGB Image Construction block of the method of FIG. 1 according one embodiment.

[0018] FIGS. 9-11 illustrate operations of the RGB Image Construction block of the method of FIG. 1 according one example.

[0019] FIG. 12 is a block diagram illustrating a method of generating a denoised image according to some embodiments.

[0020] FIG. 13 is a block diagram illustrating first Multilayer TPB Optimization operations of the method of FIG. 12 according to some embodiments.

[0021] FIG. 14 is a flowchart illustrating operations of a first layer of the first Multilayer TPB Optimization block illustrated in FIG. 13, according to some embodiments.

[0022] FIG. 15 is a flowchart illustrating operations of a subsequent layer of the first Multilayer TPB Optimization block illustrated in FIG. 13, according to some embodiments.

[0023] FIG. 16 graphically illustrates a PSNR dependence on the number N of the layers in the first Multilayer TPB Optimization block illustrated in FIG. 13 according to various examples.

[0024] FIG. 17 is a block diagram illustrating second Multilayer TPB Optimization operations of the method of FIG. 12 according to some embodiments.

[0025] FIG. 18 is a flowchart illustrating operations of the first layer of the second Multilayer TPB Optimization block illustrated in FIG. 17, according to some embodiments.

[0026] FIG. 19 is a flowchart illustrating operations of a subsequent layer of the second Multilayer TPB Optimization block illustrated in FIG. 17, according to some embodiments.

[0027] FIG. 20 graphically illustrates a PSNR dependence on the number N of the layers in the second Multilayer TPB Optimization block illustrated in FIG. 17 according to various examples.

[0028] FIG. 21 graphically illustrates example benefits of using both first and second Multilayer TPB Optimization blocks in the method of FIG. 12 according to some embodiments.

[0029] FIG. 22 is a block diagram illustrating a computing device used to implement image processing according to various embodiments. DETAILED DESCRIPTION

[0030] This disclosure and aspects thereof can be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. The foregoing is intended solely to give a general idea of various aspects of the present disclosure and does not limit the scope of the disclosure in any way.

[0031] In the following description, numerous details are set forth, such as optical device configurations, timings, operations, and the like, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely exemplary and not intended to limit the scope of this application.

[0032] In recent years, the field of image processing has experienced significant advancements, especially in machine learning. One notable approach gaining traction is the use of coordinate-based neural networks or neural fields. These Multi-Layer Perceptron (MLP) models parameterize physical properties of scenes or objects across space and time, thereby enabling efficient and accurate predictions. Neural fields have been successful in various visual computing problems, such as 3D shape and image synthesis, human body animation, 3D reconstruction, and pose estimation. Neural fields have also shown success in traditional image and video compression and processing tasks. However, in at least some cases, neural fields face computational challenges, such as speed and memory constraints.

[0033] At least some embodiments disclosed herein address the above-indicated challenges by leveraging an efficient implicit function via Tensor-Product B-Spline (TPB). Some of the disclosed embodiments address an image denoising task and an image-upsampling (e.g., super- resolution) task. Based on the provided description, a person of ordinary skill in the pertinent art will readily understand how to make and use other TPB-based embodiments directed at other image enhancement tasks without any undue experimentation. TPB-based solutions do not typically rely on iterative computing methods, such as, for example, backpropagation used in neural-network-based solutions, and need less computational power and smaller memory allocations. In various examples, TPB-based methods apply deterministic solutions via the least squares approach. These solutions aim to reduce the computational load compared to that of functionally similar neural fields while achieving a similar accuracy and efficiency.

[0034] In some examples of image denoising and super-resolution tasks, we fit multiple TPBs to model an image and optimize the function coefficients of a corresponding high- dimensional continuous function. For example, a multi-layer method with optimizing the prediction residual between the original input image and the output from the previous layer, with different layers using different respective algorithms, is introduced to achieve an enhanced performance. In some examples, a guided filter is also utilized to reduce an appearance ofartifacts. At least some TPB-based solutions beneficially provide an attractive alternative to neural-network-based solutions.

[0035] It is worth noting that various embodiments disclosed herein below differ from the Tensor-Product B-Spline with Coordinate (TPBC) described in U.S. Patent Application No. 63 / 485,427, which is incorporated herein by reference in its entirety. For example, in TPBC, both pixel values and coordinates are input to the TPB model, with a goal of providing additional input(s) to help remove ambiguity for the same pixel values in different local regions. In an implicit TPB model, only coordinates are input to the TPB model, allowing the model to explore the relationship between coordinates and pixel values, thereby further reducing prediction error. Implicit Tensor-Product B-Spline (TPB) Concepts

[0036] In the mathematical subfield of numerical analysis, a B-spline or basis spline is a spline function that has minimal support with respect to a given degree, smoothness, and domain partition. Any spline function of a given degree can be expressed as a linear combination of B- splines of that degree. Cardinal B-splines have knots that are equidistant from each other. B- splines can be used, e.g., for curve-fitting and numerical differentiation of experimental data. In computer-aided design and computer graphics, spline functions are constructed as linear combinations of B-splines with a set of control points.

[0037] A one-dimensional B-spline can be extended to multiple dimensions through the use of tensor product B-spline (TPB) constructs. Multivariate basis functions are constructed by multiplying B-splines corresponding to the different dimensions. For example, first, a B-spline subspace is defined for each dimension. Thereafter, a tensor product function in the multidimensional space is constructed as a weighted sum of one-dimensional B-spline constructs, with the weights representing the corresponding tensor elements. A brief description of main TPB concepts pertaining to various embodiments disclosed herein is provided below.

[0038] A B-spline possesses a property for approximating a given curve or a segment thereof using a polynomial with a continuity constraint of a specific order. B-splines can be used in a regression process by performing data fitting on top of a B-spline model. To capture a relatively high dimensionality of multiple input variables in video processing, a TPB can be constructed by multiplying multiple B-spline functions. Additionally, optionally, or alternatively, TPB can be used to provide continuous curves by its nature (with its inherent ability to ensure continuity up to a specific order), thereby avoiding or significantly reducing curve fitting operations afterinitial generation of predictions, transformations and / or mappings. For example, computationally intensive polynomial approximation algorithms to ensure continuity in multiple polynomial pieces as applied to the single-channel luminance predictor can be reduced or avoided with TPB techniques.

[0039] An implicit function takes the coordinate as an input, which should be contrasted with functions that take pixel value(s) as an input. One example of an implicit function is the neural field, which uses a neural network for training. In some examples, the neural field adopts a Multi-Layer Perceptron (MLP) to model (or explore) a continuous function, and the training often takes a relatively long time with a multi-iteration solver (e.g., relying on backpropagation). In contrast, various embodiments disclosed herein adopt TPB as the implicit function. The input to the TPB is the coordinate, as indicated above. With the ground truth data, such embodiments use a non-iterative least squared solution (deterministic way) to solve the target problem. As a result, using implicit TPB to model an image can be computationally light and fast. These characteristics make the implicit TPB-based solutions particularly useful, e.g., in upsampling and denoising tasks, where such solutions aid in enhancing the image.

[0040] It is important to note that, in a representative example, the input to the Implicit TPB function includes or consists of two values, typically representing coordinate points. The output of the Implicit TPB function is a single value. This input / output structure further contributes to the utility the Implicit TPB function in upsampling and denoising tasks, where precise coordinate-based manipulations are used to obtain desired image enhancements.

[0041] For clarity and consistency, we introduce specific notations to symbolize the input and output of the implicit TPB function. For instance, when we input a set of coordinate pairs into the fitted TPB, we will denote the resultant output values accordingly. These notations will not only facilitate the understanding and explanation of the function’s application but will also aid in efficiently presenting the processes involved in pertinent image-enhancement tasks. Furthermore, these notations will support a more-structured approach to the explanation of complex interactions and transformations that take place during these tasks.

[0042] Let us have a collection of 2D coordinates {(x,y)} which we map to another collection of point values^^^. The mapping can be done via a mapping function which we denote as ^^^^. This function can be referred to as the “inferencing / testing” function. We load the TPB model “^^^^” which operates based on the coordinate input^(^, ^)^and outputs the predicted values^^^as follows:^^^ = ^^^^(^(^, ^)^) (1) The model ^^^^uses coefficients ^^^^described in more detail below.

[0043] In an effort to more-clearly demonstrate the procedure of fitting a TPB to a set of input coordinate pairs and corresponding output value pairs, we will employ a specific notation system presented below. Therein, Eq. (2) expresses the “training” function to obtain the model ^^^^: ^^^^^This notation succinctly process: the input coordinate pairs are transformed into output values using the TPB fitting function. The notation system provides a concise and self-consistent “language” for discussing and analyzing the TPB fitting process. It encapsulates the transformation of input coordinates into output values within the TPB framework, facilitating the understanding of the manipulations and computations involved. This notation will be especially useful when used to elaborate on the steps and outcomes of the TPB fitting process in both upsampling and denoising tasks.

[0044] In the next portion of this section, we provide more specifics on the coordinate notation employed in the characterization of both upsampling and denoising tasks. The provided description is intended to shed light on how the used notations represent input and output data and how they facilitate visualization of the transformations that take place during the corresponding portions of image processing tasks.

[0045] In the context of implicit functions, the role of coordinates is to act as input values and trigger operations that lead to the determination of the output of the function. The function operates on these coordinates, manipulating them in a variety of ways to generate a specific result. Understanding this interaction is helpful for comprehending the underlying “mechanics” of the corresponding image processing tasks. The discussion and analysis of the coordinates, how they are used, and how they influence the output of the function will facilitate a deeper understanding of the inner workings of implicit functions. Moreover, a clear grasp of these concepts can enhance our ability to use such functions effectively in different contexts, such as the above mentioned upsampling and denoising tasks. Various embodiments disclosed herein are constructed using a patch-based approach. Herein below, we also introduce notations used to describe the patch-based approach.

[0046] Let us consider the pthpatch in an image, with the patch size being (^^× ^^). For simplicity, although there are three color channels (e.g., RGB) in one image, we use one color channel as an example. From the provided description, a person of ordinary skill in the pertinent art will readily understand how to handle the two other color channels without any undue experimentation. Within the pthpatch, we have a collection of original input pixel coordinates, denoted as ^^= ^(^, ^)^, 0 ≤ ^ ≤ 1, 0 ≤ ^ ≤ 1. These coordinates are normalized to be with the range

[0001] in both x and y dimensions of the original global image coordinate.

[0047] In some examples, to standardize these coordinates, we perform another normalization within each patch since not every patch has pixel coordinates that fully occupy the entire

[0001] range. This normalization involves reshaping the coordinates in each dimension and rescaling them to fit within the range

[0001] . To achieve this result, we find the minimum and maximum values for the x-axis in patch p, denoted as (^^ ^^^^, ^^ !) and rescale them to fit within

[0001] using the following formula: 'We also perform an analogous operation for the y-axis, with minimum and maximum values denotedThe resulting set of normalized coordinates is collected and is represented as ^+^= ^(^", ^")^. This normalization process ensures that our operations are consistent and meaningful, regardless of the original scale of the coordinates.

[0048] In the realm of super-resolution tasks, the generation of an upsampled grid of coordinates is important, as it paves the way to creating the desired higher-resolution outputs. Throughout our discussions hereinbelow, we have adopted an upsampling scale denoted as "F". To facilitate a more clear and concise understanding of this process, we define the following specific notations.

[0049] In the original input coordinate, we have ^^coordinate points along the x-axis in patch , and ^^coordinate^ = ^^:.1': ^^+1'(6)where ^^is the starting coordinate of the patch along the x-axis, and ^^is the starting coordinate of the patch along the y-axis.

[0050] For super-resolution tasks, we aim to upsample this set, creating "F" times more points in both the x and y directions. Consequently, the new upscaled patch size will be ((2 ∗ ^^) × (2 ∗ ^^)). Here, F represents the upscaling factor, ^^is the width, and ^^is the height of the patch. By creating this denser collection of coordinates, we can generate a higher- resolution patch through the interpolation and combination of the input pixel data.

[0051] We use the following notation to represent this upsampled set of coordinates:For the upsampled grid, we have:To normalize these upsampled coordinates, we transform them into a new set: ^+4^,^=process of creating these normalized upsampled coordinates is outlined below.

[0052] In a next step, we perform a normalization process on the reshaped, upsampled coordinates to ensure they fall within the

[0001] range in each dimension. This operation involves rescaling the x and y coordinates separately. For the x-axis, we first identify the minimum and maximum values within the upsampled coordinates, represented as (^4^,^ 4^,^^^^, ^^ !). We then rescale these coordinates to fit within the

[0001] range using the following formula: ^"! #;',' 4^=;' !$%&!$;'(,')#!;','(10) $%&Similarly, for the y-axis, we perform a similar operation using the minimum and maximum values denoted as (^4^,^^^^, ^4^,!^)*$()#*$%&The resulting set of normalized, upsampled coordinates is represented as ^+4^,^= 56^"4^, ^"4^78. This notation will help us describe the upsampling process in a more structured and comprehensible manner. By establishing a standardized representation for the input and output data, we can streamline the explanation of the transformations that occur during this process. This will aid in understanding the intricacies of super-resolution tasks and the role of upsamplingin generating more detailed, high-resolution images. Next, we describe the Tensor-Product B- Spline (TPB) optimization process in more detail using the above concepts.

[0053] For a comprehensive understanding of this process, let us consider a scenario where we aim to fit a TPB on a set of normalized ^+^= ^(^", ^")^. The output values corresponding to these inputs will be represented asThis scenario will serve as ourguiding example on the mechanics of TPB context of image processing.

[0054] Let us consider the case in which we want to fit a TPB on ^+^= ^(^", ^")^, and the output value corresponding to the input will be We can apply a similar process for anupsampled grid ^+4^,^following a similar method.

[0055] Consider a case in which we aim to fit a TPB on the normalized coordinate set ^+^= ^(^", ^")^. The associated output value, in relation to these input coordinates, is represented as =^^^_^^^. The process of fitting the TPB can be expressed as:Moving forward, we use two independent basis functions in each dimension for TPB prediction. Assume that we have >?, and >@basis functions along the horizontal axis (x) and vertical axis (y) dimension. Denote, A?, and A@as the knot point index in x and y dimension. By being given a set of knot points in these two dimensions, we have two individual basis function sets: B^^^,?^C(^") and B^^D^^,@(^"), using the normalized coordinate set ^+^= ^(^", E ^ )}.

[0056] The TPB basis function for predicting the output can be constructed by multiplying the two basis functions as:The prediction can be performed^CJK ^DJK∙^C,^Dare the TPB coefficients for the output image which are found through an optimization process.

[0057] We can vectorize the 2D index (e.g., A?and A@) to become a 1D index (e.g., AI) to simplify the expression. For the basis function, we have:B^^^^L(^", ^") = B^^^(^", ^") (15) Let > = > ∙ > , the+ F^^^^" = ^^^^^" =IM#. H^^^∙ B^^^^" With the selectedwe can construct the matrix form for the design matrix: ^^^ ^" ^" ^^^ ^" ^" ⋯ ^^^ ^"FThe prediction coefficients can be expressed as a vector: ^^^The prediction can then be expressed as: =^^^_^^^= F^^^^^^^(19) With this information in hand, we can now move on to optimizing the coefficients of the Implicit TPB, an important step in refining the prediction model.

[0058] Denote the ground truth vector using the coordinate ^+^= ^(^", ^")^ for the output image =^^^_^^^as follows:,N−1Note that the is collected from the coordinate ^+^= ^(^", ^")^, not the spatial reshaped coordinate. An optimal solution of ^^^^= =^^^_^^^To facilitate further discussion, we denote the following matrix and vector: \^^^= ((F^^^)^F^^^(22) ]^^^= (F^^^)^=^^^_^^^) (23) Then^^^^,([^^)=(\^^^)#.]^^^(24)After this detailed explanation of a TPB optimization process, let us now shift our focus to the application of these principles according to various embodiments. Image Super-Resolution

[0059] FIG. 1 is a block diagram illustrating a method (100) of generating an upsampled image according to some embodiments. For illustration purposes and without any implied limitations, the method (100) is described in reference to RGB images. In the example shown, the method (100) operates on an input RGB image (102) to generate an upsampled output RGB image (132). From the provided description, a person of ordinary skill in the pertinent art will be able to make and use embodiments of the method (100) operating on images employing other color schemes without any undue experimentation.

[0060] The method (100) uses a patch-based solution, according to which each patch is modeled using its own corresponding implicit TPB functions. More specifically, the input RGB image (102) is subjected to patch extraction operations (110) to be partitioned into overlapping image patches (112) with patch size ^^× ^^. Each of the image patches (112) is then subjected to two blocks of operations: a Multilayer TPB Optimization block (120) and an RGB Image Construction block (130). An output of the Multilayer TPB Optimization block (120) includes upsampled image patches (122). An output of the RGB Image Construction block (130) includes the upsampled output RGB image (132).

[0061] The Multilayer TPB Optimization block (120) employs multiple implicit Tensor Product Basis (TPB) functions to model an upsampled (e.g.,) super-resolved image patch. The modeling framework is organized into layers, with each layer focusing on a specific aspect of the image reconstruction process. In some examples, the first layer utilizes an implicit TPB function to model each patch through a TPB optimization procedure. A primary objective of this optimization is to minimize the difference between the original input image patch and the corresponding reconstructed patch generated via the implicit TPB-based method. The output of this layer is the reconstructed image patch. Subsequent layers are configured to model the residual between the image patch utilized for super-resolution and the output of the previous layer. Starting from the second layer, the block (120) is configured to alternately fit a TPB function to the residuals of different resolution (upsampled) image patches and the output from the preceding layer. A more detailed description of operations of the block (120) is provided below, e.g., in reference to FIGS. 3-6.

[0062] The RGB Image Construction block (130) operates to construct the output image (132) by seamlessly fusing the overlapping upsampled image patches (122). To ensure a smooth transition across patch boundaries, operations of the block (130) include applying a distinct weighting factor for each pixel, including within the overlapped regions. For a pixel covered by multiple overlapping patches (122), the final fused pixel value is determined by a weighted combination of pixel values from all corresponding overlapping patches (122). A normalization factor, derived from the sum of the weight factors, is employed to maintain balance and consistency in the output image (132). By using this approach, the method (100) effectively creates a cohesive upsampled (e.g., super-resolved) image (132) with smooth transitions between patch boundaries, in which visible artifacts are suppressed (e.g., minimized) and the overall visual quality is enhanced.

[0063] In some examples, the process of image Super-Resolution takes place within the RGB domain but can also be implemented in another color space. One objective is to generate an output image (132) with a resolution F times greater than that of the input image (102), where the input image’s dimensions are represented as ^ × ^. To start this process, the block (110) is configured to extract the overlapping patches (112) from the RGB image (102), with each of the overlapping patches (112) having the size of ^^× ^^.

[0064] FIG. 2 is a block diagram illustrating an example layout (200) of the overlapping patches (112) constructed in the block (112) according to some embodiments. In the example shown, the overlapping patches (112) have a size of 32x32 pixels. In other examples, other patch sizes can also be used. Only three of the overlapping patches (112) are shown in FIG. 2 for illustration purposes. The three patches (112) are labeled using the reference numerals (1121, 1122, 1123). Adjacent ones of the patches (1121, 1122, 1123) share respective overlapping regions that measure either 16x32 pixels or 32x16 pixels, wherein the first number indicates the overlap size along the X axis, and the second number indicates the overlap size along the Y axis. For example, the patches (1121, 1122) overlap within a mutual region of 16x32 pixels. The patch overlaps are used to preserve continuity and consistency in the output image (132) while minimizing potential artifacts, e.g., as illustrated in more detail below.

[0065] The use of overlapping patches (112) is beneficial for the patch fusion process of the block (130). For example, corner pixels of the output image (132) are determined by the output of a single patch (112). In contrast, edge pixels are shaped by the outputs of two neighboring patches (112). Some of the central pixels within the output image (132) are influenced by theoutputs of four intersecting patches (112). Operations of the block (120) are then individually applied to the extracted patches (112) as described in more detail below.

[0066] FIG. 3 is a block diagram illustrating operations of the Multilayer TPB Optimization block (120) according to some embodiments. As already indicated above, an input to the block (120) includes the overlapping patches (112). An output of the block (120) includes the overlapping upsampled image patches (122). Operations of the block (120) are organized using a multi-layer architecture, illustratively including N layers (3101-310N). In some examples, each layer (310n) of the block (120) has similar characteristics and incorporates operations configured to implement single-layer TPB optimization. In some examples, the layers (3101-310N) include three different types of layers (310n), e.g., as described below. By employing multiple types of layers and TPBs, the block (120) aims to achieve improved performance, accuracy, scalability, and / or reliability.

[0067] The first layer (3101) represents a first type of layers. Operations of the first layer (3101) are configured to approximate the input image (102). In some examples, operations of the first layer (3101) include: o Taking an original image patch (112) as an input to the first layer (3101); o Generating a TPB model to fit the original image patch (112); o Applying a denser sample grid via the TPB model to generate an upsampled image patch as a first output of the layer (3101); o Down-sampling the upsampled image patch to generate a TPB-approximated patch having the original resolution; o Computing a residual patch by taking a pixel-by-pixel difference between the original image patch and TPB-approximated patch and outputting the residual patch as a second output of the first layer (3101).

[0068] The second layer (3102) represents a second type of layers. In some examples, the second type of layers includes some or all of the even-numbered layers (310n). Operations implemented in the second type of the layers (310n) are configured to enhance the performance of the upsampling method, e.g., by increasing the Peak Signal-to-Noise Ratio (PSNR) through a reduction of error. In some examples, operations of a layer (310n) belonging to the second type of layers include: o Taking the residual patch outputted by the preceding layer as an input; o Generating a TPB model to fit the input residual patch;o Applying a denser sample grid via the TPB model to generate a predicted upsampled residual patch; o Computing a sum of the preceding layer TPB-model upsampled patch and the present layer TPB predicted upsampled patch and outputting the summed patch as a first output of the present layer; o Downsampling the summed patch to generate a downsampled summed patch having the original resolution; o Computing a residual patch by taking a difference between the original image patch and the downsampled summed patch and outputting this residual patch as a second output of the present layer.

[0069] The third layer (3103) represents a third type of layers. In some examples, the third type of layers includes some or all of the odd-numbered layers (310n), excluding the first layer (3101). Operations implemented in the third type of the layers (310n) are configured to reduce or eliminate “wave artifacts” and to further improve the PSNR. A wave artifact is an artifact that is known to arise in some TPB models and is recognized by persons of ordinary skill in the pertinent art. In some examples, operations of a layer (310n) belonging to the third type of layers include: o Taking the upsampled residual patch outputted by the preceding layer as an input; o Generating a TPB model to fit the received upsampled residual patch; o Applying a denser sample grid via the TPB model to generate a predicted upsampled residual patch; o Computing a sum of the preceding layer TPB-model upsampled patch and the present layer TPB predicted upsampled residual patch and outputting the summed patch as a first output of the present layer; o Downsampling the summed patch to generate a downsampled summed patch having the original resolution; o Computing a residual patch by taking a difference between the original image patch and the downsampled summed patch and outputting this residual patch as a second output of the present layer.

[0070] FIG. 4 is a flowchart illustrating operations of the first layer (3101) of the Multilayer TPB Optimization block (120) according to some embodiments. In the example shown, operations of the first layer (3101) have four blocks of operations (410, 420, 430, 440). The first block (410) is configured to perform Implicit TPB Optimization on image patches (112).Operations of the block (410) include fitting a respective TPB model (412) on the image patch (112) =^^^4^,^. For illustration purposes, the input patch (112) shown in FIG. 4 is the pthpatch extracted from the input image (102). The corresponding processing can be in accordance with Eq. (25): ^^^^,^,.= ^^^^_^^^^^^^6=^^^4^,^, ^+^7 (25) Eq. (25) is analogous to Eq. (12). As such, the TPB methods described above as being applicable to solving Eq. (12) similarly apply to Eq. (25). The inputs to the first block (410) are the pthpatch (112) and the coordinates ^+^(402).

[0071] The TPB model ^^^^,^,.(412) is used in the second block (420) to generate an upsampled image patch =[4^,^^^,^,.(422) with dimension (2 ∗ ^^) × (2 ∗ ^^). In the second block (420), the TPB model ^^^^,^,.(412) is used with an upsampled (414) asfollows:The upsampled image patch =[4^,^^^,^,.(422) is one of the above-mentioned two outputs of the first layer (3101). To simplify the notation throughout this section, we will use the following notation for the upsampled image patch (422): =[4^,^,.= =[4^,^^^,^,..

[0072] Operations of the next blocks (430, 440) are directed at calculating a residual patch =^_`,^,.(442). More specifically, operations of the block (430) include downsampling the upsampled image patch =[4^,^,.(422) to the dimensions (^^× ^^). Operations of the block (440) include subtracting the resulting downsampled image patch from the pthpatch =^^^4^,^(112). Combined processing of the blocks (430, 440) can be expressed as follows:The dimensions of =^_`,^,.(442) are (^^× ^^). In some examples, the down-sample operator used in the block (430) is implemented based on a bicubic resampling method.

[0073] FIG. 5 is a flowchart illustrating operations of an even-numbered layer (310q) of the Multilayer TPB Optimization block (120) according to some embodiments. In the example shown, operations of the even-numbered layer (310q) have five blocks of operations (510, 520, 530, 540, 550). Operations of the first block (510) include fitting a respective TPB model (512) on an image <^_`,^,i#.(504) received from the (q−1)-th layer (310q-1). An additional input to the first block (510) includes the coordinates ^+^(502). The corresponding processing can be in accordance with Eq. (28):^^^^,^,i= ^^^^_^^^^^^^6=^_`,^,i#., ^+^7 (28) Note that Eq. (28) iscan be implemented in a similar manner.

[0074] Operations of the block (520) include applying an upsampled grid ^+4^,^(518) to the TPB model ^^^^,^,i(512) to generate an output (522) including the values of =[4^,^^^,^,iwith dimensions (2 ∗ ^^) × (2 ∗ ^^) as follows: =[4^,^^^,^,i= ^^^^,^,i(^+^,4^) (29)

[0075] Operations of the block (550) include computing a first output (552) the qthlayer (310q). The computations of the block (550) use the output (522) of the preceding block (520) and an output =^_^,^,i#.(548) of the preceding the (q−1)-th layer (310q-1). The corresponding mathematical expression is as follows: =[4^,^,i= =[4^,^^^,^,i+ =[4^,^,i#.(29)

[0076] Operations of the next blocks (530, 540) are directed at calculating a residual patch =^_`,^,i(542). More specifically, operations of the block (530) include downsampling the upsampled image patch =[4^,^,i(552) to the dimensions (^^× ^^). Operations of the block (540) include subtracting the resulting downsampled image patch from the pthpatch =^^^4^,^(112). Combined processing of the blocks (530, 440) can be expressed as follows: =^_`,^,i= =^^^4^,^− abcdefH,gh(=[4^,^,i) (30)

[0077] FIG. 6 is a flowchart illustrating operations of an odd-numbered layer (310q-1) of the Multilayer TPB Optimization block (120) according to some embodiments, where q=4, 6, 8, … . The odd-numbered layer (310q-1) has some commonalities with the above-described even- numbered layer (310q) while also possessing distinct features configured to serve specific functions and positively contribute to the system's overall performance. By comparing the structures and functions of the layers (310q-1, 310q), one can gain an appreciation of the intricate design of the layered architecture and better understand how these layers interact and collaborate to achieve desirable results.

[0078] In the example shown, operations of the odd-numbered layer (310q-1) have six blocks of operations (600, 610, 620, 630, 640, 650). Operations of the first block (610) include fitting a respective TPB model (612) using implicit TPB optimization applied to the corresponding image patch. A first input to the block (610) includes a residual patch (604) from the previous layer=^_`,^,i#jhaving the size of (^^× ^^), which is upsampled by the factor F in each dimension in the pre-processing block (600). A second input to the block (610) includes the upsampled coordinates (602). The optimization operations of the block are qualitatively similar to the optimization of the block (510) described above. The corresponding mathematical expressions are as follows: =^_^,^,i#.= k,efH,gh (=^_`,^,i#j) (31)

[0079] Operations of the block (620) include applying an upsampled grid (618) to theTPB model (612) to generate an output (622) including the values of =[4^,^^^,^,i#.with dimensions (2 ∗ ^^) × (2 ∗ ^^). Operations of the block (620) are qualitatively similar to the operations of the block (520) described above.

[0080] Operations of the block (650) include computing a first output (52) the (q−1) layer (310q-1). The computations of the block (650) use the output (622) of the preceding block (620) and an output =^_^,^,i#j(648) of the preceding the (q−2)-th layer (310q-2). The corresponding mathematical expression is as follows: =[4^,^,i#.= ℎ × =[4^,^^^,^,i#.+ =[4^,^,i#j(33) Herein, the parameter h is a constant that holds a value between zero and one, with its example setting being 0.7. One purpose of using the parameter h is to weight an impact of odd-numbered layers (310q-1) on the entire super-resolution process. The value of the parameter h is an adjustable parameter of the corresponding algorithm that can be used to implement a smoother transition between adjacent layers (310), which typically leads to a more consistent quality of the final high-resolution image.

[0081] Operations of the next blocks (630, 640) are directed at calculating a residual patch =^_`,^,i#.(642). More specifically, operations of the block (630) include downsampling the upsampled image patch (652) to the dimensions (^^× ^^). Operations of the block (640) include subtracting the resulting downsampled image patch from the pthpatch =^^^4^,^(112). Combined=^_`,^,i#.= =^^^4^,^− abcdefH,gh(=[4^,^,i#.) (34)

[0082] The number N of the layers (310) in the Multilayer TPB Optimization block (120) is an algorithm parameter that can be different for different embodiments and / or setups. In someexamples, the number N can be determined based on a stopping criterion applied to iterative processing implemented using a sequence of the above-described even- and odd-numbered layers (310). In some implementations, the stopping criterion can be related to the stability of the output PSNR values over several consecutive layers (310). For example, the stopping criterion may be satisfied when the average PSNR value of the previous ten layers substantially matches the current layer’s PSNR value. In other words, the iterative process has reached a saturation point beyond which further iterations do not meaningfully improve the PSNR.

[0083] In some examples, to calculate PSNR values, we compare the downsampled (by the factor F) output of each layer (310) with the corresponding input patch (112) in accordance with the following expression:Herein, the term “average PSNR of the last ten layers” refers to the mean value of the Peak Signal-to-Noise Ratio (PSNR) obtained from the final ten layers of the above described TPB model (120). This value is calculated by summing up individual PSNR values from those ten layers and then dividing the sum by ten:In some examples, the stopping criterion is expressed as follows: ^^mno,^,i== ^^mno, @_^ ^_,^,i(37)

[0084] In various, the output for each individual patch is determined by the final layer (i.e., the Nthlayer) of the Multilayer TPB Optimization block (120). This output, which represents the enhanced, high-resolution version of the initial patch, is hereafter denoted using the following notation: =[4^,_^|,^= =[4^,^,n(38)

[0085] FIG. 7 graphically illustrates PSNR dependencies on the number N of the layers (310) in the Multilayer TPB Optimization block (120) according to various examples. More specifically, three PSNR curves (702, 704, 706) are shown. The first curve (702) graphically shows example PSNR results achieved with the block (120), wherein each layer (310) after the first layer (3101) has a structure of an “odd-numbered” layer illustrated in FIG. 6. The second curve (704) graphically shows example PSNR results achieved with the block (120), wherein each layer (310) after the first layer (3101) has a structure of an “even-numbered” layer illustrated in FIG. 5. The third curve (706) graphically shows example PSNR results achievedwith the block (120), wherein, after the first layer (3101), “even-numbered” an “odd-numbered” layers are alternated.

[0086] Through a series of comprehensive experiments, we discovered that the simultaneous utilization of both odd- and even-numbered layers results in relative quality improvements indicated by the PSNR differences between the third curve (706) and the other two curves (702, 704). In some examples, variations in the frequency of employing odd- and even-numbered layers in the setup of the block (120) did not significantly affect either the qualitative or the quantitative outcomes. This result underscores the resilience of the disclosed architecture of the block (120) to such frequency alterations, and further emphasizes an importance of integrating two types of layers for improved performance.

[0087] In various examples, the ratio of odd- and even-numbered layers may deviate from 1:1. However, the structure of the block (120) in which “odd-numbered” and “even-numbered” layers are alternated typically provides better results indicated by the third curve (706). Additionally, this structure lends itself to streamlined programming and also ensures that the benefits of both types of layers are consistently leveraged, contributing to the overall efficacy of the disclosed super-resolution methodology.

[0088] The remaining part of this section describes example embodiments of the RGB Image Construction block (130). More specifically, we describe example mechanisms of integrating overlapping patches generated as described above to form the upsampled output RGB image (132). For illustration purposes and without any implied limitations, the provided examples pertain to patches with dimensions of 32x32 pixels, wherein neighboring patches have overlapping regions measuring either 16x32 pixels or 32x16 pixels as illustrated in FIG. 2. Based on the provided description, a person of ordinary skill in the pertinent art will be able to make and use other embodiments, with other patch dimensions, without any undue experimentation.

[0089] Not that the corner pixels of the output image (132) are derived from the output of a single patch, whereas the edge pixels are shaped by the outputs from two adjoining patches. Some of the pixels located within an interior portion of the output image (132) are influenced by the outputs from four intersecting patches.

[0090] In some examples, to ensure a smooth transition across patch boundaries, it is typically not sufficient to simply overlay those patches. Therefore, various embodiments employvariable weighting factors for different pixels. For example, for a pixel encompassed by multiple overlapping patches, a final fused pixel value for the output image (132) is derived from a weighted combination of the pixel values from each of the corresponding overlapping patches. This pixel value is further normalized by a factor derived from the sum of the weight factors.

[0091] In some examples, achieving desirable image characteristics involves assigning weights based on the patch shape, which is different from a conventional isotropic circular shape weights profile. For rectangular patches described above, we aim to achieve equal weighting within the rectangular contour, ensuring a more accurate representation of the patch shape.

[0092] In one embodiment, we use a ^^× ^^weighting matrix described by Eqs. (39)-(41) for red, green, and blue channels. For illustration purposes and without any implied limitations, a square patch shape ^^= ^^= a}~is considered, where c presents the color channel: R, G, or B. Also, we use the same set of normalized, upsampled coordinates, denoted=upsampled coordinates, representedThese notations aid in streamlining the description of various embodiment and are also helpful to the programming involved. Eqs. (39)-(41) are as follows: ^}~= 0.2 ∙ a}~(39)_where ^~and c~represent diminishing factors according to the distance from the selected location (x, y) to the center (0.5, 0.5). More specifically, ^~serves as an auxiliary variable, which is contingent on the distance from the patch’s center (0.5, 0.5) and correlates with the time constant in a declining exponential function. c~signifies an actual weight value in this context. For illustration purposes, the matrices ^~and c~are fixed and are used for each patch. In some examples, for the boundary patches, which do not have a full patch size, we still apply the same equation by aligning the top-left corner of the patch with the top-left corner of the matrices. Note that, in the boundary patch case, the center of the patch may not have the highest value of the weights, as the peak value corresponds to the center of the full-size patch.

[0093] FIG. 8 is a diagram (800) illustrating a a}~× a}~weighting matrix c~used in the RGB Image Construction block (130) according one embodiment. As indicated above, the illustrated weighting matrix c~equal weights along its rectangular contour. The highest weightvalue is at the geometric center of the rectangle. The weight values decrease with the increasing distance form the center.

[0094] In some examples, the corresponding computing device is configured to use two memory spaces, each to accommodate the same size image patch, (^ × ^). The spaces are used for the quantities denoted as =^and = / , where =^is used to accumulate the weighted patch value for each pixel, and = / is used to accumulate the weights for each pixel. The red, green, and blue channels of =^are represented as =~,^(^ = s, ^, B), and the red, green, and blue channel of =~are represented as =~, / . The pixel values inside those spaces are initialized at 0.

[0095] For each red, green, and blue patch, =~^,}, the computing device operates to apply the upsampled image (122) to get the mapped value =^~^,}. The dimension of =~^,}is (2 ∗ ^^) × (2 ∗ ^^). For the sake of clarity and consistency, we will use the following notation: =^~^,}= <[^4^,_^|,^(42) In one example, the mapped values are multiplied by the weighting matrix, and then the weighted mapped values are accumulated in the corresponding pixel location in the new memory space =^^. The weighting factors processed using the same approach. The following equations provide the corresponding mathematical expressions:where (^4^, ^4^) is the corresponding pixel location in the upsampled image coordinate from the (^"4^, ^"4^) of the pthpatch coordinate.

[0096] After scanning P patches within one frame, the computing device operates to divide the accumulated weighted =~,^=~, / as follows:Note that the values of =~, / are fixed for every frame. In some examples, these values can be computed at the system boot up and stored in the memory.

[0097] FIGS. 9-11 illustrate operations of the block (130) according one example. More specifically, FIG. 9 is a diagram illustrating an accumulated weighting matrix (900) =~, / 6^4^, ^4^7, wherein a 12x18 array of the above-described weighting matrices (800) is used.FIG. 10 is a diagram illustrating a weighted reference image (1000) =~,^6^4^, ^4^7. The image (1000) is formed by applying appropriate weights to various reference patches, e.g., in accordance with Eq. (43). FIG. 11 depicts an output image (132) produced using an element- wise division between the weighted reference image (1000) and the weighting matrix F~in accordance with Eq. (45). This division normalizes thepatch (122), leading to the output image (132), wherein the patches (122) are combined according to their respective weights. Collectively, the images shown in FIGS. 9- 11 pictorially illustrate the process of constructing the F~output image (132) through aggregation of weighted patches. Such aggregation ensures that most relevant and informative parts of each input patch are preserved and combined in a proper way to form the upsampled output image (132).

[0098] The above-described approach involves the use of the overlapping patches (112), which helps to achieve a relatively high quality for the output image (132). In some examples, non-overlapping patches can similarly be used, e.g., when some higher level of image imperfections and / or artifacts can be tolerated for the intended usage of the output image (132). For example, in some cases, the use of non-overlapping patch approach may result in the presence of blocking artifacts. These artifacts are usually undesirable as they may lead to a disjointed and / or fragmented appearance of the output image (132). In contrast, the use of the overlapping patches (112) leads to substantially imperceivable image-continuity disruptions, leading to a smoother, more cohesive visual result, free of blocking artifacts. Thus, the use of the overlapping patches (112) may be preferred for at least some applications. Image Denoising

[0099] FIG. 12 is a block diagram illustrating a method (1200) of generating a denoised image according to some embodiments. For illustration purposes and without any implied limitations, the method (1200) is described in reference to RGB images. In the example shown, the method (1200) operates on an input RGB image (1202) to generate a denoised output RGB image (1262). From the provided description, a person of ordinary skill in the pertinent art will be able to make and use embodiments of the method (1200) operating on images employing other color schemes without any undue experimentation.

[0100] In the example shown, operations of a subset (1220, 1230, 1240, 1250) of blocks of the method (1200) are carried in the YCbCr domain. Accordingly, the method (1200) alsoincludes an RGB to YCbCr conversion block (1210) and a YCbCr to RGB conversion block (1260). The RGB to YCbCr conversion block (1210) operates on the input RGB image (1202) to generate a YCbCr image (1212). The YCbCr to RGB conversion block (1260) operates on a denoised YCbCr image (1252) to generate the denoised output RGB image (1262).

[0101] The method (1200) uses a patch-based solution, according to which each patch is modeled using its own corresponding implicit TPB functions. More specifically, the YCbCr image (1212) is subjected to patch extraction operations (1220) to be partitioned into non- overlapping image patches (1222) with patch size ^^× ^^. Each of the image patches (1222) is then subjected to three blocks of operations: a first Multilayer TPB Optimization block (1230), a second Multilayer TPB Optimization block (1240), and a YCbCr Image Construction block (1250). An output of the first Multilayer TPB Optimization block (1230) includes denoised image patches (1232). An output of the second Multilayer TPB Optimization block (1240) includes denoised image patches (1242). An output of the YCbCr Image Construction block (1250) includes the denoised YCbCr image (1252).

[0102] The first Multilayer TPB Optimization block (1230) employs multiple implicit Tensor Product Basis (TPB) functions to create the denoised image patches (1232). In some examples, operations of the block (1230) are organized into layers, with each layer concentrating on a particular aspect of the image reconstruction process. For example, in a first layer, the block (1230) uses an implicit TPB function to model each patch through a TPB optimization procedure. One goal of this optimization is to minimize a discrepancy between the original input image patch and the reconstructed patch produced by the implicit TPB-based technique. As a result, an output of this layer includes a reconstructed image patch. Subsequent layers are focused on modeling the respective residuals between the image patch utilized for denoising and the output of the previous layer. Starting from the second layer, the block (1230) uses a TPB function adapted to handle the residuals of image patches and the output from the preceding layer. This iterative process helps to refine the reconstructed image, ensuring enhanced image quality and a good level of noise reduction.

[0103] The second Multilayer TPB Optimization block (1240) utilizes multiple implicit Tensor Product Basis (TPB) functions to generate the denoised image patches (1242). In some examples, operations of the block (1240) are organized into layers, with each addressing a specific aspect of the image reconstruction process. For example, in the first layer, the block (1240) operates to apply an implicit TPB function to model individual patches through a TPBoptimization procedure. One goal of this optimization is to minimize a discrepancy between the original input image patch and the reconstructed patch created by the implicit TPB-based technique. The output of this layer includes a reconstructed image patch. Subsequent layers concentrate on modeling the residual between the image patch employed for denoising and the output from the previous layer. From the second layer onward, the block (1240) uses a TPB function adapted to handle the residuals of image patches and the output of the preceding layer. This iterative process helps to refine the reconstructed image. In some examples, to further improve visual quality and overall performance, the block (1240) is configured to apply a Guided Filter to the output of the TPB model following the first layer. This step aids in reducing visual artifacts, resulting in a more refined output for each layer.

[0104] The YCbCr Image Construction block (1250) is configured to assemble the non- overlapping denoised image patches (1242) together into the denoised image (1252). Examples of specific methodologies employed to configure the operations of each of the blocks (1230, 1240, 1250) are described below.

[0105] One objective of incorporating two distinct TPB optimization methods embodied in the blocks (1230, 1240) is to enhance the outcomes of the overall denoising process. Our experimental results indicate that by properly configuring the blocks (1230, 1240), a relative improvement in the performance of the method (1200) by approximately 2-3 dB can be achieved.

[0106] In some examples, using the block (1230) alone may unintentionally decrease the Peak Signal-to-Noise Ratio (PSNR). Similarly, using the block (1240) alone and using an appropriate channel as the guidance map may lead to suboptimal results akin to those observed with the application of the block (1230) alone. This observation underscores the benefits of a balanced approach that effectively harnesses the strengths of both optimization methods while attenuating their respective shortcomings.

[0107] FIG. 13 is a block diagram illustrating operations of the first Multilayer TPB Optimization block (1230) according to some embodiments. As already indicated above, an input to the block (1230) includes the patches (1222). An output of the block (1230) includes the denoised image patches (1232). Operations of the block (120) are organized using a multi-layer architecture, illustratively including N layers (13101-1310N). In some examples, the layers (13101-1310N) include two different types of layers (1310n), e.g., as described below. Byemploying multiple types of layers and TPBs, the block (1230) aims to achieve improved performance, accuracy, scalability, and / or reliability.

[0108] FIG. 14 is a flowchart illustrating operations of the first layer (13101) of the first Multilayer TPB Optimization block (1230) according to some embodiments. In the example shown, operations of the first layer (13101) have three blocks of operations (1410, 1420, 1440). The first block (1410) is configured to perform Implicit TPB Optimization on image patches (1222). Operations of the block (1410) include fitting a respective TPB model (1412) on the image patch (1222) =^^^4^,^. For illustration purposes, the input patch (1222) shown in FIG. 14 is the pthpatch extracted from the input image (1212). The corresponding processing can be in accordance with Eq. (46):Eq. (46) is analogous to Eq. (12). As such, the TPB methods described above as being applicable to solving Eq. (12) similarly apply to Eq. (46). The inputs to the first block (1410) are the pthpatch (1222) and the coordinates ^+^(1402).

[0109] The TPB model ^^^^,^,.(1412) is used in the second block (1420) to generate a predicted image patch =[4^,^^^,^,.(1422). In the second block (1420), the TPB model ^^^^,^,.(1412) is used with the grid (^+^) (1414) as follows:The image patch =[4^,^^^,^,.(1422) is one of the above-mentioned two outputs of the first layer (13101). To simplify the notation throughout this section, we will use the following notation for the image patch (1422): =[4^,^,.= =[4^,^^^,.. Operations of the next block (1440) are directed at calculating a residual patch =^_`,^,.(1442). More specifically, operations of the block (1440) include subtracting the image patch (1422) from the pthpatch =^^^4^,^(1222) as follows: =^_`,^,.= =^^^4^,^− =[4^,^,.(48)

[0110] FIG. 15 is a flowchart illustrating operations of a subsequent layer (1310q) of the first Multilayer TPB Optimization block (1230) according to some embodiments, where q>1. In the example shown, operations of the layer (1310q) have four blocks of operations (1510, 1520, 1540, 1550). Operations of the first block (1510) include fitting a respective TPB model (1512) on an image <^_`,^,i#.(1504) received from the (q−1)-th layer (1310q-1). An additional input to the first block (1510) includes the coordinates ^+^(1502). The corresponding processing can be in accordance with Eq. (49):^^^^,^,i= ^^^^_^^^^^^^6=^_`,^,i#., ^+^7 (49) Note that Eq. (49) iscan be implemented in a similar manner.

[0111] Operations of the block (1520) include applying the grid ^+^(1518) to the TPB model ^^^^,^,i(1512) to generate an output (1522) including the values of =[4^,^^^,^,ias follows:

[0112] Operations of the block (1550) include computing a first output (1552) the qthlayer (1310q). The computations of the block (1550) use the output (1522) of the preceding block (1520) and an output =^_^,^,i#.(1548) of the preceding the (q−1)-th layer (1310q-1). The corresponding mathematical expression is as follows: =[4^,^,i= =[4^,^^^,^,i+ =[4^,^,i#.(51)

[0113] Operations of the next block (1540) are directed at calculating a residual patch =^_`,^,i(1542). More specifically, operations of the block (1540) include subtracting the image patch (1552) from the pthpatch =^^^4^,^(1222) as follows: =^_`,^,i= =^^^4^,^− =[4^,^,i(52)

[0114] FIG. 16 graphically illustrates a PSNR dependence on the number N of the layers (1310) in the first Multilayer TPB Optimization block (1230) according to various examples. In some examples, the PSNR dependence of FIG. 16 is used to determine the value of N to be used in the first Multilayer TPB Optimization block (1230). The value of N determines the number of iterations performed in the block (1230) before termination. Based on the PSNR data of FIG. 16 an optimal value of N is in the range between 120 and 210. In some examples, the value of N=200 is selected because this value strikes a balance between the computational resource utilization and output quality. In many cases, N=200 provides a practical and nearly optimal choice.

[0115] FIG. 17 is a block diagram illustrating operations of the second Multilayer TPB Optimization block (1240) according to some embodiments. As already indicated above, an input to the block (1240) includes the patches (1232). An output of the block (1240) includes the denoised image patches (1242). Operations of the block (120) are organized using a multi-layer architecture, illustratively including N layers (17101-1710N). In some examples, the layers (17101-1710N) include two different types of layers (1710n), e.g., as described below. Byemploying multiple types of layers and TPBs, the block (1240) aims to achieve improved performance, accuracy, scalability, and / or reliability.

[0116] Each of the layers (17101-1710N) receives a guidance map (1702) as an input. In some examples, the guidance map (1702) for the Guided Filter is derived from the output of the first Multilayer TPB Optimization block (1230). More specifically, the guidance map (1702) can be generated from the luma channel (Y), e.g., as explained in more detail below. In that regard, some embodiments of the second Multilayer TPB Optimization block (1240) may benefit from certain features disclosed in: He, Kaiming, Jian Sun, and Xiaoou Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, v. 35, no. 6 (2012), pp. 1397- 1409, which is incorporated herein by reference in its entirety.

[0117] Let us denote the guidance maps (1702) as =^9_^ ^. A guided image filter is an efficient edge-preserving image filter. Given an input image =^^_^^, guidance image =^9_^ ^, kernel size ^, and a desired degree of smoothness ^, the guided image filter performs local weighted averaging on the input image similar to that of a bilateral filter. In some examples, the weights are calculated such that the contribution from the pixels on the same side of the edge (e.g., similar color) in the guidance image is greater than the contribution from the pixels on the opposite side (e.g., different color). In this manner, the pixels on the same side of the edge in the guidance image are averaged together, resulting in the edge-preserving property.

[0118] In operation, a guided filter imposes the filtered output result as a linear function related to the guidance image with added noise. With this configuration, a constrained optimization in each local region is formulated to minimize a prediction error; and ridge regression in each local region is used to solve the corresponding optimization problem. In some examples, efficient solution for pixel-wise ridge regression is used to speed up the filtering process.

[0119] Let us use the following notation: =^^_^^is the input image to which we apply the guided filter; =^9_^ ^refers to the guidance map that steers the filter; =[4^_^^symbolizes the output image produced after applying the guided filter. In mathematical terms, the guided filter operation can be expressed as: =[4^_^^= ^2^^^^_^(=^^_^^, =^9_^ ^) (53)Eq. (53) states that the guided filter GF is applied to the input image =^^_^^with the guidance of =^9_^ ^, resulting in the output image =[4^_^^. Details of implementing an example guided filter GF are disclosed, e.g., in the above-cited He article.

[0120] The second Multilayer TPB Optimization block (1240) is implemented as a sequence of iterative layers (1710) as illustrated in FIG. 17. This pipeline incorporates two distinct types of layers (1710), each specifically designed for a respective objective, contributing individually towards the overall processing mechanism.

[0121] An objective of the first layer (17101) is to approximate the original image effectively. Example operations performed in the first layer (17101) include: o Taking an original image patch (1222) as an input; o Generating a TPB model to fit original image patch (1222); o Applying the same grid via the TPB model to generate a denoised image patch; o Applying a guided filter to the denoised image patch, using the guidance map (1702); o Computing a residual between the original image patch (1222) and the TPB- approximated image patch.

[0122] An aim of each subsequent layer (1710q), where q>1, is to bolster the efficacy of the denoising technique, e.g., by elevating the PSNR. In some examples, this aim is achieved through systematic error minimization, leading to significant improvements in the image quality. Example operations performed in the layer (1710q) include: o Taking the previous residual patch as an input; o Generating a TPB model to fit the previous residual patch; o Applying the same grid via the TPB model to generate a predicted denoised residual patch; o Applying a guided filter to the denoised residual image patch, using the guidance map (1702). o Adding the results from both the previous-layer TPB output and the current TPB- predicted output; o Computing a residual between the original image patch (1222) and the TPB- approximated image patch.

[0123] FIG. 18 is a flowchart illustrating operations of the first layer (17101) of the second Multilayer TPB Optimization block (1240) according to some embodiments. In the example shown, operations of the first layer (17101) have four blocks of operations (1810, 1820, 1830, 1840). The first block (1810) is configured to perform Implicit TPB Optimization on image patches (1222). Operations of the block (1810) include fitting a respective TPB model (1812) on the image patch (1222) =^^^4^,^. For illustration purposes, the input patch (1222) shown in FIG. 18 is the pthpatch extracted from the input image (1212). The corresponding processing can be in accordance with Eq. (54):The inputs to the first block (1810) are the pthpatch (1222) and the coordinates ^+^(1802).

[0124] The TPB model ^^^^,^,.(1812) is used in the second block (1820) to generate a predicted image patch =[4^,^^^,^,.(1822). In the second block (1820), the TPB model ^^^^,^,.(1812) is used with the grid (^+^) (1814) as follows: =The image patch =[4^,^^^,^,.(1822) and the Guidance map (1702) are used in the third block (1830) to compute a first of two outputs of the first layer (17101), e.g., as follows: =[4^,^,.= ^2^^^^_^(=[4^,^^^^,^,., =^9_^ ^) (56)

[0125] Operations of the next block (1840) are directed at calculating a residual patch =^_`,^,.(1842). More specifically, operations of the block (1840) include subtracting the image patch (1832) from the pthpatch =^^^4^,^(1222) as follows: =^_`,^,.= =^^^4^,^− =[4^,^,.(57)

[0126] FIG. 19 is a flowchart illustrating operations of a subsequent layer (1710q) of the second Multilayer TPB Optimization block (1240) according to some embodiments, where q>1. In the example shown, operations of the layer (1710q) have four blocks of operations (1910, 1920, 1930, 1940). Operations of the first block (1910) include fitting a respective TPB model (1912) on an image <^_`,^,i#.(1904) received from the (q−1)-th layer (1710q-1). An additional input to the first block processing can be in accordance^^^^,^,i= ^^^^_^^^^^^^6=^_`,^,i#., ^+^7 (58)

[0127] The TPB model ^^^^,^,i(1912) is used in the second block (1920) to generate a predicted image patch =[4^,^^^,^,i(1922). In the second block (1920), the TPB model ^^^^,^,i(1912) is used with the grid (^+^) (1914) as follows: = = The image patch =[4^,^^^,^,i, map , -th layer TPB output (1928) =[4^,^,i#.are used in the third block (1930) to compute a first of two outputs of the first layer (1710q), e.g., as follows: =

[0128] Operations of the next block (1940) are directed at calculating a residual patch =^_`,^,i(1942). More specifically, operations of the block (1940) include subtracting the image patch (1932) from the pthpatch =^^^4^,^(1222) as follows: =^_`,^,i= =^^^4^,^− =[4^,^,i(61)

[0129] FIG. 20 graphically illustrates a PSNR dependence on the number N of the layers (1710) in the second Multilayer TPB Optimization block (1240) according to various examples. In some examples, the PSNR dependence of FIG. 20 is used to determine the value of N to be used in the second Multilayer TPB Optimization block (1240). The value of N determines the number of iterations performed in the block (1240) before termination. Based on the PSNR data of FIG. 20 an optimal value of N is in the range between 120 and 200. In some examples, the value of N=200 is selected because this value strikes a balance between the computational resource utilization and output quality. In many cases, N=200 provides a practical and nearly optimal choice.

[0130] FIG. 21 graphically illustrates example benefits of using both the first and second Multilayer TPB Optimization blocks (1230, 1240) in the method (1200) according to some embodiments. More specifically, FIG. 21 shows three PSNR dependencies (2102, 2104, 2106) on the number N of the layers (1710) for three different designs of the method (1200). More specifically, the PSNR dependence (2102) illustrates the PSNR performance of an embodiment of the method (1200) in which the block (1230) is absent and the block (1240) is present. The PSNR dependence (2104) illustrates the PSNR performance of an embodiment of the method (1200) in which the block (1230) is present and the block (1240) is absent. The PSNR dependence (2106) illustrates the PSNR performance of an embodiment of the method (1200) in which both of the blocks (1230, 1240) are present. An enhanced performance exhibited by thelatter embodiment is clearly apparent from comparison of the PSNR dependencies (2102, 2104, 2106) shown in FIG. 21. Example Hardware

[0131] FIG. 22 is a block diagram illustrating a computing device (2200) used to implement image processing according to various embodiments. In some examples, the device (2200) can be used to implement the method (100) or the method (1200). The computing device (2200) comprises input / output (I / O) devices (2210), a processing engine (2220), and a memory (2230). The I / O devices (2210) may be used to enable the device (2200) to receive various input signals (2202) and to output various output signals (2204). For example, the I / O devices (2210) may be operatively connected to send and receive signals via a communication channel.

[0132] The memory (2230) may have buffers to receive data. Once the data are received, the memory (2230) may provide parts of the data to the processing engine (2220) for processing therein. The processing engine (2220) includes a processor (2222) and a memory (2224). The memory (2224) may store therein program code, which when executed by the processor (2222) enables the processing engine (2220) to perform various data processing operations, including but not limited to at least some operations of the above-described methods (110) and / or (1200).

[0133] According to an example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS. 1-22, provided an apparatus for image enhancement comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: partition an input image into a plurality of first patches; apply TPB modeling to each of the plurality of first patches to generate a respective plurality of implicit TPB models; generate a plurality of second patches using the respective pluralities of implicit TPB models, each of the second patches representing a respective one of the first patches; and combine the plurality of second patches to form an output image representing the input image, the output image having a higher resolution or a better noise metric than the input image.

[0134] According to another example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS. 1-22, provided is a method of image enhancement comprising the steps of: partitioning an input image into a plurality of first patches; applying TPB modeling to each of the plurality of first patches togenerate a respective plurality of implicit TPB models; generating a plurality of second patches using the respective pluralities of implicit TPB models, each of the second patches representing a respective one of the first patches; and combining the plurality of second patches to form an output image representing the input image, the output image having a higher resolution or a better noise metric than the input image.

[0135] In some embodiments of the above method, the first patches are non-overlapping patches of the input image; and wherein the second patches are non-overlapping patches of the output image.

[0136] In some embodiments of any of the above methods, the combining comprises joining corresponding edges of neighboring ones of the second patches to form the output image.

[0137] In some embodiments of any of the above methods, the first patches are overlapping patches of the input image; and wherein the second patches are overlapping patches of the output image.

[0138] In some embodiments of any of the above methods, the combining comprises: generating a weighting matrix for the plurality of second patches, the weighting matrix having smaller weights corresponding to a peripheral patch portion compared to weights corresponding to a middle patch portion; for each pixel of the output image, computing a respective sum of weighted pixel values of corresponding overlapping pixels of the second patches using the weighting matrix; and computing a pixel value for the output image by dividing said respective sum by a sum of weights used to compute said respective sum.

[0139] In some embodiments of any of the above methods, the input image is an RGB image or a YCbCr image; and wherein the output image is an RGB image or a YCbCr image.

[0140] In some embodiments of any of the above methods, the applying comprises subjecting a selected first patch to processing in a first sequence of serially connected layers, each of the serially connected layers being configured to at least perform TPB optimization to generate a respective one of the respective plurality of implicit TPB models.

[0141] In some embodiments of any of the above methods, the first sequence comprises an input layer and a plurality of second layers.

[0142] In some embodiments of any of the above methods, the first sequence further comprises a plurality of third layers; and wherein the second layers and the third layers are interleaved in the first sequence.

[0143] In some embodiments of any of the above methods, the input layer is configured to: generate an initial implicit TPB model by fitting the selected first patch; apply a denser sample grid via the initial implicit TPB model to generate an initial upsampled patch; down-sample the initial upsampled patch to generate an initial TPB-approximated patch having an original resolution; and compute an initial residual patch by taking a pixel-by-pixel difference between the selected first patch and the initial TPB-approximated patch.

[0144] In some embodiments of any of the above methods, a second layer is configured to: receive a residual patch and an upsampled patch computed by a preceding layer; generate an implicit TPB model by fitting the received residual patch; apply a denser sample grid via the generated implicit TPB model to generate a predicted upsampled residual patch; compute a sum of the received upsampled patch and a predicted upsampled patch computed via the implicit TPB model and outputting a resulting summed patch as an output of the second layer; down-sample the summed patch to generate a downsampled summed patch having the original resolution; and compute an output residual patch by taking a difference between the selected first patch and the downsampled summed patch.

[0145] In some embodiments of any of the above methods, a third layer is configured to: upsample a residual patch computed by a preceding layer; generate an implicit TPB model by fitting the upsampled residual patch; apply the denser sample grid via the implicit TPB model to generate a predicted upsampled patch; compute a sum of the upsampled patch and the predicted upsampled patch and outputting a resulting summed patch as an output of the third layer; down- sample the summed patch to generate a downsampled summed patch having the original resolution; and compute an output residual patch by taking a difference between the selected first patch and the downsampled summed patch.

[0146] In some embodiments of any of the above methods, the applying comprises: subjecting a selected first patch to processing in a first sequence of serially connected layers, each of the serially connected layers of the first sequence being configured to at least perform TPB optimization to generate a respective one of the respective plurality of implicit TPB models; and subjecting an output patch generated via the first sequence to processing in a second sequence of serially connected layers, each of the serially connected layers of the secondsequence being configured to use a guidance map and to at least perform TPB optimization to generate another respective one of the respective plurality of implicit TPB models.

[0147] In some embodiments of any of the above methods, the second sequence comprises an input layer and a plurality of second layers.

[0148] In some embodiments of any of the above methods, the input layer is configured to: generate an implicit TPB model by fitting the selected first patch; apply an original grid via the implicit TPB model to generate a denoised patch; apply a guided filter to the denoised image patch using the guidance map; and compute a residual patch by computing a difference between the selected first patch and the denoised patch.

[0149] In some embodiments of any of the above methods, a second layer is configured to: receive a residual patch and a TPB-predicted patch computed by a preceding layer; generate an implicit TPB model by fitting the received residual patch; apply an original grid via the implicit TPB model to generate a predicted denoised residual patch; apply a guided filter to the predicted denoised residual image patch using the guidance map; compute a summed patch by adding the received TPB-predicted patch and a current TPB-predicted patch; and compute a residual patch by computing a difference between the selected first patch and the summed patch.

[0150] In some embodiments of any of the above methods, the first sequence comprises an input layer and a plurality of second layers.

[0151] In some embodiments of any of the above methods, the input layer is configured to: generate an initial implicit TPB model by fitting the selected first patch; apply an original sample grid via the initial implicit TPB model to generate an initial denoised patch; and compute an initial residual patch by taking a pixel-by-pixel difference between the selected first patch and the initial denoised patch.

[0152] In some embodiments of any of the above methods, a second layer is configured to: receive a residual patch and a denoised patch computed by a preceding layer; generate an implicit TPB model by fitting the received residual patch; apply an original sample grid via the generated implicit TPB model to generate a predicted residual patch; compute a sum of the received denoised patch and a predicted patch computed via the implicit TPB model and outputting a resulting summed patch as an output of the second layer; and compute an output residual patch by taking a difference between the selected first patch and the summed patch.

[0153] According to yet another example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGS. 1-22, provided is non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the above methods.

[0154] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.

[0155] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0156] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0157] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

[0158] While this disclosure includes references to illustrative embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the disclosure, which are apparent to persons skilled in the art to which the disclosure pertains are deemed to lie within the principle and scope of the disclosure, e.g., as expressed in the following claims.

[0159] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit.

[0160] Some embodiments can be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented invention(s). Some embodiments can also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium including being loaded into and / or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as a computer or a processor, the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0161] Unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about” or “approximately” preceded the value or range.

[0162] The use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate theinterpretation of the claims. Such use is not to be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0163] Although the elements in the following method claims, if any, are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence.

[0164] Reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments necessarily mutually exclusive of other embodiments. The same applies to the term “implementation.”

[0165] Unless otherwise specified herein, the use of the ordinal adjectives “first,” “second,” “third,” etc., to refer to an object of a plurality of like objects merely indicates that different instances of such like objects are being referred to, and is not intended to imply that the like objects so referred-to have to be in a corresponding order or sequence, either temporally, spatially, in ranking, or in any other manner.

[0166] Unless otherwise specified herein, in addition to its plain meaning, the conjunction “if” may also or alternatively be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” which construal may depend on the corresponding specific context. For example, the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event].”

[0167] Also, for purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energy is allowed to be transferred between two or more elements, and the interposition of one or more additional elements is contemplated, although not required. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements.

[0168] As used herein in reference to an element and a standard, the term compatible means that the element communicates with other elements in a manner wholly or partially specified by the standard and would be recognized by other elements as sufficiently capable of communicating with the other elements in the manner specified by the standard. The compatible element does not need to operate internally in a manner specified by the standard.

[0169] The functions of the various elements shown in the figures, including any functional blocks labeled as “processors” and / or “controllers,” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

[0170] As used in this application, the terms “circuit,” “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / orfirmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0171] It should be appreciated by those of ordinary skill in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0172] “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification is intended to introduce some example embodiments, with additional embodiments being described in “DETAILED DESCRIPTION” and / or in reference to one or more drawings. “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

CLAIMS What is claimed is:

1. A method of image enhancement, comprising: partitioning an input image into a plurality of first patches; applying tensor-product B-spline (TPB) modeling to each of the plurality of first patches to generate a respective plurality of implicit TPB models, wherein only coordinates of each of the first patches are input to the implicit TPB models; generating a plurality of second patches using the respective pluralities of implicit TPB models, each of the second patches representing a respective one of the first patches; and combining the plurality of second patches to form an output image representing the input image, the output image having a higher resolution or a better noise metric than the input image.

2. The method of claim 1, wherein the first patches are non-overlapping patches of the input image; and wherein the second patches are non-overlapping patches of the output image.

3. The method of claim 1, wherein the combining comprises joining corresponding edges of neighboring ones of the second patches to form the output image.

4. The method of claim 3, wherein the first patches are overlapping patches of the input image; and wherein the second patches are overlapping patches of the output image.

5. The method of any preceding claim, wherein the combining comprises: generating a weighting matrix for the plurality of second patches, the weighting matrix having smaller weights corresponding to a peripheral patch portion compared to weights corresponding to a middle patch portion; for each pixel of the output image, computing a respective sum of weighted pixel values of corresponding overlapping pixels of the second patches using the weighting matrix; and computing a pixel value for the output image by dividing said respective sum by a sum of weights used to compute said respective sum.

6. The method of any preceding claim, wherein the input image is an RGB image or a YCbCr image; and wherein the output image is an RGB image or a YCbCr image.

7. The method of any preceding claim, wherein the applying comprises subjecting a selected first patch to processing in a first sequence of serially connected layers, each of the serially connected layers being configured to at least perform TPB optimization to generate a respective one of the respective plurality of implicit TPB models.

8. The method of claim 7, wherein the first sequence comprises an input layer and a plurality of second layers.

9. The method of claim 8, wherein the first sequence further comprises a plurality of third layers; and wherein the second layers and the third layers are interleaved in the first sequence.

10. The method of claim 8 or 9, wherein the input layer is configured to: generate an initial implicit TPB model by fitting the selected first patch; apply a denser sample grid via the initial implicit TPB model to generate an initial upsampled patch; down-sample the initial upsampled patch to generate an initial TPB-approximated patch having an original resolution; and compute an initial residual patch by taking a pixel-by-pixel difference between the selected first patch and the initial TPB-approximated patch.

11. The method of claim 9 or 10, wherein a second layer is configured to: receive a residual patch and an upsampled patch computed by a preceding layer; generate an implicit TPB model by fitting the received residual patch; apply a denser sample grid via the generated implicit TPB model to generate a predicted upsampled residual patch; compute a sum of the received upsampled patch and the predicted upsampled residual patch computed via the implicit TPB model and outputting a resulting summed patch as an output of the second layer;down-sample the summed patch to generate a downsampled summed patch having the original resolution; and compute an output residual patch by taking a difference between the selected first patch and the downsampled summed patch.

12. The method of any one of claims 9 to 11, wherein a third layer is configured to: upsample a residual patch computed by a preceding layer; generate an implicit TPB model by fitting the upsampled residual patch; apply the denser sample grid via the implicit TPB model to generate a predicted upsampled patch; compute a sum of the upsampled patch which is the output of the preceding layer and the predicted upsampled patch and outputting a resulting summed patch as an output of the third layer; down-sample the summed patch to generate a downsampled summed patch having the original resolution; and compute an output residual patch by taking a difference between the selected first patch and the downsampled summed patch.

13. The method of any preceding claim, wherein the applying comprises: subjecting a selected first patch to processing in a first sequence of serially connected layers, each of the serially connected layers of the first sequence being configured to at least perform TPB optimization to generate a respective one of the respective plurality of implicit TPB models; and subjecting an output patch generated via the first sequence to processing in a second sequence of serially connected layers, each of the serially connected layers of the second sequence being configured to use a guidance map and to at least perform TPB optimization to generate another respective one of the respective plurality of implicit TPB models.

14. The method of claim 13, wherein the second sequence comprises an input layer and a plurality of second layers.

15. The method of claim 14, wherein the input layer is configured to: generate an implicit TPB model by fitting the selected first patch; apply an original grid via the implicit TPB model to generate a denoised patch;apply a guided filter to the denoised image patch using the guidance map; and compute a residual patch by computing a difference between the selected first patch and the denoised patch.

16. The method of claim 14 or 15, wherein a second layer is configured to: receive a residual patch and a TPB-predicted patch computed by a preceding layer; generate an implicit TPB model by fitting the received residual patch; apply an original grid via the implicit TPB model to generate a predicted denoised residual patch; apply a guided filter to the predicted denoised residual image patch using the guidance map; compute a summed patch by adding the received TPB-predicted patch and a current TPB- predicted patch; and compute a residual patch by computing a difference between the selected first patch and the summed patch.

17. The method of any one of claims 13 to 16, wherein the first sequence comprises an input layer and a plurality of second layers.

18. The method of claim 17, wherein the input layer is configured to: generate an initial implicit TPB model by fitting the selected first patch; apply an original sample grid via the initial implicit TPB model to generate an initial denoised patch; and compute an initial residual patch by taking a pixel-by-pixel difference between the selected first patch and the initial denoised patch.

19. The method of claim 17 or 18, wherein a second layer is configured to: receive a residual patch and a denoised patch computed by a preceding layer; generate an implicit TPB model by fitting the received residual patch; apply an original sample grid via the generated implicit TPB model to generate a predicted residual patch; compute a sum of the received denoised patch and a predicted patch computed via the implicit TPB model and outputting a resulting summed patch as an output of the second layer; andcompute an output residual patch by taking a difference between the selected first patch and the summed patch.

20. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of any one of claims 1-19.

21. An apparatus for image enhancement, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: partition an input image into a plurality of first patches; apply tensor-product B-spline (TPB) modeling to each of the plurality of first patches to generate a respective plurality of implicit TPB models; generate a plurality of second patches using the respective pluralities of implicit TPB models, each of the second patches representing a respective one of the first patches; and combine the plurality of second patches to form an output image representing the input image, the output image having a higher resolution or a better noise metric than the input image.

Images