Image processing apparatus and method

Abstract

An image processing device for transforming an image, the device comprising a processor for implementing a trained algorithm to process a received first image and generate a parameter set from the received first image, and to generate a second image by applying a local parametric filter to the first image, the local parametric filter using the parameter set as input. Thus, the device can predict a set of filter parameters required to apply the local parametric filter to the first image to generate a second image with improved image quality.

Description

Technical Field

[0001] The present invention relates to image processing, and in particular to image enhancement or transformation using a parameter filter. Background Art

[0002] Human digital artists and photographers can improve the aesthetic quality of digital photographs through manual image enhancement and retouching. The goal is to create a processed image that is visually appealing and a perceptual improvement over the original input. Manual enhancement methods comprise a variety of post-processing editing options, such as image sharpening tools and color adjustments. Professional-level software tools such as Photoshop and Lightroom allow for the application of these modifications through interactive and semi-automated methods. In addition to basic global tools such as contrast enhancement and brightening, advanced editing capabilities are also available through local image adjustments and adaptive image adjustments.

[0003] Figures 1(a) and 1(b) illustrate two different types of local filters available, for example, in Adobe Lightroom and Photoshop. Figure 1(a) illustrates a gradient filter that applies a linear adjustment to a selected area using three parallel lines 102, 104, and 106. Figure 1(b) illustrates a radial filter that applies a local adjustment in a radial manner indicated by an ellipse 108.

[0004] Manual enhancement remains challenging for non-expert users who may lack the appropriate skills and knowledge to improve images perfectly and efficiently. The final image quality is highly dependent on the end-user's skills and subjective aesthetic judgment. Even with sufficient skills, extensive manual editing is often required to achieve a satisfactory enhancement. Semi-automated tools may only require adjusting a few hyperparameters, thereby speeding up the process to some extent, but the results can be highly sensitive to parameter values, again requiring user experience and expertise. Furthermore, semi-automated methods are often based on hard-coded heuristic rules that encapsulate human perceptual rules of thumb, such as enhancing details or stretching image contrast. This in turn leads to brittleness and low-quality final results.

[0005] Existing automated image enhancement methods primarily employ data-driven, learning-based approaches. Such models typically perform fully supervised learning from input / output image pairs, defined as before and after images edited by professional digital artists or photographers. Early work in this area typically extracted handcrafted features such as intensity distribution from the input image and then learned appropriate mappings from low-level image features to enhancement tool parameter settings accordingly. This strategy has previously been applied to single global editing tools, such as global tonal and color adjustments. Recent learning-based work has instead used deep convolutional neural networks (CNNs) to automatically learn image features. The learned features encode, layer by layer, a series of low-level to high-level image features that attempt to capture semantic information.

[0006] In this area, CNNs enable context-sensitive photo editing. Deep CNNs have brought significant improvements to photo retouching and have been used for spatially varying color mapping based on semantic information and handcrafted global and local features, as described in Yan et al., "Automatic Photo Adjustment with Deep Neural Networks," ACM Transactions on Graphics (TOG), 35.2 (2016): 11; and for color vision constancy, where semantic understanding helps resolve estimation ambiguity, as described in Hu et al., "Exposure: A Framework for White-Box Photo Post-Processing," ACM Transactions on Graphics (SIGGRAPH), 2017. CNNs have also been trained to predict local affine transformations in bilateral space, which can be used as an approximation for edge-aware image filters and color / tone adjustments, as described in Gharbi et al., "HDRNet: Deep Bilateral Learning for Real-Time Image Enhancement," ACM Transactions on Graphics (SIGGRAPH), 2017. Gharbi et al.'s work learns a local affine color transformation for each RGB channel and computes a scale map on a low-resolution version of the image. However, this approach does not perform any global adjustments. A recent photo post-processing framework proposed by Hu and colleagues predicts a global retouching curve in RGB space: [Hu et al., 2018, "Exposure: A white-box photo post-processing framework," ACM Transactions on Graphics (TOG), 37.2 (2018): 26.] However, this approach is limited to monotonic curves, which in turn limits flexibility, as local adjustments based on image spatial regions cannot be performed. The deep photo enhancer presented in the following paper can learn the enhancement function using a generative adversarial network (GAN) setting: Chen et al., "Deep Photo Enhancer: Unpaired Learning of Image Enhancement from Photos with GANs," CVPR, 2018. However, this setting can lack interpretability and must also account for the well-known min / max game training problem associated with GAN models. Work by Park et al., as described in the following paper, instead employs deep reinforcement learning to enhance image adjustments: Park et al., "Warp and Restore: Color Enhancement with Deep Reinforcement Learning," CVPR, 2018. However, they can only apply global image modifications (such as contrast or saturation), which limits the expressiveness of the model. In contrast, in the following recent "Deep Brightness Estimation" paper, an encoder-decoder setting is used to learn a proportional brightness map: "Underexposed Photo Enhancement with Deep Illumination Estimation", CVPR, 2019. However, global adjustments cannot be performed. The learned mapping can be considered highly complex and depends critically on the regularization strategy adopted.Recently, image enhancement has been learned in a multi-task setting, as described in this paper: Kong et al., "Multi-task Bilateral Learning for Real-Time Image Enhancement," Journal of the Society for Information Display, 2019. In summary, there has been a lot of interest in recent years in learning-based methods for image enhancement using neural network architectures.

[0007] There is a need to develop automated photo enhancement tools that can replace the manual work of non-professional users or provide professional artists with an improved starting point for manual editing. Summary of the Invention

[0008] According to one aspect, an image processing device for transforming an image is provided. The device includes a processor configured to implement a trained algorithm to process a received first image and generate a parameter set based on the received first image, and to generate a second image by applying a local parametric filter to the first image, the local parametric filter using the parameter set as input. The processing of the first image automatically predicts and generates a set of parametric filters using the trained algorithm for enhancing the image, thereby improving image quality.

[0009] A parametric filter is a filter that can be defined using a finite number of parameters. Local parametric filters act within the local region of the image where the filter is applied. The fewer the number of parameters required to define the filter, the more efficient it is to predict the values ​​of those parameters.

[0010] The local parametric filter can be used to apply different types of transformations to different regions of the first image, and the parameter set includes one or more parameters that determine the region in the first image to which the filter is applied for a particular type of transformation. Thus, the image processing can support combining multiple filters in multiple regions of the image to achieve optimal resulting image quality.

[0011] The particular kind of transformation of the local parameter filter can depend on the parameters input to the filter, and the parameter set can include one or more parameters for determining the particular kind of filter transformation. Using local parameter filters means that these simple filters can be learned using a smaller set of parameters compared to standard models. Using fewer parameters can also mean less memory used during implementation and less computation required.

[0012] The filter may be one of the following filters: a gradient filter, an elliptic filter, and a polynomial filter. These filters are commonly used simple parametric filters that are easy to learn and can provide improved output images.

[0013] The filter can adjust one or more of hue, saturation, chroma, brightness, and color channel balance. Adjusting these characteristics of the image can improve image quality.

[0014] The first image may include data in a set of multiple color channels, and the filter may adjust less than all color channels. This can increase the speed of image transformation because only necessary color channels are affected to improve image quality.

[0015] The trained algorithm may be a machine learning algorithm. Recent advances in machine learning, namely deep learning, can more accurately predict the types of parameter filters that may be needed to generate higher quality images.

[0016] The trained algorithm can process the first image by identifying features in the first image, and the device can be used to generate the parameter set based on the identified features. By learning the features, the method of applying the filter by a human operator can be mimicked, thereby saving manpower required to apply the same filter in the same manner.

[0017] The device may include a memory that stores code executable by the processor to implement the trained algorithm and the filter in a non-transitory manner. Storing the required elements locally can improve the time and resource efficiency of the process, thereby improving image quality.

[0018] The apparatus is configured to generate the second image by applying a plurality of local parametric filters to the first image, each of the local parametric filters using as input a subset of the parameter set. By applying a plurality of local parametric filters that each use a subset of the parameter set, efficient application of the filters can be achieved, thereby improving image quality.

[0019] At least two of the plurality of local parameter filters may have filter functions different from each other.Using multiple types of local parameter filters can improve image quality compared to using only one type of filter.

[0020] At least two of the plurality of local parametric filters may have the same filter function but be applied at maximum intensity at different locations in the first image. Applying the same local parametric filter function at multiple locations in the same image allows for efficient application of filters, thereby improving image quality. This can be achieved by concentrating different filters of the same type where they are needed, rather than expanding a single filter to include all areas requiring that type of filter, but also including areas of the image where it is not needed.

[0021] The first image may include data in a set of a plurality of color channels, one of the plurality of local parametric filters may adjust a first subset of the color channels, and a second of the plurality of local parametric filters may adjust a second subset of the color channels different from the first subset. By applying the plurality of local parametric filters, each for adjusting a subset of the color channels, efficient application of filters may be achieved, thereby improving image quality.

[0022] According to another aspect, a method for transforming an image is provided, the method comprising: receiving a first image; processing the first image with a trained algorithm to generate a set of parameters from the first image, and generating a second image by applying a local parametric filter to the first image, the local parametric filter using the set of parameters as input. The processing of the first image automatically predicts and generates a set of parametric filters using the trained algorithm for enhancing the image.

[0023] Prior to the processing step, the method may include training the algorithm through a machine learning process. Recent advances in machine learning, namely deep learning, can more accurately predict the type of parametric filters that may be needed to generate higher quality images. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0025] Figure 1(a) and Figure 1(b) show two different types of available parametric filters;

[0026] Figure 2(a) schematically shows how to use five variables to parameterize the elliptic filter;

[0027] Figure 2(b) shows how to apply an elliptical filter to an image;

[0028] Figure 3(a) shows the standard equation of the ellipse of rotation represented by five learnable parameters;

[0029] Figure 3(b) schematically shows how to use five variables to parameterize the elliptic filter;

[0030] Figure 3(c) shows how to apply an elliptical filter to an image;

[0031] Figure 4(a) shows a schematic diagram of a gradient filter and how it is parameterized using three variables;

[0032] Figure 4(b) shows an example of applying a combination of two gradient filters to the blue channel (B channel) on the left, and the adjustment applied to the flower image on the right;

[0033] Figure 5(a) shows a cubic function that depends on the pixel position and pixel value in the image. The equation defines twenty parameters from A to T.

[0034] Figure 5(b) shows the effect of the cubic filter of Figure 5(a) on the right as a heat map based on the image on the left;

[0035] Figure 6 A schematic flow chart illustrating a regression neural network that can be used to calculate parameters of a parametric filter;

[0036] Figure 7 A schematic diagram of a filter-type specific fusion block is shown;

[0037] Figure 8 A schematic diagram showing an example of fusing filter outputs using a first parallel approach;

[0038] Figure 9 Schematic diagram showing examples of fusing filter outputs using different second parallel methods;

[0039] Figure 10 A schematic diagram showing an example of fusing filter outputs using a sequential approach;

[0040] Figure 11 A schematic diagram of an example camera for implementing an image processor to process images captured by an image sensor in the camera is shown. DETAILED DESCRIPTION

[0041] To achieve the benefits associated with a fully automated tool for image enhancement, this paper describes a learning-based approach based on supervised learning in the form of input and output (augmented) image pairs.

[0042] The image enhancement method described here enables different types of local image edits and constitutes a method to jointly apply these edits to visually improve a provided input image. The proposed model is able to enhance images by learning to appropriately apply spatially local filters that draw inspiration from tools commonly used by human digital artists. Local filters provide additional fine-grained control over image edits, thereby offering refinement and detail enhancement that is not achievable through global modifications alone. By constraining our model to learn to use tools similar to those in the digital artist's toolbox, we provide a natural form of model regularization and enable interpretable, intuitive adjustments to achieve visually pleasing results. Given appropriate training data, we can additionally reproduce accurate and recognizable restorations of individual artistic styles.

[0043] This approach involves framing the learning problem as selecting one or more parametric filters that operate locally on the image. These learnable, parameterized components offer several advantages. First, they align well with commonly used intuitive, well-understood human artistic tools, which naturally generate attractive effects through image transformations that are somewhat familiar to human observers. Second, the use of parameterized local filters both constrains model capacity and regularizes the learning process, mitigating overfitting and helping to produce pleasing results at a modest cost in memory space.

[0044] In the context of filters, a parametric filter is a filter with a small set of parameters used to adjust an image. An example of such a parametric filter is the elliptical filter, which operates on an image based on the parameters of an ellipse. Figure 2 illustrates these parameters and an example use case. This particular example uses an elliptical filter (an example of a parametric filter) to brighten an image.

[0045] Figure 2(a) shows an elliptic filter that is parameterized by five variables. These variables are: center position Rotation angle semi-minor axis and semi-major axis The length of the filter can be configured with different positions and sizes using these five adjustable parameters to achieve various image enhancements. Within circle 108, image properties such as brightness or color can be adjusted using a linear scale from 100% at the center (H, K) to 0% at the edge of circle 108. Outside circle 108, the image can remain unchanged. The parameterized representation of the filter allows the filters to be expressive yet easy to interpret because they are inspired by adjustments an artist might make in photo editing software.

[0046] Figure 2(b) shows an example of using an elliptical filter to brighten a human face. Specific properties assigned to the elliptical filter are applied to the face in the photograph based on the position and size of the ellipse 108. In this example, the filter is a circle 108, which is a special case of an ellipse.

[0047] Furthermore, the filters in the methods described herein are local filters in that they operate with a fixed spatial extent. Similar to the example in FIG1 , the filters adjust the image within the radius of circle 108. This process differs from global filters, which transform all pixels in the image in the same way, regardless of their location in the image.

[0048] In the method described herein, machine learning is used to automatically predict the filter parameters. An example of such a learnable model is a neural network. During training, the network learns how to use pairs of source and enhanced images to predict filter parameters that minimize the error between the enhanced image and the ground truth. When presented with a new, unenhanced image during inference, the network predicts the filter parameters, which are then used to adjust the image. The method is implemented by a trained algorithm that includes the results of the trained network and is capable of generating a set of filter parameters and subsequently applying at least one local parametric filter using these parameters as input. In other words, the trained algorithm is a machine learning algorithm.

[0049] The method described here leverages deep learning by using deep learning modules to adjust local image properties using parametric functions. Deep learning modules are reusable components of neural networks that perform specific tasks. Modules can be inserted at different locations within a neural network. Therefore, the method described here can be combined with other existing networks to perform image enhancement. The modules operate locally, meaning they have a limited spatial extent on the image. This restricts processing to specific regions of the image.

[0050] The use of parametric functions allows the described methods to have simple mathematical models describing the effects of filters on images. These functions are easily understood by the user because they can be visualized. If desired, they can also be manually adjusted in a post-processing step. The parametric functions implicitly provide a form of smoothing or regularization for the enhancement they produce. They can also be efficiently estimated using neural networks.

[0051] The present method is described in conjunction with three examples of local parametric filters. However, it will be appreciated that the method described can be used in conjunction with many other such parametric filters.

[0052] Parameters that are determined to be learnable or predictable scalar parameters are denoted in uppercase bold throughout the document for easier understanding.

[0053] The method described is a deep learning approach for learning parameters of a parametric function that can locally adjust key image attributes. Attributes include color channels such as, but not limited to, red (R), green (G), blue (B), as well as brightness, chroma, hue, and saturation. In the following examples, adjustments are made to each channel in the RGB space, but the method is not limited to this color space (for example, other color spaces include CIELAB or LAB, YUV, HSL / HSV, and CMYK).

[0054] The currently described approach includes two key elements:

[0055] Use local parametric filters.

[0056] Use deep neural networks to predict the parameters of these local parametric filters.

[0057] The learned parametric function (also referred to herein as a "filter") can be any parametric function that can adjust image properties in a local region of the image. The parametric nature and local effect of the filter are key properties that distinguish this method from typical methods.

[0058] In one embodiment of the adjustments performed, the filter adjusts the value of each pixel by multiplying it by a learned scalar value. The scalar can take any value between 0 and a preset upper limit (e.g., 2, 3, 4, 5, etc.), thereby increasing and decreasing the selected adjustment for a given pixel. In other embodiments, the adjustment can be performed using addition of adjustment factors or other mathematical operations such as division, subtraction, or a combination thereof. One or more filters can be applied to the same image attribute. For example, in the LAB color space, there can be three filters operating on the image luminance channel.

[0059] The filters can take any parametric form, as long as they adjust properties in a local, constrained region of the image. This paper describes three possible implementations of parametric local filters, each with a small number of deep neural networks that can learn the predicted parameters.

[0060] The method described herein will now be described with reference to two specific filter examples, an elliptical filter and a gradient filter. In both examples, there are so-called 100% scale regions and 0% scale regions. This means that within the 100% scale region, 100% of the prediction scale factor is applied. The area can include a single pixel or an area containing multiple pixels. In other words, the pixels in the 100% area are multiplied by Within the 0% scale area, pixels are multiplied by 1, meaning no adjustment is made and a 0% scale factor is applied. (i.e., not applied at all.) The difference between the Elliptical filter and the Gradient filter is how the scale factor is used between the 100% and 0% scale areas. Converts the scale factor value between 0 and 1.

[0061] In the first example, the parametric function is an ellipse function that uses multiplication to adjust pixels that are inside or outside the ellipse boundary 108. Figure 3(a) shows the standard equation for a rotated ellipse. The rotated ellipse 108 is parameterized by five learnable parameters that determine its position and orientation in the image. Defines the length of the semi-major axis. Defines the length of the semi-minor axis. and Defines the coordinates of the center of the ellipse. Defines the semi-major axis The rotation angle from the horizontal in a counterclockwise direction.

[0062] In addition, the parameter Defines the scale applied to pixels inside or outside the ellipse. Given a prediction scale factor The scale factor decays according to a linear or non-linear function that determines how the scale factor is gradually modified as pixels move further away from the center of the ellipse.

[0063] There are two possible versions of the elliptical filter. The first version is when the 100% scale area is located at the center of the ellipse 108, and the 0% scale area is located outside the ellipse. The second version is when the 100% scale area is located outside the ellipse, and the 0% scale area is located inside the ellipse. This second version is called the inverted structure. The principle of the elliptical filter is to apply local image adjustments within the scene area in an elliptical manner.

[0064] Figure 3(b) shows a schematic diagram of the first version of the elliptical parametric filter. Apply a prediction scale factor of 100% The scale factor is then gradually adjusted until it reaches 1 at the boundary of the ellipse 108. Pixels outside the ellipse are not scaled (ie, the scale factor is 1).

[0065] Figure 3(c) shows an example of applying an elliptical (or radial) filter to a photograph to adjust the redness of a person's face. On the left is the image 302 to be adjusted, with the elliptical boundaries of the filter 108 displayed on the image. On the right is a heat map 304 showing the results of the filter on the red channel of the image 302. Arrows 306 indicate the area of ​​the heat map 304 that represents the changes made within the filter ellipse in the red channel of the image 302 to adjust the redness of the person's face.

[0066] In the second example, the learned parameter function is a gradient filter. The gradient filter is characterized by three parallel lines, the top line 102, the center line 104 and the bottom line 106, where the center line intercept Centerline slope and vertical offset between lines is defined as a learnable parameter. In addition, the learned scaling factor is applied in the 100% scale region The 100% scale region is located above the uppermost line 102 (when referred to as the non-inverted version) or below the lowermost line 106 (when referred to as the inverted embodiment). Between the lines, the scale factor Gradually adjusts it linearly until it reaches 1 at the boundary of the line farthest from the 100% scale region (depending on the version used, this may be the boundary of the bottommost line 106 or the topmost line 102). These learnable parameters are predicted by a deep learning network. The principle of the gradient filter is to gradually apply an adjustment (e.g., brightness change) in a linear manner to a local area of ​​the scene.

[0067] Figure 4(a) shows a schematic diagram of a gradient parameter filter. The image height dimension is shown by the dotted line 402 on the left. The 100% scale region is the region above the topmost line 102. In this region, the scale factor Apply the selected adjustment scale factor (e.g., increase or decrease brightness). Between lines, the scaling factor The linear adjustment is 50% at the center line 104 and 0% at the bottom line 106. The steepness of the adjustment depends on the spacing between the lines, for example, closer lines indicate a sharper adjustment. The 0% scale area leaves the pixels in that area unchanged (i.e., a scale factor of 1).

[0068] FIG4( b ) shows an example of applying a combination of two gradient filters to the blue channel (B channel) in the form of a heat map 404 on the left, and the adjustments applied to the flower image 406 on the right.

[0069] In another example, the local parameter filter can be a cubic function that depends on the pixel position and pixel value, and the equation defining the parameters is shown in Figure 5(a). There are twenty learnable parameters (denoted as and ), these parameters define a four-dimensional surface relating pixel coordinates and values ​​to adjustment factors added to the input image (e.g., brightness values, color values, saturation values, etc.). Other versions of this filter can be characterized by the order of the polynomial, such as linear, quadratic, quartic, etc. However, it has been found that the cubic filter provides very good image quality while alleviating concerns about excessive flexibility, which is essentially overfitting the data. The cubic filter function effectively defines a smooth surface on the image that smoothly varies the adjustment scaling factor across the image. This surface can be applied to any type of pixel value, such as intensity (brightness), color, saturation, hue, etc. In contrast to the gradient and radial filters, this particular filter does not have 100% and 0% areas. Furthermore, in the optimal formulation for the cubic filter, the calculated adjustment is applied to the image in an additive manner.

[0070] For all filter types, the output of each filter is a two-dimensional adjustment map of the same width and height as the image. At each pixel position in the adjustment map, there is a scalar value generated by the filter, called a scale factor, which is used to directly replace or adjust that pixel in the corresponding image. There are many mechanisms for using the adjustment map to modify the image, the simplest being to multiply it with the corresponding pixel in the image. However, there are other methods which we will describe later. The adjustment map can be visually presented here as a heat map.

[0071] Figure 5(a) shows a cubic filter equation, where x and y are the horizontal and vertical coordinates of a pixel within an image, and z is the pixel value (e.g. luminance, hue or saturation). The output of the function F(x, y, z) is added to the image to generate an adjusted image. This equation defines a smoothly varying four-dimensional surface.

[0072] Figure 5(b) shows the effect of the cubic filter on the left image 502 as a heat map 504. Different colours or temperatures on the scale in the heat map 504 can be used to indicate the strength of the adjustment applied to the image 502 at each pixel position by the cubic filter. In this example, the adjustment is being made in the blue channel of the image 502. With more similar scale factors modifying positions adjacent to each other in the image, the resulting image has locally spatially smooth adjustments. That is, sharp changes in the filter application level within the image are avoided.

[0073] Some specific examples of local parametric filters have been defined and discussed above. Each filter has a set of parameters which can be learned, which define the specific form of that filter and bring the best image enhancement results for the task at hand. For example, the placement position of a radial filter, its size and the strength of the adjustment it makes in the image can be learned. The parameters are learned by training a deep neural network based on a training dataset consisting of example inputs and ground truth images. In this case, the input image is the image to be adjusted, and the ground truth image specifies the result that is desired to be achieved by applying one or more parametric filters to that input image.

[0074] A specific deep learning framework will now be described which can be used to implement the regression of filter parameters in this supervised learning setting.

[0075] Figure 6 A schematic diagram of a regression neural network which can be used to compute the parameters of a parametric filter is shown. It will be appreciated that any regression neural network architecture can be used conceptually to infer the parameters, so the described approach is not limited to this specific example using this architecture.

[0076] The neural network receives a tensor or feature tensor composed of convolutional neural network features of shape H x W x C. Where H is the height spatial dimension, W is the width spatial dimension, and C is the channel (feature) dimension. This feature tensor can be provided by any backbone neural network capable of generating a set of convolutional feature maps, such as U-Net, ResNet, etc. Once given these features, the method will apply a set of alternating convolutional and max-pooling operations, down-sampling the feature tensor in the spatial dimensions H and W, giving a feature map of shape H / 128, W / 128, C. This feature map captures a global, holistic understanding of the image content. The down-sampled feature tensor is input into a global average pooling layer, which folds the spatial dimensions to H = 1, W = 1, followed by a fully connected layer that regresses the N parameters of the prediction (and learning) parameter filter.

[0077] Figure 6 A flowchart 600 of a regression neural network that predicts (and learns) the N parameters of a parameter filter is shown. The convolutional feature map 602 is input from a backbone network (e.g. U-Net, as described in the following article: Ronneberger, O., et al. U-Net: Convolutional Networks for Biomedical Image Segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2015; or ResNet, as described in the following article: He, Kaiming, et al. Deep Residual Learning for Image Recognition. CVPR, 2016). The feature map is passed through a series of convolutional layers and max-pooling operators 604, which have the effect of down-sampling the feature map 606. A global average pooling layer 608 folds the spatial dimensions and the resulting feature map 610 is fed to a fully connected layer 612 that regresses the N parameters.

[0078] An image in any color space (e.g. HSV, RGB, LAB, etc.) has three channels. There can be one or more local filters applied to each channel. To mix together the results of multiple filters of the same type for each channel, the method proposes a filter fusion architecture, an example of which is shown in Figure 7 In one simple example of this fusion architecture, the outputs of the filters are multiplied together. The result of this multiplication is then applied to the original image to perform the overall modification. In another example, the outputs can be added to the original image. In yet another example, the fusion can be done by learning how to fuse the different filters together, for example by using a learned scalar for each filter application to fuse.

[0079] Figure 7A schematic diagram of a filter-type-specific fusion block is shown. The outputs of filters of the same type are fused. A deep neural network architecture is used to combine the image enhancement results of multiple filters applied to a given image channel. In this example, three filters 702, 704, and 706 are learned for the R channel of an RGB image. In this example, the filters are blended by multiplying the scaling factors of each filter by 708 on a pixel-by-pixel basis.

[0080] It is also possible to have multiple filters of different types per channel of an image: for example, three elliptical filters and two gradient filters, and a cubic filter operating on the same channel (e.g. the R channel in RGB color space). This mixture of different filter types is applied in two steps. In the first step, the outputs of filters of the same type are combined using the fusion described previously and Figure 7 As shown in the example in . In the second step, given the fused results from filters of the same type, the method then blends these results to obtain an overall adjustment applied to the image. This blending can be done in a parallel or sequential manner, the difference being whether the modifications are applied to the image all at once (parallel approach) or whether filter-type-specific modifications are applied one after another in a pipeline, with the results fed from one step to the next (sequential approach).

[0081] In a parallel approach, an example could be the unweighted multiplication of the fused outputs from each filter type, as in Figure 8 As shown. This blending can also be performed by learning a set of scalars for combining the fused results before multiplication or addition. It should be understood that there are multiple alternative ways to combine the different results of multiple filters. For example, the addition of the fused outputs from each filter type can be used to obtain an overall adjustment map to be applied to the input image. Alternatively or additionally, the fused results from a subset of filter types (e.g., gradient, elliptical) can be multiplied together. After this, the fused result from another filter type (e.g., cubic filter) can be added to the input image, and then this adjusted image is multiplied by the fused adjustment map produced by the gradient and elliptical filters. It should be understood that the fusion functions (addition, multiplication, subtraction, division, etc.) can be used in any combination to obtain the desired adjustment map. Figure 9 Examples of combining different filter results using different types of fusion are shown.

[0082] Figure 8A schematic diagram illustrates an example of fusing filter outputs using a parallel approach. In this example, the fused outputs from three different filter types—elliptical 802, gradient 804, and cubic 806—are combined to form an overall image adjustment map, received from a filter-type-specific fusion block 808. This overall adjustment is obtained by multiplying the adjustment maps from these three different filters together 810. The resulting adjustment map is then applied to the image (in this case, via a final multiplication) to obtain an enhanced result 812.

[0083] Figure 9 A schematic diagram illustrates an example of fusing filter outputs using different parallel methods. In this example, the fused outputs from three different filter types—elliptical 802, gradient 804, and cubic 806—are combined to form an overall image adjustment map, received from the filter-type-specific fusion block 808. This overall adjustment is achieved by multiplying 902 the two adjustment maps 802 and 804 from one or more elliptical and gradient filters. The result 904 is multiplied 906 by adding 910 the cubic adjustment map 806 to the input image 912, giving an image 908.

[0084] Alternatively, the fused results of each filter type can be applied to the image individually in sequence, i.e. one at a time, as in Figure 10 For example, the fused result of three elliptic filters can be applied to an input image, and then the result is adjusted by the fused result of two gradient filters (i.e., applied to the output image of the previously applied elliptic filters), and then the result is adjusted using a cubic filter. That is, each fused result of the subsequent filter type is applied to the image one after another until all filters have been applied to the image in some form or combination.

[0085] Figure 10 A diagram showing an example of using a sequential approach to filter output fusion. In this example, the fused outputs from three different filter types—elliptical 802, gradient 804, and cubic 806—are combined by applying them sequentially to the input image, one at a time. The output of one adjustment is fed into the next.

[0086] It is also possible to combine sequential and parallel fusion methods. For example, an image adjusted by a cubic filter's adjustment map can be fed into a parallel fusion architecture that combines the elliptical adjustment map and the gradient adjustment map using multiplication. The result can be multiplied by the enhanced image from the cubic filtering stage.

[0087] Figure 11A schematic diagram shows an example of a camera implementing an image processor to process images captured by an image sensor 1102 in a camera 1101. Such cameras 1101 typically have some onboard processing capability. This may be provided by a processor 1104. The processor 1104 may also be used for basic device functions. The camera typically also includes memory 1103.

[0088] The transceiver 1105 is capable of communicating with other entities 1110, 1111 via a network. These entities may be physically remote from the camera 1101. The network may be a publicly accessible network, such as the Internet. The entities 1110, 1111 may be cloud-based. In one example, entity 1110 is a computing entity and entity 1111 is a command and control entity. These entities are logical entities. In practice, each of them may be provided by one or more physical devices (e.g., a server and a data store), and the functionality of two or more entities may be provided by a single physical device. Each physical device that implements an entity includes a processor and a memory. The device may also include a transceiver for sending data to and receiving data from the transceiver 1105 of the camera 1101. The memory stores, in a non-transient manner, code that can be executed by the processor to implement the corresponding entity in the manner described herein.

[0089] The command and control entity 1111 can train the artificial intelligence models used in each module of the system. This is typically a computationally intensive task, even if the resulting models can be efficiently described, so it may be more efficient to perform algorithm development in the cloud, where significant energy and computing resources are expected to be available. This can be expected to be more efficient than developing such models on a typical camera.

[0090] In one implementation, once a deep learning algorithm is developed in the cloud, the command and control entity can automatically form a corresponding model and transmit it to the relevant camera device. In this example, the system is implemented by the processor 1104 at the camera 1101.

[0091] In another possible implementation, an image may be captured by the camera sensor 1102 and the image data may be sent to the cloud by the transceiver 1105 for processing in the system. The resulting target image may then be sent back to the camera 1101, as shown in FIG. Figure 11 As shown in 1112.

[0092] Therefore, the method can be deployed in a variety of ways, such as in the cloud, on-device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or improve existing ones. Depending on the computing power close to the data corpus, training can be performed close to the source data or in the cloud, for example using an inference engine. The system can also be implemented in a camera, dedicated hardware, or in the cloud.

[0093] It should be understood that the term "enhancement" is a well-known term or technique in the field of image processing and does not necessarily imply any objective improvement. That is, an image can be enhanced without significantly improving its objective or subjective quality. Therefore, in this article, the term "transforming" an image is synonymous with the term "enhancing" an image.

[0094] The above method has the following advantages:

[0095] The method can be automated through machine learning. It leverages recent advances in machine learning, namely deep learning, to automatically predict parametric filters for image enhancement. The automation provided by machine learning can reduce the manual effort required to adjust the image.

[0096] The method can produce high image quality. The local parameter filter learned by the method can produce high image quality.

[0097] This method provides a human-inspired and interpretable image editing process. The filters in this method are inspired by the image editing tools available to users in software programs such as Photoshop and Lightroom. Users can easily understand the filters applied to the image. This differs from many neural network methods that act as "black boxes," that is, they take an image as input and generate an enhanced image as output, but expose limited information about the network's inner workings to the user. Human-interpretable image editing has several advantages. The learned filters can be visualized, allowing users to understand how the described method enhances the image. Furthermore, it provides a mechanism for users to manually adjust the processing as post-processing by interacting with the predicted filters.

[0098] This approach can improve parameter efficiency. Neural networks typically rely on many weights to transform inputs into outputs. Simple parametric filters can be learned using a smaller set of weights than standard neural network models, yet in some implementations this can still yield better results. Fewer weights can actually mean less memory and computational overhead during inference. In other words, a neural network has a set of model "parameters" or weights. These parameters are learned during model training. The trained model is then able to predict the correct filter parameters for a parametric image filter to enhance an image.

[0099] The method provides image processing with built-in regularization. The parametric filter is spatially smooth, allowing for gradual adjustments to the image. Consequently, the adjustments applied to the image are also smooth, avoiding noticeable seams or irregularities.

[0100] The method provides a reusable module. The method can be thought of as a reusable neural network module. It can be combined with other networks in various ways to enhance images.

[0101] The method described provides a scalable image resizing process. Specifically, we consider three parametric filters. However, the framework is extensible to other parametric filters, especially those that operate locally on the image. For example, the framework can also be combined with global filters.

[0102] Applicants hereby disclose individually each individual feature described herein and any combination of two or more such features. It would be common knowledge for a person skilled in the art to be able to implement such features or combinations as a whole based on this specification, without regard to whether such features or combinations of features solve any of the problems disclosed herein, and without limiting the scope of the claims. Applicants indicate that aspects of the present invention may consist of any such individual features or combinations of features. In view of the foregoing description, it will be apparent to a person skilled in the art that various modifications may be made within the scope of the present invention.

Claims

1. An image processing device (1101) for transforming an image, characterized in that The apparatus comprises a processor (1104) configured to implement a trained algorithm to process a received first image and generate a parameter set based on the first image; and to generate a second image by applying a local parametric filter to the first image, the local parametric filter using the parameter set as input; Wherein, when the parameter set includes at least A, B, T, H, and K, the local parameter filter is an elliptical filter; in the parameter set, A represents the semi-major axis of the ellipse, B represents the semi-minor axis of the ellipse, T represents the rotation angle, and H and K represent the center position of the ellipse; or, when the parameter set includes at least C, M and O, the local parameter filter is a gradient filter; the gradient filter has three parallel lines; in the parameter set, C represents the center line intercept, M represents the center line slope, and O represents the vertical offset between lines, or, when the parameter set includes at least pixel position and pixel value, the local parameter filter is a polynomial filter; the local parameter filter is used to apply different types of transformations to different areas of the first image, and the parameter set includes one or more parameters, and the one or more parameters are used to determine the area in the first image to which the local parameter filter is applied to perform a specific type of transformation; the second image is obtained after enhancing the first image.

2. The image processing device according to claim 1, wherein The particular kind of transformation of the local parametric filter depends on parameters input to the local parametric filter, and the parameter set includes one or more parameters for determining the particular kind of transformation of the local parametric filter.

3. The image processing device according to claim 1, wherein The local parameter filter is used to adjust one or more of hue, saturation, chroma, brightness and color channel balance.

4. The image processing device according to claim 1, wherein The first image includes data in a set of a plurality of color channels, and the local parametric filter adjusts fewer than all of the color channels.

5. The image processing device according to claim 1, wherein The trained algorithm is a machine learning algorithm.

6. The image processing device according to claim 5, characterized in that The trained algorithm processes the first image by identifying features in the first image, and the device is configured to generate the parameter set based on the identified features.

7. The image processing device according to claim 6, wherein The apparatus includes a memory that stores, in a non-transitory manner, code executable by the processor to implement the trained algorithm and the filter.

8. The image processing device according to claim 1, wherein The apparatus is configured to generate the second image by applying a plurality of local parametric filters to the first image, each local parametric filter taking as input a subset of the set of parameters.

9. The image processing device according to claim 8, wherein At least two of the plurality of local parametric filters have filter functions different from each other.

10. The image processing device according to claim 8 or 9, characterized in that At least two of the plurality of local parametric filters have the same filter function but are applied at maximum intensity at different locations in the first image.

11. The image processing device according to claim 10, wherein The first image includes data in a set of a plurality of color channels, one of the plurality of local parametric filters adjusts a first subset of the color channels, and a second of the plurality of local parametric filters adjusts a second subset of the color channels different from the first subset.

12. A method for transforming an image, characterized in that The method comprises: receiving a first image; processing the first image through a trained algorithm to generate a parameter set based on the first image; generating a second image by applying a local parametric filter to the first image, the local parametric filter using the set of parameters as input; Wherein, when the parameter set includes at least A, B, T, H, and K, the local parameter filter is an elliptical filter; in the parameter set, A represents the semi-major axis of the ellipse, B represents the semi-minor axis of the ellipse, T represents the rotation angle, and H and K represent the center position of the ellipse; or, when the parameter set includes at least C, M and O, the local parameter filter is a gradient filter; the gradient filter has three parallel lines; in the parameter set, C represents the center line intercept, M represents the center line slope, and O represents the vertical offset between lines; or, when the parameter set includes at least pixel position and pixel value, the local parameter filter is a polynomial filter; the local parameter filter is used to apply different types of transformations to different areas of the first image, and the parameter set includes one or more parameters; the one or more parameters are used to determine the area in the first image to which the local parameter filter is applied for a specific type of transformation; the second image is obtained after enhancing the first image.

13. The method according to claim 12, characterized in that The method includes training the algorithm through a machine learning process prior to the processing step.

Images