Image Processor

Abstract

An image processing module for implementing an artificial intelligence model that has been trained in multiple parts, wherein the image processing module is used to: receive an input image; implement a first part of the model to determine a first transformation of the image in a first color space; apply the first transformation to the image to form a first adjusted image; implement a second part of the model to determine a second transformation of the image in a second color space; apply the second transformation to the first adjusted image to form a second adjusted image; and output an image derived from the second adjusted image.

Description

Technical Field

[0001] The present invention relates to image formation in digital photography, and more particularly to using a trained artificial intelligence model to execute stages of an Image Signal Processor (ISP) pipeline. Background Art

[0002] In digital photography, the conversion of raw camera sensor data into visually pleasing, high-quality digital images is handled by a series of image processing algorithms based on image signal processing technology, which are collectively referred to as Image Signal Processor (ISP).

[0003] In the prior art, ISP is implemented using signal processing methods. Existing ISPs consist of a large number of modules, for example 20, usually arranged in a sequential pipeline, each module being responsible for a well-defined image processing operation, such as denoising, demosaicing, and color correction. Figure 1 An example of an ISP pipeline is shown. The output of one module is fed to the next module along the pipeline.

[0004] Typically, an ISP has a large number of exposure parameters that need to be tuned for new sensors. As a result, prior art approaches suffer from image quality issues and difficulties in tuning the ISP to produce high quality results.

[0005] At the same time, designing existing ISP pipelines is often very difficult due to the complex dependencies between modules in the pipeline. For example, changes to the denoising module may cause the color correction module to generate undesirable artifacts.

[0006] Advances in deep learning have brought new state-of-the-art methods to many image processing methods that belong to existing ISPs, such as demosaicing and denoising. Results generated using deep neural networks are more visually beautiful than those based on existing signal processing algorithms. Machine learning methods can simplify optimization and are less dependent on large datasets. Several ISPs using artificial intelligence have been proposed.

[0007] DeepISP, as described in “DeepISP: Learning End-to-End Image Processing Pipeline” by E. Schwartz, R. Gillies, A. Bronstein, published on ArXiv in 2018, is a single-network deep learning approach that takes raw data and uses a simple bilinear interpolation method as input to a low-level network that performs denoising and demosaicing. The high-level network then learns dynamic range correction and tone mapping. This approach replaces the ISP pipeline with a single end-to-end trainable deep neural network. This model requires a set of images that represent the desired mapping from the sensor raw data to the output digital image. DeepISP has been shown to produce high-quality output images that, based on human visual inspection, are considered more aesthetically pleasing than those produced by hardware-based ISPs. However, this approach does not ensure consistency between images of different scales when performing local pixel adjustments. In addition, this approach does not allow for adjusting properties of an image independently of other properties.

[0008] We hope to develop an improved ISP based on a deep learning component that ensures consistency between images of different scales when performing local pixel adjustments and allows independent adjustment of global image properties to provide high-quality images. Summary of the invention

[0009] According to a first aspect, there is provided an image processing module for implementing a multi-part trained artificial intelligence model, wherein the image processing module is configured to: receive an input image; implement a first part of the model to determine a first transformation of the image in a first color space; apply the first transformation to the image to form a first adjusted image; implement a second part of the model to determine a second transformation of the image in a second color space; apply the second transformation to the first adjusted image to form a second adjusted image; and output an image derived from the second adjusted image. The operation of the module can improve image quality by independently adjusting properties of the image in different color spaces.

[0010] The module may also be used to: receive the second adjusted image; implement the third part of the model to determine a third transformation of the image in a third color space; apply the third transformation to the image to form a third adjusted image; and output an image derived from the third adjusted image. Each of the first, second, and third color spaces may be a color space in the following group: an RGB space, an HSV space, and a Lab space. Adjustments of all three spaces (RGB, Lab, and HSV) are particularly beneficial for achieving high image quality.

[0011] At least one of the first transform, the second transform, and / or the third transform is a piecewise linear scaling curve. Compared to typical deep learning-based models, the curve layer provides human-interpretable insights into how the network performs mapping because the curve can be easily read and understood by humans. In other implementations, one or more transformations can be represented as parameters of a surface or equation.

[0012] Each of the first scaling curve, the second scaling curve and the third scaling curve may include nodes determined by the first part and the second part of the model respectively. Adding nodes increases the modeling flexibility of the curve and is conducive to modeling complex relationships between inputs and outputs.

[0013] The module can be used to apply the corresponding transformations to the images corresponding to these transformations so as to adjust the properties of the corresponding images in the corresponding color space. This facilitates independent adjustment of image properties.

[0014] The first transform and the second transform may associate a property of the image in the respective color space with a scaling factor represented as a scaling curve. The scaling curve may be applied to the image in a color space to adjust the property of the image.

[0015] The properties of the image may be color, brightness, saturation or hue. Adjustment of these properties may produce a more aesthetically pleasing image.

[0016] The module may be used to adjust properties of the corresponding image at a global scale of the image.

[0017] According to a second aspect, a processing system is provided, comprising the image processing module described above and a second module for implementing an artificial intelligence model, wherein the image processing module described above is used to receive a combined feature map formed by the second module and process the feature map as an input image, and the second module is used to: receive an input feature map of an original image, wherein the input feature map includes data on multiple channels for each of multiple positions in the image; extract features from the input feature map at a scale less than the global scale of the input feature map to form an intermediate-level feature map; extract features from the input feature map at a global scale of the input feature map to form a global feature map; combine the intermediate-level feature map and the global feature map with the input feature map to form a combined feature map. This method can help ensure consistency between different scales of the image when performing local pixel adjustments.

[0018] The second module can be used to extract features from the input feature map to form the intermediate level feature map and the global feature map using a corresponding convolutional neural network. Using deep neural networks in image processing can produce more visual results than results based on traditional signal processing algorithms. Machine learning methods can simplify optimization and are less dependent on large data sets.

[0019] The second module may also be used to extract features from the input feature map using maximum pooling at a global scale of the input feature map. The second module may also be used to fuse the combined feature map through 1×1 convolution.

[0020] The system can be used to perform one or more of the following on the original image: demosaicing, denoising, local brightness correction, local color correction, global brightness correction or global color correction. The pipeline-based ISP method is replaced with a single neural network that learns mappings directly from data and can be jointly optimized, performing demosaicing, denoising and color correction in a single neural network architecture, and allowing the synergy between these operations to be captured and exploited.

[0021] According to a third aspect, a method for transforming an original image captured by a camera in an image processor is provided. The processor is used to implement a multi-part artificial intelligence model, and the method includes: receiving an input image; implementing a first part of the model to determine a first scaling curve of the image in a first color space; applying the first scaling curve to the image to form a first adjusted image; implementing a second part of the model to determine a second scaling curve of the image in a second color space; applying the second scaling curve to the first adjusted image to form a second adjusted image; and outputting an image derived from the second adjusted image. The method can improve image quality by allowing independent adjustment of image properties in different color spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will now be described by way of example with reference to the accompanying drawings. In the accompanying drawings:

[0023] Figure 1 A conventional ISP is shown consisting of a large number of operations implemented using signal processing methods.

[0024] Figure 2 A neural network-based image processing architecture with low-level and high-level blocks is shown.

[0025] Figure 3 (a) shows a mosaic image. At each pixel, blue, green or red is acquired. Figure 3In (b), the mosaic image is packed into four color channels representing R, G1, G2, and B. In the packed form, the spatial resolution of each color channel is half the resolution of the original mosaic image.

[0026] Figure 4 Shows Figure 2 How the low-level blocks of the architecture are implemented.

[0027] Figure 5 Shows Figure 4 An example of a skip connection architecture with multi-scale contextual feature fusion for low-level blocks of .

[0028] Figure 6 An example of an operation method of a low-level block is shown.

[0029] Figure 7 Shows Figure 2 How the high-level blocks of the architecture are implemented.

[0030] Figure 8 An example of a piecewise linear scaling curve for adjusting hue according to saturation is shown.

[0031] Fig. 9 An example of a piecewise linear scaling curve for adjusting brightness is shown.

[0032] Fig.10 An example of a method for transforming a raw image captured by a camera in an image processor for implementing a multi-part artificial intelligence model is shown.

[0033] Fig.11 Results obtained using the methods described herein are shown. Fig.10 (a) shows an image input. Due to the large dynamic range content, the image is visible in dark colors. Fig.10 (b) shows the output image. The image brightness is corrected, and the color and exposure are greatly improved.

[0034] Fig.12 An example of a camera for implementing the methods described herein is shown. DETAILED DESCRIPTION

[0035] The present invention relates to the use of an AI-enabled ISP to generate optimal image quality using deep learning. The traditional ISP pipeline is replaced by a single end-to-end trainable neural network that can learn an ISP mapping from raw data to output high-quality images based on a representative training data set that includes input raw data and output digital image pairs. The neural network performs the core functions of traditional ISP image processing, namely denoising, demosaicing, and color correction. Denoising can reduce the generation of noise in the image. The denoising stage can be limited to denoising the raw data, or optionally as an additional stage performed on RGB data. Demosaicing interpolates the raw data to generate a color (RGB) image. These operations can be performed in different orders in the network. During the training phase, these operations are uniformly optimized by the neural network.

[0036] Figure 2 An example of a neural network architecture is shown. A raw mosaic image is input to the network at 20. The raw sensor input 20 is a noisy mosaic pattern consisting of red, green and blue color values, with one value per pixel position, consistent with a specific pattern called a Bayer matrix. The architecture comprises two interconnected neural network blocks 21, 22. Block 21 is referred to herein as a low-level network block. The low-level block 21 performs local pixel adjustments, i.e., demosaicing, denoising and correction of local brightness and color in the image. Block 22 is referred to herein as a high-level block. The high-level blocks include a feature extraction block 23 and a parameter prediction block 24. At Figure 2 In the example of , the parameter prediction block includes a series of neural curve layers 25, 26, 27 that globally adjust the image properties. The high-level block can adjust the image brightness, color, and saturation (based on hue). The image block can also be enhanced by transformations such as saturation adjusted based on saturation and hue adjusted based on hue, gamma correction, contrast adjustment, or exposure adjustment. The curve layer is a fully learnable component of the image processing architecture and does not have to be specified in advance by the user. Therefore, the high-level (global) image processing block 22 uses a neural curve layer to adjust the properties of the image range. The output 28 of the neural network is a color-corrected RGB frame with a dynamic range suitable for display on a standard device (e.g., a device with 256 levels per color channel).

[0037] Figure 2 The neural network architecture is trained end-to-end. A single end-to-end trainable neural network can learn an ISP mapping from input data 20 to a high-quality image output 28 based on a representative training dataset that includes input raw data and output digital image pairs. The image pairs constitute an example of the network, illustrating how to transform input raw data to produce a high-quality output digital image that is ready for consumption by an end user.

[0038] The following will refer to Figures 3 to 5The operation of the low-level block 31 is described in more detail.

[0039] The input of the low-level block is the raw mosaic image from the camera sensor ( Figure 2 20 in the above). Figure 3 (a) shows an example of a mosaic image. In the mosaic image, each pixel has a red value of 30, a green value of 31, or a blue value of 32. The network demosaicizes this image. The goal of demosaicing is to produce an RGB image where each pixel has a red value, a green value, and a blue value. The low-level blocks use a convolutional neural network (CNN) to process the original mosaic image. CNN learns a set of filters that are applied to the image via convolution. Convolution is designed to be spatially invariant, i.e., convolution has the same effect when applied anywhere in the image. There is a problem in applying convolution on a mosaic image: due to the presence of the colour filter array (CFA), the convolution is no longer spatially invariant. For example, when the filter is centered on a blue pixel, the effect may be different than when it is centered on a red pixel. This problem can be solved by packing the data into channels of similar colors, and then each channel can be processed in the CNN using a spatially invariant convolution. In Figure 3 In (b), the mosaic image is packed into four color channels 33, 34, 35 and 36 representing R, G1, G2 and B colors respectively. The low-level block receives the packed color channels as its input. The resolution of the packed image is half of the full RGB image.

[0040] In a preferred embodiment, the system is implemented as an encoder-decoder neural network architecture according to the design of the performance Unet model, as described in "U-Net: Convolutional Networks for Biomedical Image Segmentation" published by Olaf Ronneberger, Philipp Fischer, and Thomas Brox in MICCAI in 2015. Figure 4 As shown, the encoder-decoder architecture includes a contraction (downsampling) path on the left and an expansion (upsampling) path on the right. The downsampling path allows learning what the image is about, e.g., a dog in a field, while the upsampling path allows learning where salient objects and details are located.

[0041] The downsampling path increases the receptive field of the model, allowing increasingly greater degrees of image context to be considered in image processing operations. Each downsampling operation is technically implemented by a max pooling operator, shown by a class of arrows at type 40, which takes the maximum value of a set of feature values ​​in each local neighborhood. The resulting downsampled feature map is followed by a double convolution block, shown by a class of arrows at 41. As shown at 41a, the double convolution block consists of a 3×3 convolution followed by a ReLU activation, and another 3×3 convolution followed by a ReLU activation. This double convolution block doubles the number of channels in the feature map. The downsampling path terminates at a “bottleneck layer” 42, which is Figure 4 The lowest level of the architecture shown. The bottleneck layer 42 can be viewed as a distillation of the “what” of the image, i.e. important image properties such as global color distribution, global brightness distribution and coarse-grained image structure.

[0042] Then, the feature map from the bottleneck layer 42, which has the lowest spatial resolution but the most channels, is progressively upsampled in the expansion path, as Figure 4 This upsampling path progressively increases the spatial resolution of the feature map by a factor of 2 in each dimension until the feature map has the same width and height as the input packed image, which is half the height and width of the full RGB image.

[0043] In one non-limiting embodiment, upsampling is performed using bilinear interpolation, shown by a class of arrows at 43, followed by a 1×1 convolutional layer. Since high-frequency image details will inevitably be lost in the contracting path, image features from the contracting path are passed directly to the corresponding part of the expanding path via skip connections 44. These skip connections ensure that high-resolution, high-frequency image details can be fully restored in the upsampling path.

[0044] In order to further improve the information capacity passed along the skip connection 44, in the encoder-decoder architecture of the present invention, the features passed along the skip connection are enhanced by additional global and intermediate level features, which can enhance the preservation of global, intermediate level and local image properties in the upsampling path.

[0045] The fusion of multiple different context image features in the encoder-decoder skip connection is denoted as “multi-scale context fusion” skip connection. Figure 5 An example of a technical implementation of the skip connection 44 is shown in FIG.

[0046] An input feature map 50 of height H and width W includes data in multiple channels C, where regions of the feature map correspond to multiple locations in the image. Along paths 52 and 53, convolutional layers with dilation rates 2 and 4, respectively, are used to achieve a larger receptive field and extract intermediate-level context feature maps from the input feature map 50 to form intermediate-level feature maps 55 and 56, respectively. The intermediate-level feature maps 55, 56 have the same spatial resolution as the input tensor 50 to the block. On path 51, a series of convolutional layers with a stride of 2 are used to extract global image features, followed by ReLU activations and then maximum pooling operations. These layers are then followed by global average pooling and fully connected layers.

[0047] The fully connected layer outputs a fixed-dimensional feature vector 54 that is replicated across the height and width dimensions of the input. Feature maps 54, 55, 56 from the intermediate layers and the global feature extraction network are concatenated to the input at 57, and the combined tensor is fused via a 1×1 convolution, shown at 58, to produce a tensor 59 with a much smaller number of channels than the input tensor 50.

[0048] This fusion operation combines local, intermediate level, and global information in a way that is effective for the raw-to-RGB dialogue task. The output feature map 59 from the multi-scale context fusion skip connection is concatenated with the feature map of the upsampling path at this particular level.

[0049] As shown in 45, Figure 4 The output of the extended path of the low-level block is a set of feature maps whose width and height are exactly half of the full RGB image. These feature maps are added to the input packed image, shown at 46, which has been replicated four times along the channel dimension via long skip connections. In this way, the low-level block effectively learns the residual adjustments to be applied to the packed input image to produce the packed output image. This residual connection facilitates the gradient propagation of the deeper neural network. As shown by the class of arrows at 47, the final convolution is applied to the feature map with 256 convolution filters, resulting in an output feature map with 256 channels, shown at 48. Finally, shown at 49, the reshape operation, also known as pixel reorganization (shuffle), rearranges the elements in the tensor of shape (C×r^2, H, W) into a tensor of shape (C, H×r, W×r), where, in this case, r is 2. The output of the low-level block is a tensor of shape W×H×64, where 64 is the number of feature maps / channels, and this tensor serves as the input to the high-level block.

[0050] To shorten training time and reduce the number of network parameters, each skip connection preferably shares the same multi-scale context fusion block parameters. However, different parameters may require more training time and a larger neural network.

[0051] Figure 6 An example of an operating method for transforming a low-level block of an original image captured by a camera in an image processor for implementing a trained artificial intelligence model is shown. In step 601, the method includes receiving an input feature map of an original image, wherein the input feature map includes data on multiple channels for each of a plurality of locations in the image. The method then proceeds to step 602, extracting features from the input feature map at a scale less than a global scale of the input feature map to form an intermediate-level feature map. In step 603, extracting features from the input feature map at a global scale of the input feature map to form a global feature map. In step 604, the intermediate-level feature map and the global feature map are combined with the input feature map to form a combined feature map.

[0052] An example of the advantage of such skip connections is that information about the global image color distribution is better maintained by such a variant of skip connections, allowing accurate reproduction of image color and brightness. The method described in this article enforces consistency between local pixel adjustments and mid- and high-level image properties (e.g., global color distribution). This enforcement reduces the incidence of unsightly image artifacts. For example, operating low-level pixel attributes without considering the global context may cause the model to make spatially inconsistent local decisions. For example, in the case of a blue sky, the hue of the sky changes abruptly in a local pixel neighborhood. Therefore, the low-level (local) pixel processing neural network block 21 uses a multi-scale context fusion neural network layer when learning local image transformations (i.e., demosaicing, denoising, local brightness, and local color adjustments) to capture the synergistic effects between low-level, mid-level, and high-level image properties. This architecture has been shown to be effective for image conversion tasks that produce high-quality output.

[0053] Additionally, the downsampling performed by the model on high-resolution images enables low-level blocks to process large megapixel images on a single GPU without splitting the image into smaller pieces. Splitting the image into smaller pieces and processing each piece independently poses challenges for existing models that apply convolutions directly on the full-resolution image. This can produce unsightly image artifacts, such as inconsistent colors between adjacent pieces. Figure 4 and Figure 5 The encoder-decoder avoids this problem.

[0054] The output image from the low-level block is a color RGB image with corrected brightness, contrast, color, and exposure.

[0055] The high-level blocks receive a rich set of features from the low-level blocks, represented as a tensor of dimension W×H×C, where W is the input image width, H is the input image height, and C is the number of channels / feature maps produced by the low-level network. The first three channels of this tensor are treated as the image to be globally adjusted, and the remaining channels are the feature set used to learn the transformation that performs image adjustment.

[0056] Non-limiting implementations of high-level block neural network architectures are e.g. Figure 7 In this example, first, the input feature map of dimension H×W×64 is split into two tensors 70 and 71. Tensor 70 represents the image to be corrected of dimension H×W×3, and tensor 71 represents the remaining feature maps of dimension H×W×61.

[0057] To independently adjust image properties such as hue, saturation, color, and brightness, transforms are learned in different standard image processing color spaces / domains. These domains are RGB, Lab, and HSV. Figure 7 In the example of , first, the high-level block learns three piecewise linear scaling curves to adjust the brightness in Lab space, then adjust the color in RGB space, and then adjust the final saturation according to the chromaticity in HSV space. The conversion between HSV, Lab, and RGB color spaces is differentiable, allowing the curves to be constructed using standard deep learning optimization methods.

[0058] As shown in 72, the image is first converted to Lab space in a differentiable manner, allowing end-to-end learning of parameters in high-level blocks. Lab space is useful because it separates brightness (L channel) from color (a, b channels). Operating on pixels in this space allows precise adjustment of image brightness without affecting color. The Lab image is concatenated with the H×W×61 features to form a H×W×64 tensor, which is input into the feature extraction block 73. The feature extraction block 73 in Lab space consists of a series of convolutional layers, ReLU and maximum pooling operations, followed by global average pooling and fully connected layers. The fully connected layer regresses the parameters of the L channel scaling curve, as shown in 74. The scaling curve 74 scales the pixel values ​​in the L channel to form an adjusted image. Finally, the Lab image is converted back to RGB using a differentiable Lab to RGB conversion, as shown in 75.

[0059] Next, the H×W×3 RGB image from the previous step is concatenated with the H×W×61 feature map and fed into the second feature extraction block 76. This block regresses the knots of the RGB scaling curve 77. This curve 77 is applied to the RGB image to adjust the colors in the image.

[0060] Next, the H×W×3 RGB image from the previous step is converted to HSV (hue, saturation, value) space via a differentiable RGB to HSV conversion, as shown in 78. The HSV space separates the hue, saturation, and value (similar to brightness) attributes of an image and is particularly powerful in allowing independent control of hue, saturation, and value. The HSV image is concatenated with the H×W×61 feature map tensor and used as input to a feature extraction block 79, which regresses known points on a saturation scaling curve 80. This curve 80 is applied to the HSV image to adjust the saturation based on the chroma, and the HSV image is converted back to RGB space via a differentiable HSV to RGB conversion, as shown in Figure 81, and the image is output at 82.

[0061] exist Figure 7 In the example of , the transformations learned by the model in the high-level block are curve layers, each of which consists of a piecewise linear scaling curve defined by a set of "knots" that are output for curves 74, 77, 80. A knot is a connection point between each linear segment of the curve. Figure 7 In the above, the W×H×C feature set is passed through feature extraction blocks 73, 76, and 79 to a set of 3×3 convolutional layers with a stride of 2, each followed by a maximum pooling, a global average pooling, and a fully connected layer. The output of the fully connected layer of each feature extraction block 73, 76, and 79 is the node of the curve. Adding nodes increases the modeling flexibility of the curve and is conducive to modeling complex relationships between input and output.

[0062] Thus, given any input image, the model learns to construct a piecewise linear scaling curve unique to that input image that globally adjusts the desired image attribute, such as hue, saturation, brightness, or color. One curve is learned per image attribute. In each color space, the learned piecewise linear scaling curve accurately scales the amplitude of image pixels in a way that is optimized for reproducing the color, brightness, and saturation of the target ground truth image.

[0063] Figure 8 An example of a scaling curve for adjusting the tone of an image is shown. The linear segments 85 of the scaling curve are connected by nodes, as shown at 86. The convolutional and fully connected layers of the high-level block predict the nodes 86 of the scaling curve. The curve adjusts the image by scaling pixels using the formula:

[0064]

[0065] Among them, k0 is the starting point, k i are the parameters (knots) predicted by the deep network (output of the FC layer), clip(.) is the function that constrains the data to a certain range, L is the number of knots, x is the input value (e.g. brightness, hue), and S(x) is the scaling factor to be applied.

[0066] In another example, the high-level block can learn a scaling curve that has the effect of adjusting the global brightness of the image. The curve is as follows Fig. 9 , with linear segments 91 and knots 92. The x-axis of the curve is brightness, and the y-axis is the scaling factor applied to the pixel to adjust the brightness. This curve boosts low brightness pixels by a factor of 50, while high brightness pixels remain unchanged. The adjustment is performed in Lab space, adjusting only the L channel.

[0067] In yet another example, the feature extraction block can learn different types of transformations. For example, the image processing method can also adjust global image properties by predicting a surface or by predicting parameters of a parametric formula. Thus, non-limiting examples of transformations learned by the network are scaling curves, which can be piecewise linear curves, surfaces, or parameters of equations. For example, the formula can be used to adjust contrast, white balance, or perform gamma correction in the color space of the image.

[0068] Experiments have shown that adjustments in all three spaces (RGB, Lab, and HSV) are particularly beneficial for achieving high image quality compared to simply adjusting the colors in RGB space. However, adjustments can be performed in any combination of color spaces. Individual color spaces can also be applied repeatedly. For example, four transformations can be learned: one in RGB space, one in Lab space, and two in HSV space. Transformations for adjusting different attributes can also be learned and performed in the same color space by different feature extraction blocks. In one example, one curve can be learned in HSV space for saturation based on hue, and another curve can be learned in HSV space for saturation based on hue. Figure 7 The example of describes the use of three scaling curves: one curve is learned in each of the three color spaces. However, more than four curves may also be used, for example for tone mapping based on hue, or saturation mapping based on saturation.

[0069] In other embodiments, the application of the curve may follow Figure 7 For example, you can adjust brightness first, then color, and then saturation.

[0070] In the above steps, the convolutional layers of the feature extraction block can share parameters to reduce the number of parameters of the model, and can also be independent for each transformation. This is a form of multi-task learning (called hard parameter sharing).

[0071] In all cases, the fully connected layers are independent for each transformation. If the transformation is a scaling curve, then these curves are responsible for regressing the nodes of each curve. Compared to typical deep learning based models, the curvilinear layers provide human interpretable insights into how the network learns the mapping, as the curves can be easily read and understood by humans.

[0072] Fig.10 A method for transforming a raw image captured by a camera in an image processor is summarized, the processor being used to implement a multi-part artificial intelligence model. In step 1001, the method includes receiving an input image. In a next step 1002, the method includes: implementing a first part of the model to determine a first transformation of the image in a first color space. In step 1003, the method also includes: applying the first transformation to the image to form a first adjusted image. In step 1004, implementing a second part of the model to determine a second transformation of the image in a second color space. In step 1005, the method also includes: applying the second transformation to the first adjusted image to form a second adjusted image. In step 1006, outputting an image derived from the second adjusted image.

[0073] As previously referenced Figure 2 As described, the low-level blocks and the high-level blocks are interconnected to form a complete model, thereby facilitating joint learning of local and global image operations using a training pair dataset. The high-level block can be used to receive a combined feature map formed by the low-level blocks and regard a portion of the feature map as an input image.

[0074] During training, the model jointly learns the convolution filters and the fully connected layers of the low-level blocks and high-level blocks in an end-to-end manner. This is done using training pairs, each consisting of an original image and a corresponding RGB image. The training pairs can be generated by capturing images in RAW and JPG formats using a high-resolution camera. Initially, the convolution filters are set to random values. The original input is fed into the network, which regresses the output image. The difference between the regressed output image and the RGB image forms an error, which is then back-propagated through the network from the output to the input via the gradient. The weights of the network are then updated to reduce the error. The training process iterates using a large set of images until the network weight error decreases.

[0075] Once the network is trained, it can be applied to the original image to generate an 8-bit dynamic range image with correct brightness, contrast, color, and exposure. Correct brightness, contrast, color, and exposure are important because bright areas should not be overexposed and dark areas should not be underexposed. Fig.11 (a) provides an example of an input raw image. Due to the large dynamic range content, the image is visible in dark colors. Fig.11 (b) shows the output of the method described in this paper. The image brightness is corrected, and the color and exposure are greatly improved. Therefore, the system is used to implement an end-to-end trained artificial intelligence model based on deep learning to transform raw data into high-quality images.

[0076] Fig.12 An example of a camera for processing images captured by an image sensor in the camera using the methods described herein is shown. The camera 1 typically includes some onboard processing capabilities. The onboard processing capabilities may be provided by a processor 4. The processor 4 may also be used for basic functions of the device.

[0077] The transceiver 5 is capable of communicating with other entities 10, 11 via a network. These entities may be physically remote from the camera 1. The network may be a publicly accessible network, such as the Internet. The entities 10, 11 may be based on a cloud environment. Entity 10 is a computing entity. Entity 11 is a command and control entity. These entities are logical entities. In practice, these entities may be provided by one or more physical devices, such as a server and a data store, respectively, and the functionality of more than two entities may be provided by a single physical device. Each physical device that implements an entity includes a processor and a memory. These devices may also include a transceiver for sending and receiving data between the transceivers 5 of the camera 1. The memory stores code in a non-transient manner, which code may be executed by the processor to implement the corresponding entity using the manner described herein.

[0078] The command and control entity 11 may train the artificial intelligence model used in the ISP. Such training is typically a computationally intensive task, even if the resulting model can be efficiently described, so it may be efficient to develop the algorithm to be executed in the cloud, where it is expected that large amounts of energy and computing resources are available. It may be expected that this is more efficient than forming such a model on a typical camera.

[0079] In one implementation, once the deep learning algorithm is developed in the cloud, the command and control entity can automatically form a corresponding model and transmit the model to the relevant camera device. In this example, the ISP is implemented by the processor 4 at the camera 1.

[0080] In another possible implementation, the image may be captured by the camera sensor 2, and the image data may be sent to the cloud by the transceiver 5 for processing. Then, the obtained target image may be sent back to the camera 1, such as Fig.12 As shown in 12.

[0081] Thus, these methods can be deployed in a variety of ways, such as in the cloud, on the device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new algorithms or modify existing algorithms. Depending on the computing power close to the data corpus, training can be performed near the source data or in the cloud using an inference engine or the like. These methods can also be implemented at the camera, in a dedicated hardware, or in the cloud.

[0082] Therefore, the traditional ISP pipeline is replaced by a single end-to-end trainable neural network that can learn an ISP mapping from raw data to output high-quality images based on a representative training dataset consisting of input raw data and output digital image pairs. The single neural network learns the mapping directly from the data and is able to jointly optimize demosaicing, denoising, and color correction in a single neural network architecture, and allows the synergistic effects between these operations to be captured and exploited. The trained deep neural network receives raw sensor data as its input and maps that data to visually pleasing output digital images for end-user consumption, which has been shown to significantly improve output image quality in some implementations.

[0083] Replacing the traditional ISP pipeline with a deep neural network brings the following further technical advantages.

[0084] Compared to the multiple stages (eg, 20 stages) of conventional ISP, the present invention involves one processing stage.

[0085] The low-level blocks are designed to more efficiently capture low-level, mid-level, and global image statistics and ensure consistency when performing local pixel adjustments with crosstalk between these image scales. Due to the use of downsampling operations, they are also more suitable for processing full megapixel images on a single GPU without having to break the image into smaller fragments (avoiding fragment-based reasoning).

[0086] The high-level block proposes a neural curve layer that globally adjusts key image attributes such as saturation, brightness, and color in a human-interpretable way. The neural curve layer maps input pixel values ​​to output pixel values ​​using a piecewise linear scaling curve learned for each individual image. Humans can visually inspect the learning curve for a specific image to understand precisely how the network adjusts the image attributes.

[0087] Using piecewise linear curves provides additional modeling flexibility. Deep ISP uses polynomial surfaces to model RGB pixel transformations, which are less flexible than piecewise linear curves with enough knots. In addition, compared to other techniques (e.g., Exposure, HDRNet), piecewise linear curves are not restricted to be monotonic and are able to model quantities that naturally wrap values ​​(e.g., hue).

[0088] The curve can also be learned in three different image processing color spaces (Lab, RGB, HSV) instead of just one, gaining the benefits these color spaces offer in allowing image properties to be adjusted independently of other properties (e.g., changing brightness without affecting color).

[0089] The invention also achieves better handling of variation. Digital photos vary greatly, for example, they can be taken in different conditions such as indoors / outdoors, day / night, zoomed in / out, etc., and the content in the photos may vary greatly. It is difficult for traditional algorithms to maintain consistent operation across these sources of variation. However, by learning from a large number of examples, deep learning methods can adaptively generate high-quality results based on the image content.

[0090] Deep learning of neural networks has been shown to outperform traditional methods in most image enhancement tasks. The model is able to handle complex image transformations better than existing techniques. In addition, there are limited parameters to adjust, and in some cases no parameters need to be adjusted. Deep learning models have a large number of parameters, however, these parameters are learned automatically during training. Components in traditional ISPs expose parameters that must be adjusted manually or algorithmically. Parameters adjusted in the early stages of ISP affect all later stages, making the adjustment of traditional ISPs a complex task.

[0091] There are also advantages in terms of the hardware required to implement the invention. Traditional ISPs are implemented on dedicated application-specific integrated circuits (ASICs). These chips are expensive to develop and require a dedicated chip to be included in the device. AISP can be implemented on a general-purpose neural processing unit (NPU), saving the expense and complexity of developing a custom ISP chip to include in the device.

[0092] The approach described in this paper improves upon known approaches by providing a very different neural network architecture for the end-to-end ISP task, with advantages in terms of image quality and computational memory requirements.

[0093] This method works for both still photos and videos.

[0094] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. In this sense, in view of the common sense of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, without considering whether such features or combinations of features can solve any problem disclosed herein, and without limiting the scope of the claims. This application shows that various aspects of the present invention can be composed of any such individual features or combinations of features. In view of the above description, it will be obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. An image processing module for implementing an artificial intelligence model trained in multiple parts, characterized in that: The image processing module is used for: receiving an input image; implementing a first portion of the model to determine a first transformation of the image in a first color space; applying the first transform to the image to form a first adjusted image; implementing a second portion of the model to determine a second transform of the image in a second color space; each of the first transform and the second transform being a piecewise linear scaling curve, each of the first transform and the second transform comprising knots determined by the first portion and the second portion of the model, respectively; applying the second transform to the first adjusted image to form a second adjusted image; An image derived from the second adjusted image is output.

2. The image processing module according to claim 1, characterized in that: The module is also used to: receiving the second adjusted image; implementing a third portion of the model to determine a third transformation of the image in a third color space; applying the third transform to the image to form a third adjusted image; An image derived from the third adjusted image is output.

3. The image processing module according to claim 2, characterized in that: Each of the first, second and third color spaces is a color space in the following group: RGB space, HSV space and Lab space.

4. The image processing module according to any one of claims 1 to 3, characterized in that: The module is used to apply corresponding transformations to images corresponding to these transformations to adjust properties of the corresponding images in the corresponding color space.

5. The image processing module according to claim 4, characterized in that: The first transform and the second transform relate properties of the image in the respective color spaces to scaling factors.

6. The image processing module according to claim 4 or 5, characterized in that: The attribute of the image is color, brightness, saturation or hue.

7. The image processing module according to any one of claims 4 to 6, characterized in that: The module is used to adjust the properties of the corresponding image at a global scale of the image.

8. An image processing system, comprising an image processing module according to any one of claims 1 to 7 and a second module for implementing an artificial intelligence model, characterized in that: The image processing module according to any one of claims 1 to 7 is used to receive the combined feature map formed by the second module and use a part of the feature map as an input image, wherein the second module is used to: Receiving an input feature map of an original image, wherein the input feature map includes data on multiple channels for each of a plurality of locations in the image; Extracting features from the input feature map at a scale that is less than a global scale of the input feature map to form an intermediate level feature map; Extracting features from the input feature map at a global scale of the input feature map to form a global feature map; The intermediate-level feature map and the global feature map are combined with the input feature map to form a combined feature map.

9. The image processing system according to claim 8, characterized in that: The second module is used to extract features from the input feature map to form the intermediate level feature map and the global feature map using a corresponding convolutional neural network.

10. The image processing system according to claim 8 or 9, characterized in that: The second module is also used to extract features from the input feature map using maximum pooling at a global scale of the input feature map.

11. The image processing system according to any one of claims 8 to 10, characterized in that: The second module is also used to fuse the combined feature map through 1×1 convolution.

12. The image processing system according to any one of claims 8 to 11, characterized in that: The system is used to perform one or more of the following operations on the original image: demosaicing, denoising, local brightness correction, local color correction, global brightness correction or global color correction.

13. A method for transforming a raw image captured by a camera in an image processor, the processor being used to implement a multi-part artificial intelligence model, characterized in that The method comprises: receiving an input image; implementing a first portion of the model to determine a first scaling curve for the image in a first color space; applying the first scaling curve to the image to form a first adjusted image; implementing a second portion of the model to determine a second scaling curve for the image in a second color space; each of the first scaling curve and the second scaling curve comprising nodes determined by the first portion and the second portion of the model, respectively; applying the second scaling curve to the first adjusted image to form a second adjusted image; An image derived from the second adjusted image is output.

Images