Image signal processing

Abstract

And image signal processing. The present disclosure provides a differentiatable module of a differentiatable model of an image signal processor, the image signal processor comprising a pipeline of functional blocks, where the differentiatable module is configured to implement a single functional block of the pipeline, the differentiatable module comprising: a base logic, a module configured to receive an input image signal and process the received input image signal by executing a base image processing function representing a task of a functional block of a pipeline implemented by the module; a refinement function configured to receive the input image signal and process the received input image signal in parallel with the base logic processing the received input image signal; and combinatorial logic configured to combine the processed image signal from the base logic and the processed image signal from the refinement function to determine an output image signal to be output from the differentiatable module. The invention further provides a corresponding method.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to UK patent application GB2310032.4 filed on June 30, 2023, UK patent application GB2310035.7 filed on June 30, 2023, and UK patent application GB2310033.2 filed on June 30, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to image signal processing, for example, on an inference device such as a neural network accelerator. Background Art

[0004] Image signal processing, which may be performed by an image signal processor (hereinafter referred to as an ISP), is the process of modifying raw image data representing an image captured by a camera. An ISP may be specifically used to produce images for consumption by human users. Alternatively, an ISP may be specifically used to process images for consumption by computer vision algorithms, or it may output multiple images for use in various computer vision algorithms and / or human observation. The processing of raw images typically involves performing many tasks, such as denoising, demosaicing, sharpening, and color correction of the image, although this list is not exhaustive. An ISP (or a specific image processing function performed by an ISP) may be considered in terms of reversing the image capture process performed by a camera (or reversing corresponding specific aspects of the image capture process). In some cases, an ISP may not completely reverse the image capture process. For example, when used for computer vision algorithm input, color correction is typically not required, so this image processing function may be omitted from the ISP pipeline.

[0005] ISP is most commonly implemented as dedicated silicon hardware using a streaming (usually linestore-based) architecture that includes a pipeline of well-defined, manually engineered fixed-function modules. Each module can be configured in hardware to perform a specific image processing function, such as denoising, demosaicing, sharpening, color correction, etc. These image processing functions can be considered as tasks performed by the ISP. However, the engineering ISP that implements the pipeline of well-defined, manually engineered fixed-function modules as modules of the ISP generates two significant hidden costs. The first cost comes from the process of adjusting the parameters that control the behavior of each module. This adjustment is a laborious and time-consuming process that is usually performed manually or semi-manually. The second cost is the process of engineering modules implemented in fixed-function hardware, which makes the cost of development, debugging and changes high because (i) the complexity of the hardware implementation must be considered during algorithm design; (ii) the algorithm needs to be adjusted during development to assess quality; (iii) the cost of (re)implementation in, for example, a hardware description language; and (iv) the need to (re)verify the hardware design.

[0006] WO 2022193514 A1 discloses an end-to-end differentiable machine vision system, a training method, and a processor-readable medium. A differentiable image signal processor (ISP) can be trained using machine learning techniques to adapt raw images received from a new sensor to adapted images of the same type (i.e., in the same visual domain) as images previously used to train a perception module, without fine-tuning the perception module itself. Summary of the invention

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0008] According to a first aspect of the present disclosure, a differentiable module of a differentiable model of an image signal processor is provided, the image signal processor comprising a pipeline of functional blocks, wherein the differentiable module is configured to implement a single functional block of the pipeline, the differentiable module comprising: a basic logic, the basic logic being configured to receive an input image signal and process the received input image signal by executing a basic image processing function, the basic image processing function representing the task of the functional block of the pipeline implemented by the module; a refinement function, the refinement function being configured to receive the input image signal and process the received input image signal in parallel with the basic logic processing the received input image signal; and a combinational logic, the combinational logic being configured to combine a processed image signal from the basic logic and a processed image signal from the refinement function to determine an output image signal to be output from the differentiable module.

[0009] The differentiable module described above, wherein the differentiable module can be represented by a command stream as a combination of operations from a set of basic neural network operations available on an inference device for implementation on the inference device.

[0010] The differentiable module described above, wherein the set of basic neural network operations may consist of one or more of the following: a convolution operation; a pooling operation; an element-wise operation; an activation operation; a local response normalization operation; a tensor rescaling operation; a channel permutation operation; and a reshaping operation.

[0011] The differentiable module described above, wherein the inference device may be a neural network accelerator.

[0012] The differentiable module described above, wherein the logic of the differentiable module may be configured to perform a basic image processing function may be a function that, when applied to an image signal, refines the image signal in one aspect.

[0013] The differentiable module described above, wherein the refinement function may be configured to supplement the refinement performed by the base image processing function.

[0014] The differentiable module described above, wherein in supplementing the refinement performed by the base image processing function, the refinement function may be configured to process the input image signal in such a manner that the combinational logic corrects errors remaining in the processed image signal after the base logic has processed the received input image signal.

[0015] The differentiable module described above, wherein the refinement of the image signal that can be performed by the refinement function can be small compared to the refinement of the image signal performed by the base logic.

[0016] The differentiable module described above, wherein the differentiable module can be any one of a demosaicing module, a sharpener module, a black level subtraction module, a spatial denoiser module, a global tone mapping module, a channel gain module, an automatic white balance or a color correction module.

[0017] Another aspect provided by the present disclosure may be an inference device for image processing, the inference device being configured to implement a command stream that may represent a differentiable model of an image signal processor, the differentiable model including the differentiable module described above.

[0018] According to another aspect of the present disclosure, a method for processing an image signal using an inference device is provided, wherein the inference device is configured to implement a command stream representing a differentiable model of an image signal processor, the image signal processor having a pipeline of functional blocks, wherein the model of the image signal processor includes a differentiable module configured to implement a single functional block of the pipeline, the method comprising: receiving an input image signal at the differentiable module; processing the received input image signal by executing a basic image processing function, the basic image processing function representing a task of the functional block of the pipeline implemented by the module; processing the received input image signal using refinement in parallel with processing the received input image signal by executing the basic image processing function; combining the processed image signal from executing the basic image processing function and the processed image signal from using the refinement function to determine an output image signal; and outputting the determined output image signal from the differentiable module.

[0019] The method described above, wherein the command stream can represent the differentiable model of the image signal processor as a combination of operations from a set of elementary neural network operations available on the inference device.

[0020] The method described above, wherein the set of basic neural network operations can be composed of one or more of the following: convolution operation; pooling operation; element-wise operation; activation operation; local response normalization operation; tensor rescaling operation; channel permutation operation; reshaping operation; cascade; reduction operation, including summation, averaging, minimization and maximization.

[0021] In the method described above, the inference device may be a neural network accelerator.

[0022] The method described above, wherein processing the received input image signal by executing the basic image processing function can in one aspect refine the received input image signal.

[0023] The method described above, wherein said processing of the received input image signal using said refinement function may supplement the refinement performed by said base image processing function.

[0024] The method described above, wherein said processing of the received input image signal using said refinement function may supplement refinement performed by said base image processing function such that said combination corrects errors in the processed image signal from performing said base image processing function.

[0025] The method described above, wherein the thinning of the image signal performed by processing the received input image signal using the thinning function may be small compared to the thinning of the image signal performed by processing the received input image signal by performing the base image processing function.

[0026] In the method described above, the tasks of the functional blocks of the pipeline implemented by the module can be any of demosaicing, sharpening, black level subtraction, spatial denoising, global tone mapping, channel gain application, automatic white balance or color correction.

[0027] The method described above, wherein the inference device may be embodied in hardware on an integrated circuit.

[0028] A computer readable code may be provided, the computer readable code being configured to cause the method described above to be performed when the code is executed.

[0029] An integrated circuit definition data set may be provided which, when processed in an integrated circuit manufacturing system, may configure the integrated circuit manufacturing system to manufacture an inference apparatus as described above.

[0030] A training device for training a differentiable model of an image signal processor can be provided, wherein the image signal processor has a pipeline of separate image signal processing functions, wherein the differentiable model of the image signal processor includes at least two differentiable modules, each of the differentiable modules of the differentiable model of the image signal processor is configured to execute a corresponding single image signal processing function of the pipeline, and the training device includes one or more processors, wherein the one or more processors are configured to: receive a reference image; and train a first differentiable module of the differentiable model of the image signal processor to execute a first image signal processing function by iteratively performing the following operations, while not training other differentiable modules of the differentiable model of the image signal processor A differentiable module: inputting a degraded image signal representing known degradation of the reference image into the differentiable model of the image signal processor, wherein the degradation is related to the first image signal processing function; processing the degraded image signal using the differentiable model of the image signal processor to generate a first processed image, wherein the processing includes executing the first image signal processing function using the first differentiable module; calculating an error between the first processed image and the reference image by comparing the first processed image with the reference image; and updating the first image processing function executed by the first differentiable module based on the calculated error without updating image processing functions executed by other differentiable modules of the differentiable model of the image signal processor.

[0031] A training method for training a differentiable model of an image signal processor can be provided, wherein the image signal processor has a pipeline of separate image signal processing functions, wherein the differentiable model of the image signal processor includes at least two differentiable modules, each of the differentiable modules of the differentiable model of the image signal processor is configured to perform a corresponding single image signal processing function of the pipeline, and the training method includes: receiving a reference image; and training a first differentiable module of the differentiable model of the image signal processor to perform a first image signal processing function by iteratively performing the following operations, while not training other differentiable modules of the differentiable model of the image signal processor: providing the image signal processor with a reference image; and A differentiable model of the image signal processor is inputted with a degraded image signal representing known degradation of the reference image, wherein the degradation is related to the first image signal processing function; the degraded image signal is processed using the differentiable model of the image signal processor to produce a first processed image, wherein the processing includes using the first differentiable module to perform the first image signal processing function; calculating an error between the first processed image and the reference image by comparing the first processed image with the reference image; and updating the first image processing function performed by the first differentiable module based on the calculated error without updating image processing functions performed by other differentiable modules of the differentiable model of the image signal processor.

[0032] An inference device may be provided, the inference device being configured to implement a command stream that represents a differentiable model of an image signal processor as a combination of operations from a set of elementary neural network operations available on the inference device, the image signal processor having a pipeline of two or more functional blocks, the differentiable model of the image signal processor comprising: at least two differentiable modules, each of the at least two differentiable modules being configured to implement a corresponding single functional block of the pipeline.

[0033] A method for processing an image signal using an inference device may also be provided, the inference device being configured to implement a command stream, the command stream representing a differentiable model of an image signal processor as a combination of operations from a set of basic neural network operations available on the inference device, the differentiable model of the image signal processor having a pipeline of two or more functional blocks, the method comprising: processing the image signal using the inference device in a manner corresponding to processing by a first differentiable module of the model of the image signal processor to produce a first processed image signal, wherein the first differentiable module of the model of the image signal processor is configured to implement a first functional block of the pipeline; and processing the first processed image signal using the inference device in a manner representing processing by a second differentiable module of the model of the image signal processor to produce a second processed image signal, wherein the second differentiable module of the model of the image signal processor is configured to implement a second functional block of the pipeline, wherein each of the first differentiable module and the second differentiable module of the model of the image signal processor is configured to implement a corresponding single functional block of the pipeline.

[0034] The inference apparatus, neural network accelerator (NNA), or training device may be included in hardware on an integrated circuit. A method of manufacturing an inference apparatus, NNA, or training device at an integrated circuit manufacturing system may be provided. An integrated circuit definition data set may be provided, which, when processed in the integrated circuit manufacturing system, configures the system to manufacture the inference apparatus, NNA, or training device. A non-transitory computer-readable storage medium may be provided, having stored thereon a computer-readable description of the inference apparatus, NNA, or training device, which, when processed in the integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit containing the inference apparatus, NNA, or training device.

[0035] An integrated circuit manufacturing system may be provided, the integrated circuit manufacturing system comprising: a non-transitory computer-readable storage medium having stored thereon a computer-readable description of an inference device, an NNA, or a training device; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit including the inference device, the NNA, or the training device; and an integrated circuit generation system configured to manufacture the inference device, the NNA, or the training device based on the circuit layout description.

[0036] Computer program code for performing any of the methods described herein may be provided.A non-transitory computer readable storage medium may be provided on which computer readable instructions are stored which, when executed in a computer system, cause the computer system to perform any of the methods described herein.

[0037] As will be apparent to the skilled person, the above features may be combined as appropriate and may be combined with any aspect of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0039] Figure 1 shows the general image capture and inversion process that can be used to train an ISP;

[0040] Figure 2 An exemplary overview of a general image capture process and an ISP pipeline designed to inverse the image capture process is shown;

[0041] Figure 3 The general training and testing methods of the present disclosure, including camera characterization, are shown;

[0042] Figure 4 The general training method of the present disclosure is shown;

[0043] Figure 5 An exemplary structure of an ISP demosaic module of the present disclosure is shown within a diagram of an ISP demosaic module training process;

[0044] Figure 6 An example of an ISP denoising module of the present disclosure is shown within a diagram of an ISP denoising module training process;

[0045] Figure 7 An example of an ISP sharpening module is shown within a diagram of a sharpening module training process;

[0046] Figure 8 An example of a computational graph of a modular ISP pipeline that can be implemented on the NNA of the present disclosure is shown;

[0047] Fig. 9 Another example of a modular ISP pipeline that can be implemented on the NNA of the present disclosure is shown;

[0048] Fig.10 shows a plot of intensity variance versus pixel intensity measured in the original input image as a means of measuring photon capture noise in the noise characterization of the sensor;

[0049] Fig.11 An inference device, in this case an NNA of the present disclosure, is shown;

[0050] Fig.12 A computer system in which an inference device and / or a training apparatus is implemented is shown;

[0051] Fig.13 An integrated circuit manufacturing system for producing an integrated circuit including an inference device or a training apparatus is shown; and

[0052] Fig.14 A neural network that can be used for a bilateral filter is shown.

[0053] The accompanying drawings show various examples. It will be appreciated by the skilled person that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the accompanying drawings represent an example of boundaries. In some examples, it may be the case that an element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the accompanying drawings to indicate similar features. DETAILED DESCRIPTION

[0054] The following description is presented by way of example to enable one skilled in the art to make and use the invention.The present invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0055] Embodiments will now be described by way of example only.

[0056] In contrast to conventional approaches that implement ISP in dedicated hardware modules using a streaming (usually row-store based) architecture consisting of well-defined fixed-function modules, several attempts have been made to train large undifferentiated convolutional neural networks (CNNs) to achieve the same (or similar) output as handcrafted ISPs. Although undifferentiated CNNs can in principle model a very wide variety of functions and are therefore useful in applications where we know little about the form the solution should take (e.g., image classification), they have significant drawbacks, including: (i) lack of modularity, which means that interpretability and debuggability are lost; (ii) a large number of parameters (usually on the order of hundreds of thousands to millions) requiring large amounts of training data and long training times; and (iii) a lack of specialized functions, resulting in low computational efficiency (i.e., high bandwidth, large amounts of computation, long execution time, high power consumption).

[0057] In addition, since the undifferentiated CNN is trained from a general state to a specific ISP state, the functions performed by a conventional ISP are no different in the CNN implementation, and therefore all of them are trained as one. Therefore, it is not possible to refine or train a specific function (e.g., demosaicing or denoising or sharpening, etc.) without affecting other functions. For example, a single loss function can be used to optimize the entire CNN. This means that it is difficult to provide a CNN that can be easily adapted or to improve different functions individually and independently.

[0058] While conventional CNNs can be considered “modular” in that operations and neurons are organized into defined units forming layers, these units are general and depend on their effects on other units. In other words, when considered in isolation, the layers of a conventional CNN do not have well-defined functions, but can only operate in the context of the rest of the network. When a conventional CNN is trained as a whole to implement an ISP, each unit of the CNN therefore does not represent a different functional block (module) of the ISP. For example, it cannot be said that a given unit is a “demosaicer” or “related to demosaicing” because this functionality is an overall property of the network, not of one or several given units within it. Therefore, the ISP implemented on these CNNs cannot be trained in a modular manner because the architecture of these CNNs requires a holistic training approach, for example, all aspects of the ISP must be trained simultaneously because each unit of the CNN will depend on the function of every other unit in the CNN.

[0059] The above approaches to training large undifferentiated CNNs do not exploit the modular nature of image signal processing or the historical domain understanding that has been gained from ISPs implemented on hand-crafted hardware.

[0060] WO 2022193514 A1 mentioned in the background section above relates to a processor-readable medium for training a machine vision system based on machine learning. WO 2022193514 A1 represents the use of a differentiable image signal processor that can be trained using a generative adversarial network. Although the differential image signal processor of WO 2022193514 A1 has a certain modularity, since the modules are not designed to take into account the target hardware, it is difficult to efficiently implement the modules on some types of hardware, or it is difficult to ensure that the modules remain functionally different, for example, by training individually rather than together. Therefore, such modules are not optimized for implementing and performing efficient image signal processing on specific target hardware. In addition, the block-by-block differentiable ISP of WO 2022193514A1 includes an enhancement network, which includes a convolutional neural network to generate an enhanced output image by filtering non-Gaussian noise. This type of enhancement network is undifferentiated and is not customized to perform a specific task; instead, it can perform various functions by learning aspects such as denoising, defective pixel correction, or robust color correction. This enhanced network of WO 2022193514 A1 can be connected to the ISP in series with the module, and the enhanced network lacks modularity, which means that the disadvantages discussed above with reference to the undifferentiated CNN also apply to the enhanced network, namely: (i) loss of interpretability and debuggability, (ii) a large number of parameters requiring a large amount of training data and long training time, and (iii) lack of dedicated functions, resulting in low computational efficiency. In addition, the unpaired training method of WO 2022193514 A1 introduces considerable complexity and possible training instability, which is avoided in the present disclosure by using a capture model to maintain point-to-point correspondence between the input image and the reference image during training.

[0061] The examples described in detail below provide solutions to the above shortcomings and provide a customizable and adaptable differentiable model ISP that can be implemented on an inference device. The model ISP is modular and can be trained in a less time-consuming manner than a conventional ISP pipeline that adjusts hardware modules or an ISP pipeline implemented on a large undifferentiated CNN. The model ISP is adaptable in the sense that a module can be removed and replaced with another module without substantially retraining other modules. For example, a sharpening module can be replaced with an alternative sharpening module that implements sharpening in a different manner without substantially retraining the denoiser or demosaicing module. Throughout this disclosure, an inference device may be referred to as a neural network accelerator (NNA) by way of example only. In addition, in the examples described below, the model ISP implemented on the NNA is more efficient than implementing a large undifferentiated CNN in terms of the number of calculations performed, power consumption, and execution time.

[0062] In the example described in this article, a model ISP formed by different modules is represented, each module is configured to implement a functional block of the ISP pipeline, and the functional block performs a predetermined task of the ISP pipeline, such as denoising (denoising module). The task of each module is different from the tasks of other modules in the model ISP. Each module may include a series of layers or steps that can be executed to perform the task, but these steps are limited to the module in question and do not affect other modules. In other words, in the model ISP, there is a one-to-one module-task relationship. Therefore, each module of the model ISP will correspond to a specific single task performed by the ISP pipeline. As in the present disclosure, this modularity and relationship between modules and tasks of the ISP pipeline allows for improved trainability, because each specific module of the model ISP and therefore each task can be trained independently, as will be described later. It also improves interpretability and debuggability, and allows any modifications to be limited to the module corresponding to a given task.

[0063] Figure 1 An overview of the general training setup is presented, which is applicable both to training individual modules and to fine-tuning the ISP model as a whole (once each of its modules has been trained independently), while Figure 5 , Figure 6 and Figure 7 Demonstrates the training of a single module. Figure 1 In ISP, the reconstruction process can be performed by a model represented as a neural network in a deep learning framework (e.g., software that can be used to define and train neural networks, such as TensorFlow or PyTorch). To train the reconstruction process, it can be executed, for example, on a CPU or GPU. Figure 1 . The image capture model can be modular and designed and configured (e.g., using measured characteristics of the physical image capture device) to simulate a physical image capture device (e.g., a camera). The reconstruction process aims to minimize an error 17, which can be calculated between a reference image 11 and an image 15 output from the model ISP, as will be described in more detail below. Minimization of this error is achieved when the processed image 15 matches the reference image 11 as closely as possible, i.e., the degradation introduced by the capture model 12 is reversed as closely as possible by the reconstruction process 14. This approach may be referred to as an "autoencoder approach" by analogy to the autoencoder architecture known in the field of deep learning. Figure 1It is shown how the image capture model and the reconstruction process can be combined to produce a processed image from a reference image (which can be an image of a scene) for training purposes. An RGB (reference) image 11 can be passed through a capture model 12. The capture model 12 can be composed of multiple components, for example, there can be a part of the capture model 12 that imitates / models / simulates the lens. In other words, a part of the model can imitate the effect of a part of the image capture process (such as a specified lens) on the reference image, and apply this effect (usually degradation) to the reference image when the reference image is passed through this part of the model. The capture model can include one or more different configurable components, which can be regarded as modules, each of which can represent the effect that a real-world camera may apply when capturing an image. By configuring these components, the capture model can adapt to specific camera characteristics and therefore model any desired camera / device capable of capturing images. Therefore, the reference image 11 is processed by the image capture model 12 to generate a corrupted / degraded version of the input (the original input image for the reconstruction process 14), in which information is lost (or equivalently, artifacts are introduced) due to the simulated capture process. This degradation will be an accumulation of modular degradations (artifacts) introduced by the capture model 12. This is the same (or similar) degradation that would occur when a physical camera captured an image due to effects such as lens blur. The reconstruction process 14 can process the damaged / degraded image 13 produced by the capture model 12 to produce an output 15 of the same size and shape as the reference image with the same number of channels (three in the case of RGB images). Typically, the reconstruction process will mirror the capture model and attempt to invert the degradation / artifacts introduced by each module of the capture model 12. Since this reconstruction (inversion) problem is ill-posed (there are many possible reference images 11 that may result in the same degraded image 13), it may be interpreted as an approximate best fit to the given data, so the following description of the reconstruction process is used. Figure 4 The numerical optimization (i.e. training) method described in . Other known image sizes and numbers / types of channels can be used instead of the RGB used as an example here. The ISP can be used as Figure 1 Parameters 16 that are trained as part of the process shown are taken as additional inputs. This reconstruction process (in this case, using the model ISP to reverse the degradation of the image capture process) can be viewed as reducing or minimizing the error between the reconstruction of the captured degraded image and the ideal reference image (which can be viewed as the "reconstruction error"), or more intuitively as correcting aspects of the image signal that have been corrupted by the capture process. The image signal can be viewed as raw sensor data or raw sensor data that has been modified. The image signal represents an image and may include information such as pixel values. Reduction of the reconstruction error is also referred to below as "restoration", "reconstruction", "thinning" or "correction".

[0064] After the reconstruction process 14 (model ISP) has been trained to a desired level (i.e., to minimize the error between the reference image (scene) 11 and the processed image 15), it can be put into use. The reconstruction process 14 (model ISP) can be put into use by deploying it on an inference device such as an NNA. The inference device can be optimized to efficiently perform a predetermined set of operations, and no support for backpropagation or training may be provided (i.e., the inference device and the training device may be different). The inference device can be used to apply the reconstruction process to degraded images produced by the image capture device (e.g., a camera) that the capture model 12 is designed to simulate. In this way, the deployed reconstruction process can (at least partially) reverse the degradation introduced into the newly captured image by the image capture device.

[0065] Reference now Figure 2 , which shows an overview of an exemplary image capture process and an exemplary reconstruction process in the form of an ISP pipeline, in other words, a specific implementation of the reconstruction process (i.e., in use, in Figure 1 ). An example of this is implementing a differentiable model ISP on an inference device (NNA). In this case, the differentiable model ISP can first be converted into a command stream, which can then be executed on the inference device (NNA), which configures the inference device to act as an ISP. The neural network accelerator (NNA) acting as an ISP is then used to reverse at least a portion of the image capture process 220 performed by the camera by processing a degraded (or "raw") image 207 to improve image quality, thereby producing, for example, an output image 213, which would be viewed as high quality, for example, to a human observer. The NNA is configured to reverse the image capture process 220 will correspond to the image capture model ( Figure 1 12) (differentiable model ISP), and thus the image capture model 12 simulates the image capture process 220 of a real camera or other image capture device. In some cases, the inference apparatus (NNA) can be configured to implement a command stream that represents the differentiable model of the image signal processor as a combination of operations from a set of basic neural network operations available on the inference apparatus. It should be noted that the command stream itself is non-differentiable, and once the differentiable model ISP is converted into a command stream, the differentiable property of the model ISP is lost. Therefore, the inference apparatus is configured to implement the model ISP to reconstruct an image that has been "corrupted" during the image capture process 220. Figure 2 An overview of the implemented reconstruction process 230 can be seen in Figure 1 The reconstruction process 14 may include one or more modules of the model ISP, wherein each module of the model ISP may have been trained separately from other modules as described herein.

[0066] Figure 2 An exemplary image capture process and an exemplary image signal processing pipeline (reconstruction process) for correcting the captured image for display are shown. Typically, the image capture process produces a corrupted version of the image. The corrupted / degraded image may contain, for example, noise, incorrect colors, and missing samples (due to the use of a color filter array). Figure 2 As can be seen in , when a scene 200 is captured by an image capture device such as a camera, light (image) 201 typically passes through a lens 202 of the image capture device, which will cause some distortion to the image. This distortion can be in the form of blur and / or other distortion(s), and is a form of corruption of the captured image. The characteristic form of degradation (e.g., blur) caused can be referred to as "artifacts." The distorted image 203 can then pass through one or more color filters 204 or a color filter array, such as a Bayer filter. As a result, other information related to the distorted image is lost, for example, a Bayer filter will remove information related to two of the R (red), G (green), or B (blue) color data for each pixel. The remaining information can then be detected using a sensor 205, and can be passed to an analog-to-digital converter 206 as part of the image capture process. This creates a digital representation of the captured scene. This digital representation can form the underlying raw image data (degraded image) 207, in which each pixel contains only partial information about the captured scene. This raw image data may be considered as an image of the scene that has been degraded (corrupted) during the image capture process.

[0067] like Figure 2 As further seen in , in order to restore the degraded original image 207 to an image that can be displayed to a user, the image may undergo image signal processing (e.g., a reconstruction process). This image signal processing may be performed by an ISP pipeline, each module of which may perform a function of correcting aspects of the degradation of the image caused by the image capture process. An example of this may be that the image signal processing pipeline may include a sharpening module that is intended to reverse the blurring effect caused by the lens during the image capture process. Figure 2 The image signal processing pipeline shown in FIG. 2 includes a denoising module 208, a demosaicing module 209, a color correction module 210, and a sharpening module 211. Figure 2 Each of these modules of the image signal processing pipeline shown in the example of is shown only for the purpose of explanation, because once the image signal processing pipeline is deployed on the inference device, these modules are no longer different from each other, however, the overall functionality of the trained modular model ISP is contained in the command flow representing the image signal processing pipeline (as explained in more detail below). Therefore, Figure 2Each block of the image signal processor represents a separate task of the pipeline and can be considered as a different functional block performed by the pipeline before being converted into a command stream for implementation. In other words, Figure 2 The blocks 208, 209, 210 and 211 of the image signal processor of the present invention represent the modules of the differentiable model ISP for the purpose of explanation only so that the connection between the differentiable model ISP and the implemented ISP can be understood. This image signal processing pipeline can be implemented on the NNA using a model image signal processing pipeline having modules corresponding to the modules of the desired image signal processing pipeline. Prior to implementation, the image signal processor (an example of which is shown as Figure 2 Boxes 208, 209, 210, and 211 of ) can be modeled as a model ISP for training purposes (e.g., Figure 1 The reconstruction process shown in 14).

[0068] Once the raw image has been processed by the ISP pipeline, the image may be displayed 212 to a user via a monitor or similar display device 212 by outputting light 213 representing light 201 from the originally captured scene 200. Arrow 213 represents light from the display device.

[0069] The image capture process 220 can be simulated using the mathematical capture model 12, as previously described with reference to Figure 1 introduced. This enables pixel perfect registration between the reference image and the corrupted image, which is critical for training, and avoids the need to use complex unpaired training methods, such as Generic Adversarial Networks (GANs). This pixel perfect registration would otherwise be difficult to achieve, for example, using a physical camera and a reference chart. Another reason for simulating the image capture process is that parts of the capture model 12 can then be selectively enabled and disabled, which allows modularity to be maintained when training the ISP. Again, this is difficult to achieve using a physical capture process because, for example, the lens and sensor must be used to capture any image captured, so it is impossible to remove parts of the capture process that relate to these images. The simulated image capture process 220 also allows the possibility of creating new data without the need to perform new captures, and also provides the ability to train using "ideal" scenarios, i.e., pixel perfect test patterns are not possible when using physical capture. In other words, each aspect of the image capture process 220 can be mathematically modeled and combined into the model 12 of the image capture pipeline, as is known in the art, so that the above effects can be achieved. An example of this is that the way light propagates through a lens can be described mathematically and implemented, for example in software, so that when an image is input into the mathematical model, the blurring effect that the lens would apply to the image also appears when the image is processed by the model.

[0070] The capture model 12 can be set with parameters that control its behavior, for example, a Gaussian blur used to model a lens can be parameterized by its variance, and a parameterized random distribution can be used to represent the capture noise of a sensor. Therefore, the structure of the capture model 12 can be largely the same for different cameras of different ranges, because it provides a framework into which the measured characteristics of a given camera can be input to configure its behavior to closely correspond to the capture process 220 of that camera. The parameters / characteristics of a physical camera can be measured under controlled conditions in a process that can be referred to as "characterization". Such controlled conditions may include the use of an imaging laboratory with controlled lighting, exposure, gain, test charts, etc. The experimental setup may vary depending on which parameters need to be measured. This process typically requires manual operation and has a time scale of several days, but it should be noted that this process only needs to be performed once for a new sensor and / or lens. The capture model 12 can then be set using this characterization of the properties of the camera so that the model mimics the desired camera. Some examples of parameters of the capture model 12 are noise intensity, amount and type of lens blur, required color space conversion, etc. The capture model 12 described herein is measured in conjunction with the model ISP (reconstruction process) 14 (such as Figure 1 This makes it possible to bypass / selectively disable portions of the capture model 12 and the model ISP 14 for training purposes, so that individual modules of the model ISP 14 can be trained in isolation from other modules of the model ISP.

[0071] In some examples, the parameters of the capture model 12 may be manually varied from measurements from the physical camera in order to achieve certain aesthetic effects. For example, the strength of the lens blur may be artificially increased ("over-blurred") in order to achieve greater sharpening in the ISP, which produces an artificially sharper image when applied to an image captured with the physical camera.

[0072] Figure 1 The reconstruction process (ISP model) can be represented using one or more modules. Figure 2 A specific implementation of ISP 230 is shown, but in order to reverse the capture process 220, ISP 230 is shown as having modules to better explain the implementation of ISP relative to Figure 1 The model of ISP (Reconstruction Process) 14 functions. Figure 1 The model ISP (Reconstruction Process) 14 may include Figure 2 The modules of the ISP model are differentiable to allow training. Figure 2As shown in , this differentiability is lost when the trained model ISP is implemented as a command stream on an inference device. The four exemplary modules correspond to various degradations introduced to the image during the capture process 220. Each module of the model ISP 14 represents a specific task performed by the ISP pipeline and may correspond to an element of the corruption introduced during the capture process 220 / capture model 12 as described above.

[0073] Tasks can be thought of as functional blocks of the ISP pipeline. Each functional block of the ISP pipeline can be performed by a separate module of the model ISP 14, which is then implemented as a command stream on the inference device. Each module of the model ISP 14 can implement only one functional block (task) of the ISP pipeline, thus allowing each module to be independently customized and trained to implement the desired functional block operation. A series of modules of the model ISP 14 will process the image input in a sequential manner, and each module reconstructs an aspect of the input image by performing, for example, denoising, sharpening, demosaicing, etc. In other words, the functional blocks of the ISP pipeline can be embodied by modules: for example, there may be a denoising functional block implemented by processing applied by a denoising module 208. The capture process includes a lens 202 that introduces blur into the image, and the ISP pipeline includes a sharpening module 211 that will reverse the blur introduced during the capture process. Each module of the ISP can be represented by multiple layers or operations of the model ISP. Furthermore, when the Model ISP is implemented on the inference device (by converting to a command stream for execution), each layer or operation of the Model ISP can be mapped to one, a portion of, or multiple hardware modules on the NNA, as described later. Most commonly, the order in which the image processing modules are applied is roughly the reverse order of the corresponding physical capture process that degraded the image 201.

[0074] exist Figure 2 In the example, the model ISP pipeline, when implemented after training, will include functions related to the denoising module 208, the demosaicing module 209, the color correction module 210, and the sharpening module 211. Figure 5 , Figure 6 and Figure 7 Training of some of these modules is described.The raw image data output from the image capture process may be processed by each of these modules in sequence until each module has acted on the raw image data / image signal.

[0075] The model ISP 14 described above may include at least two modules, for example, two or more of the modules described above. Each of the modules described above is functionally different from the other modules and can therefore be trained and / or used independently to process images / image signals. The modules of the model ISP may be arranged in a pipeline, such as Figure 2 and Figure 8As shown. Although modularity and differentiability are important during training, they are not important when deployed, and therefore, a deployed model ISP may not remain modular or differentiable. However, the behavior of the model ISP when processing an input image is retained. Therefore, the image originally input to the inference device is refined by the inference device according to the behavior of the model ISP including at least two modules before being output. In the context of the present disclosure, "refined" may mean processed by one or more modules or inference devices, each module (or inference device) performing a functional block of the ISP pipeline in one aspect to produce an image with improved quality, for example, in the case of the model ISP, the denoising module will process the image to reduce noise in the image. "In one aspect" should be understood to mean the way in which a module (functional block) performing a single task is intended to process an image (different from the way other modules of the ISP can process an image). When implemented on the NNA after training, the modules and the model ISP as a whole should have the required functionality to reverse the capture process of a given physical camera.

[0076] Once the capture model 12 has been created, the Model ISP 14 pipeline can be trained using the capture model to be used to invert the degraded image 13 produced by the capture model 12, and therefore invert the capture process 220 of the physical camera when implemented. This is because the capture model 12, if correctly formed, will mimic the capture process 220 of the physical camera, thereby providing the same degradation to the reference image 11 as the physical camera would provide to the captured image 200. This degradation is what the NNA acting as a Model ISP is intended to reduce / invert in order to produce a processed image that is a true representation of the original scene 200 when applied to the output of a given camera.

[0077] The differentiable model ISP is trained using the process described below to produce processed images that correct the degraded images that may be produced by the captured model.

[0078] As an example of generating a model ISP for configuring an inference device to act as an ISP, a capture model is first generated, which is a general model with parameters that allow the model to represent a specific camera with the characteristics of a built-in camera. Once this capture model has been generated, it can be used to generate a degraded image 13. This can be done by degrading the reference image 11. Then, the reconstruction process 14 (representing the model ISP) will receive the image / image signal degraded by the capture model and perform the functions of the ISP, thereby attempting to reverse the degradation caused by the capture model in the image. It should be understood that the degradation applied to the reference image 11 to produce the degraded image 13 is related to the function (first / second image processing function) performed by one or more modules of the differentiable model ISP that will be trained as part of the reconstruction process 14. This allows the reference image 11 to be compared with the image output when the degraded image 13 is processed by one or more modules being trained. An example of this is that if the denoising module 208 is to be trained, the reference image 11 will be degraded to introduce a known amount of noise into the reference image 11 to produce the degraded image 13. The degraded image 13 with the introduced noise degradation can then be used to train a denoising module 208, which will be configured to reverse the degradation and restore the degraded image 13 as close as possible to the reference image 11. The model ISP is trained (by adapting the parameters 16 of the reconstruction process 14) so ​​that the processed image 15 is as close as possible to the reference image. When the model ISP has been trained, it can be converted into a command stream to be implemented on an inference device (e.g., an NNA). This inference device can then be used to process the degraded image 207 received from the image capture process 220 in order to determine the image.

[0079] Now about Figure 3 A high-level procedure for training a differentiable model ISP to correct for the characteristics of a specific capture model (simulating an image capture device) is described. The first step S10 of the general training process is to characterize the image capture device (camera) whose captured images the user wishes to correct / enhance with the final trained ISP implemented as a command stream on an inference device. The characterization of the camera is described in more detail later in this disclosure, however, in general, the characterization of the camera may include characterizing every aspect of the camera, such as the lens, the sensor that detects light, etc. Such characterization involves defining the properties of the camera in question.

[0080] Once the camera has been characterized and its properties are known, the user then configures the capture model to simulate these properties and therefore mathematically simulate the camera. In this step S11, each aspect of the camera is modeled as part of the capture model. For example, the lens is mathematically modeled and imitated by a part of the capture model so that when an image is processed by this part of the capture model, it degrades in the same way as the image observed through the lens, for example, the artifacts found in the image will be the same. This is true for each part of the camera, so that once the capture model is configured in step S11, it represents a model of the camera, which model, once characterized, will then process the image in the same way as the camera in a controlled manner. Optionally, the capture model can be configured (partially) based on the requirements of the end user of the ISP. In other words, the capture model can be configured to further adapt the input image to suit the needs of the user, for example to perform a greater degree of blur than is normally performed by the camera.

[0081] After the capture model has been generated in step S11, it can be used to degrade the reference image to produce a reference image and a degraded image pair, which can be used to train the differentiable model ISP before implementation on the inference device. Before training each module of the differentiable model ISP, a training image set is obtained, which can be used to iterate through the model ISP in order to refine it. Therefore, step S12 can be performed, in which one image training set or multiple image training sets can be identified for training each module. These training images can be, for example, artificial test images, natural photos, or a mixture of both. The technician will understand that any appropriate image can be used in this step. These images can be regarded as reference images and may have corresponding pixel-perfectly registered degraded images corresponding to them on a pixel-to-pixel basis. However, in the absence of such a pair with perfect correlation, the reference image will be input into the capture model to identify the image training set formed by the input reference image and the paired degraded image that has been processed by the capture model, and thus appropriate artifacts are introduced, for example, with respect to the training of the sharpening module, the artifacts introduced by the capture model processing may be blurring effects.

[0082] Once an image training set or multiple image training sets have been identified, they can be used to train a differentiable model image signal processor in step S13, as described below with respect to Figure 4 As will be described, the differentiable model ISP is trained on a module-by-module basis to start model fine-tuning, and only after this operation is the model fine-tuned as a whole.

[0083] After the differentiable model ISP has been trained using the identified training images in step S13, the differentiable model ISP can be implemented to process images from a physical camera, as in step S14. In this case, the differentiable model ISP can be converted into a command stream that will then be implemented on an inference or other suitable device (e.g., a neural network accelerator), or alternatively can be run on the same device used for training for testing purposes. When implemented on an inference device, the command stream will cause the inference device to reproduce the functionality of an ISP with trained properties, and thus reverse the capture process of the physical camera in the case of the present disclosure. The results from processing the test image can be compared with the reference image to test whether the inference device acting as an ISP meets the user's requirements. The user can then determine whether the trained ISP implemented on the inference device in S15 performs the correct image processing (reversing the capture process of the camera) or whether further training is required. Once such an assessment has been made by the user S15, the user may return to any of steps S10 to S13 to correct deficiencies, if necessary, in order to produce a more accurate recreation of the captured image or to customize the processing performed by the implemented ISP, for example by retraining a given module and / or changing the mathematical functions that define it, or selecting a different set of training images. This process continues until the functionality of the trained ISP meets the user's requirements, at which point a command stream is prepared for configuring the inference device to reproduce the functionality of the ISP, such as Figure 3 As indicated by the “Yes” arrow in the figure.

[0084] In some cases, steps S10 to S12 may be bypassed or omitted from a process in which a suitable set of registered real images (reference images) and degraded images are obtained by other means (e.g., by controlled physical capture under laboratory conditions). However, it is expected that the approach using the captured model is more efficient overall from an engineer's perspective. Therefore, the skilled person will understand that if the reference image and degraded image pair are available, steps S10 to S12 of this process are optional. In this context, "registered" means that there is a direct spatial correspondence between two images (e.g., between a real image and a degraded image pair).

[0085] Now about Figure 4 A training method and a training device for training a differentiable model of an image signal processor are described. The model ISP may include at least two differentiable modules. The training method will be described first, followed by a training device that can implement the method.

[0086] The training method can be used to train a differentiable model of an ISP that may include at least two modules. As described above, each of these modules performs functions independent of the other modules. In addition, each module is separate (i.e., different) from the other modules that form part of the model ISP. Each module of the model ISP is configured to perform a corresponding single image processing function of the image signal processing pipeline, for example, a demosaicing module only performs a demosaicing function without performing a sharpening function. The training method includes receiving a reference image, and may include processing the reference image using a capture model with known properties to produce a degraded image with pixel-perfect registration with the reference image. In other words, pixels of the reference image can be compared with pixels of the degraded image that correspond in a 1-to-1 manner. For example, the capture process / model will have known properties, such as properties related to a specific image capture device that the capture model is imitating, and therefore the degradation applied to the reference image will be known. As previously described, registered image pairs obtained by other means or pre-calculated offline using a capture model may be used instead. For example, in some cases, the received one or more pairs of (registered) reference images and degraded images may have been captured using a controlled physical capture device to correspond to each other in a 1-to-1 manner prior to reception. Alternatively, as discussed above, the method may include the step of degrading the received reference image to produce a degraded image corresponding to the reference image.

[0087] In some examples, the training of the modules can be processed in batches, that is, multiple image pairs (reference and degraded) input to or output from the differentiable model ISP can be processed simultaneously, and the gradients can be accumulated before updating the parameters of each module. In this way, groups (batches) of image pairs can be used in the training method for one or more modules to simultaneously consider the gradients generated by each image pair and improve the speed, convergence and accuracy of the training process.

[0088] The training of the modules of the model ISP is performed on a module-by-module basis. Therefore, while each module is being trained, the modules that are not being trained are fixed, for example, the parameters that form the functions of other modules are fixed and unchanged. This allows only the parameters of the modules being trained to be updated without affecting the behavior of the modules that are not being trained. This ensures that the independence of the modules is maintained by eliminating the possibility that each module (e.g., a demosaicer) attempts to correct errors introduced by unrelated parts of the capture process (e.g., lens blur, which may be intended to be addressed by a different module such as a sharpening module). The ability to maintain modularity during training is an important advantage of using a configurable capture model.

[0089] The training method can differ between modules and can be performed as follows, with respect to Figure 1 and Figure 4 described. Later in this disclosure Figure 5 , Figure 6 and Figure 7 Describing a specific example of the application of the training method for a single module, the figure shows the training of a demosaicing module, a denoising module, and a sharpening module.

[0090] A training method for training a model ISP that can be implemented on an inference device such as an NNA according to the present disclosure is as follows. The model ISP that can be implemented on the inference device may include at least two modules, each of which is configured to perform a corresponding single image signal processing function. The method for training the model ISP may include receiving one or more pairs of registered degraded images and a reference image 11. In some cases, the reference image 11 may be data, possibly in the form of an image or an image signal, which is input to the capture model 12 so that the capture model 12 degrades the reference image 11 in the same manner as the hardware (image capture device) that the capture model is emulating. Such hardware may be a specific camera with image capture properties that the capture model then emulates.

[0091] Preferably, the capture model 12 can be used to degrade the reference image 11 to form a degraded image 13, which can be used to train the model ISP ( Figure 3 Step S13 or Figure 4 Step 112). However, a degraded image and reference image pair may be used which does not include an image degraded by the capture process, but is derived from a controlled capture using a physical camera. Where the reference image is degraded by processing with the capture model, at each stage of the process, damage may be introduced into the image. This results in the production of an image 13 which is degraded compared to the reference image 11. Since the sensor that captured the image will have been characterized, the image is degraded by a known amount and therefore the properties will be known. As discussed above, images with known degradation may be used as long as they have a corresponding reference image and represent the degradation of the reference image 11.

[0092] The first module of the reconstruction process (model ISP) 14 that performs the first image signal processing function is then trained (i.e., optimized with respect to the error 17) using an iterative digital method. While the first module is being trained, any other modules that are part of the model ISP 14 are not being trained, e.g., as discussed above, the parameters (which may include weights) of these modules that are not being trained remain fixed, unchanged, or are removed / bypassed. In this way, only the first module is trained in the following steps. During the training of the first module, the image signal of the degraded image 13 will only be processed by the desired modules that will be trained in the model ISP pipeline 14, and the parameters of the first module will only be adapted based on the results.

[0093] The training of the first module includes the step of inputting a degraded image signal (or a batch of degraded image signals) 13 to the model ISP (step S112). The degraded image signal is a signal representing Figure 1 The capture model 12 shown in FIG. 1 is a known degraded image signal of a reference image signal. Alternatively, the reference image 11 and the degraded image 13 may be used instead of the corresponding image signals. For example, in the case of training a denoising module, the degradation applied to the reference image signal applies noise 201 to the image to degrade it. The characteristics of the noise applied to the reference image are controlled and known to the user.

[0094] Once the degraded image signal 13 has been input to the model ISP 14, the degraded image signal 13 is processed (e.g., "corrected") using the model ISP pipeline in step S113 to produce a first processed image 15. The model ISP processes the degraded image signal using the model ISP pipeline in step S113. In other words, the degraded image signal 13 is passed through each active module of the model ISP in succession to refine the properties of the image. Here, an active module is a module that is being trained, while an inactive module is a module that has been removed or bypassed and can therefore be considered to be absent from the processing pipeline of the model ISP.

[0095] Taking the denoising module as an example, the degraded image will be processed by the model ISP pipeline including the denoising module, and the denoising module can implement the bilateral filter 603 for performing bilateral filtering (see Figure 6 ) as the first image signal processing function. The bilateral filter 603 may include a variance parameter that controls the selectivity of the filter, and this may be trained during the training method. The denoising module of this example will output a first processed image 605, but in other examples it may alternatively output a first processed image signal representing the first processed image 605. The first processed image 605, 15 is an image that has been partially corrected back to the reference image 11 using the modules of the model ISP 14. Specifically, with respect to the training of the denoising module, the first processed image 605 will be an image in which the noise of the degraded image 602 has been at least partially reduced by the denoising module.

[0096] The first processed image 15 is then compared with the reference image 11 in order to calculate (in step S114) an error 17 between the two. Reducing this error 17 corresponds to "correcting" or improving the reconstruction quality of the image, or equivalently reversing the capture process. The error 17 between the first processed image 15 and the reference image can be calculated in any suitable way. Minimizing the error corresponds to providing the best possible correction by updating the parameters and / or weights of the functions of each module that constitutes the model ISP. The purpose of training is to reduce the error between the first processed image and the reference image (i.e., the difference between the processed image and the reference image will be smaller). This is achieved by minimizing the error, which can be as follows.

[0097]

[0098] Where w* is the ideal parameters of the module / model ISP, x is a specific image, X is the image training set, f is the capture model, and g is the parameterized ISP (with parameters (including weights w)). The model ISP is trained to invert the degraded image from the capture mode in the same way as solving the general optimization problem described above by selecting the best parameters w*. The terms within argmin are examples of the error between the reference image and the refined output image that should be minimized. This minimization corresponds to training the module parameters of the model ISP.

[0099] Once the error 17 between the first processed image 15 and the reference image 11 has been calculated, the parameters of the first image processing module being trained are updated in step S115 based on this calculated error 17. This update may involve updating, in other words changing, the parameters of the function performed by the module. In this step S115, only the module being trained may be updated. This can be done by a numerical optimization algorithm, such as gradient descent by back propagation of the error. Gradient descent is a class of numerical optimization algorithms that includes stochastic gradient descent and Adam optimizer. The gradient itself can be conventionally calculated by back propagation. In other words, gradient descent is built on back propagation. In the case where the gradient is calculated using back propagation of the error for training purposes, it is important that the modules in the ISP pipeline should be differentiable. Tools for defining differentiable computational graphs and performing back propagation and gradient descent-based optimization are typically available in deep learning frameworks such as TensorFlow and PyTorch, and can therefore be used to train such ISPs according to the methods described in the present disclosure.

[0100] In the example of a denoising module including a spatial denoiser, the denoising module is implemented with a (range) variance parameter (see Figure 6) as the basic image processing function 603, in order for the standard optimization algorithm (using gradient descent through back propagation) to be effective when trained using the above method, the algorithm of the bilateral filter must be differentiable. In other words, it must be possible to backpropagate the error (with respect to the derivative of the error) to the parameter being trained so that the parameter can be updated. Figure 6 In the case of the denoising module shown including a spatial denoiser, a trainable parameter determines the strength of the filter. A filter that is too strong or too weak will result in undesirable errors in the output. Therefore, it is advantageous to minimize the output error by selecting the optimal denoiser strength that balances the preservation of features (e.g., details and edges) while removing as much noise as possible.

[0101] Since the above training is an iterative process, once the first image processing function of the module being trained has been updated, the training steps are repeated. Figure 4 The iterative step indicated by the arrow S116 of . In other words, the same or different (multiple) degraded images are input to the model ISP and corrected again using the model ISP. However, in this subsequent iteration, the module being trained processes the image according to the updated parameters of the basic image processing function, thereby producing another processed image corrected by a different amount than the first processed image. This can be an improved correction that brings the properties of the other processed image closer to those of the reference image. The other processed image is then compared to the reference image in the same manner as discussed above, and a new error is calculated based on this comparison. This error is then used to update the parameters of the basic image processing function of the module being trained again, and the iterative process can be repeated again until the termination condition has been met (for example, the training has been performed for the required number of iterations, or the error is within an allowable range). In some cases, it may be necessary to update some parameters of the module rather than other parameters, such as parameters related to the basic image processing function or parameters related to the refinement processing function (described below), or alternatively, update part of the parameters of the first processing function and / or the refinement processing function.

[0102] The training method may also include stopping the iterative process when the error 17 between the processed image 15 (first or subsequent) and the reference image 11 is less than a threshold. This threshold may be set by the user, or automatically calculated as a percentage of the property being trained. For example, in the case of a denoising module, the threshold may be that the error 17 between the processed image 15 and the reference image 11 is within a certain predefined tolerance. In some cases, the error used to determine when to stop training may be different from the error used for optimization. For example, this may be done in the case where the metric for the error 17 used to judge image quality is not differentiable, and we are actually optimizing with the aid of a differentiable function.

[0103] Alternatively, instead of stopping the iterative process when a threshold is met, the iterative process can stop after a set number of epochs or a set length of time has passed. In other words, a finite number of iterations can be set, and once these iterations have been performed, the iterative process will stop. In some cases, this allows a training schedule to be set where each module of the model ISP is iteratively trained for a set number of iterations / epochs / time periods before moving on to the next module.

[0104] In some cases, the first module (e.g., the module being trained) may include a basic image processing function as described above, which performs the tasks to be performed by the module. In some cases, a basic logic may be provided, which is configured to receive an input image signal and process the received input image signal by performing a basic image processing function representing the tasks of the functional blocks of the pipeline implemented by the module. The module may also be composed of a refinement function in the form of a small neural network. In this configuration, the basic image processing function may also be composed of a basic image processing function and a refinement function performed by the neural network. When the basic image processing function is updated in the last step of the iterative process discussed above, the parameters 16 of both the basic image processing function and the refinement function are updated simultaneously. In this way, when the module includes a refinement function as a neural network, the neural network within the module can be trained and refined as part of the iterative process. The same is true for the basic image processing function performed by the module. However, this may not be the case, and instead of updating the parameters of both functions simultaneously, only the parameters of one of the basic processing function or the refinement function may be updated. Alternatively, only a portion of one or both of the functions may be updated based on the required refinement (adaptation).

[0105] In some cases, a refinement function may be provided, the refinement function being configured to receive an input image signal and process the received input image signal in parallel with the base logic processing the received input image signal. Combination logic may also be provided, the combination logic being configured to combine the processed image signal from the base logic and the processed image signal from the refinement function to determine an output image signal to be output from the differentiable module. In other words, in some examples, each module may be considered a differentiable module of a differentiable model of an image signal processor, the image signal processor comprising a pipeline of functional blocks, wherein the differentiable module is configured to implement a single functional block of the pipeline, the differentiable module comprising the base logic, the refinement function, and the combination logic discussed above. The refinement function may be configured to process the input image signal in such a manner that the combination logic corrects errors remaining in the processed image signal after the base logic has processed the received input image signal.

[0106] When updating the parameters of the basic image processing function and / or the refinement function, the parameters may be updated based on at least one first preset learning rate. The learning rate may be a scaling factor by which the error gradient with respect to each parameter is multiplied before incrementing each parameter. The learning rate is a super parameter set empirically by the user. Typically, the first preset learning rate may be set to a positive value significantly less than 1 (typically within 10 -3 Up to 10 -6 More generally, the first preset learning rate can be as high as 10, depending on the choice of optimization algorithm. -1 or as low as 10 -9 In other words, in 10 -1 Up to 10 -9 . This value is selected so that when multiplied by the error gradient about a given parameter, a small adjustment to the value of the parameter is produced. A learning rate that is too large or too small will usually prevent convergence in a reasonable time. In this way, small adjustments can be made to the parameters in each iteration. It should be understood that the learning rate may be different for each module and / or part of the module. For example, a denoising module may have a different learning rate from a demosaicing module. In addition, a refinement function within a module may have a different learning rate from a basic image processing function. Depending on which part of the model ISP you want to train, the relevant learning rate may be applied to the iterative process. The user can change the learning rate before or during training. The ability to customize the learning rate for each part of the model ISP allows the incremental iterative process to better converge to the global minimum of the error, and therefore more accurately refine the modules of the model ISP. It should be understood that other optimization super parameters (including learning rates) may be used, and they may be different for different modules, parameters within a module, and / or stages of training.

[0107] When there is a second module to be trained after the first module, the training method may also include additional steps. The second module of the model ISP may be trained to perform a second image signal processing function that is different from the image signal processing function of the first module. In the same manner as the first module, the second module is trained independently of the first module, so when the second module is being trained, the first module is not trained or may not exist at all. The second module is then trained by performing the same iterative steps as the first module and comparing the reference image 11 with the processed image 15 as discussed above. A second degraded image or a batch of degraded images representing a second known degradation of one or more reference images is input to the model ISP. The second degraded image may be degraded by the capture model in a different manner from the first degraded image because it will be used to train different modules. For example, if the first degraded image was used to train the denoising module, noise will have been added, while the second degraded image may be used to train the sharpening module, so a blurring effect may have been added. The same reference image may be used for degradation, or different reference images may be used, for example, if by doing so, degradation can be more easily achieved or measured.

[0108] Once the second degraded image has been input, it can be processed (e.g., "corrected", "thinned") using the second module to perform a second image signal processing function. For example, where the second module is a sharpening module, the second image signal processing function can be a sharpening function or a deblurring function. When executed on the second degraded image, this second image signal processing function produces a second processed image. This second processed image is similar to the first processed image, but differs in that the properties of the second module (e.g., sharpness) have been corrected.

[0109] The second processed image is then compared to the reference image and an error is calculated between the two. In the example of a sharpening module, the error may be the amount of blur remaining in the second processed image. The calculated error may then be used to update the second image processing function in a manner similar to that described above for updating the first image processing function.

[0110] Training of the second module can occur before or after training of the first module, but not at the same time (unless multiple GPUs are used, each GPU training a separate module). In this way, each module can be refined individually without changing the other module and without incorrectly compensating for effects caused by other modules. For example, when training the sharpening module, the trained demosaicing module (if present) can be enabled, but the parameters are not updated during the update step. In addition, when training the sharpening module, the degraded image produced by the capture model can include lens blur simulation and color filter array simulation. The same is true when training the denoising module, both the noise simulation and the color filter array can be enabled in the capture model, and both the demosaicing module and the denoising module can be activated in the model ISP. Alternatively, the second module can be trained in isolation from the other modules, so that the model ISP only includes the second module to be trained. Then, once the second module is trained, it can be combined with the first module to form a model ISP that includes both the first module and the second module.

[0111] Additionally, and optionally, in order to train each module of the model ISP individually and separately, the model ISP may be trained as a whole after each module has been trained individually. This allows for further refinement of the model ISP as a whole, and may be considered fine-tuning. In this case, the training method may include a further step of simultaneously training two of the at least two modules (and other modules in the model ISP, if any) by performing minor adjustments that help the modules work better together as a coherent pipeline. This is achieved by first receiving one or more reference images in the same manner as for training both the first module and the second module.

[0112] Then, the training method of this example includes iteratively inputting a third degraded image signal representing known degradation of the reference image into the first module of the model ISP. Unlike the first degraded image signal and the second degraded image signal, the third degraded image signal may not only include a subset of the functions of the capture model, such as only adding noise or only adding blur, but may also be an image that is degraded in all aspects modeled by the capture model (e.g., a combination of noise, blur, and CFA simulation) after being processed by the capture model.

[0113] The first module of the model ISP can then process the third degraded image signal by executing the base image signal processing function to produce a first processed image signal. This correction performed by the first module is the same as the correction performed when training the first module when executing the same function, except that the degraded image on which the function is executed is different. An example of a first module correction can be a denoising module that provides a denoising correction.

[0114] The first processed image signal is then input into a second module in the pipeline of the ISP, which processes the first processed image signal by performing a second image signal processing function. This produces a third processed image that has been processed by both the first module and the second module. An example of a second module correction may be a sharpening module that performs sharpening / blur correction. Once the first module and the second module have performed their processing and a third processed image has been produced by the second module, the error between the third processed image and the reference image is calculated by comparing the third processed image to the reference image. This is done in a similar manner as when the first module and the second module are trained individually, however, all properties of the processed image and the reference image are taken into account to calculate the error. The third processed image can be considered as a composite corrected image with the corrections performed by the first module and the second module.

[0115] Once the error between the third processed image and the reference image has been calculated, the error is used to update the first and / or second image processing function performed by the corresponding first module and / or second module. This may include updating the first image processing function and / or the second image processing function by a second amount at each iteration based on at least one second preset learning rate that is less than the first preset learning rate. The second preset learning rate may be an order of magnitude smaller than the first preset learning rate. This allows the model ISP pipeline to be fine-tuned as a whole while ensuring that only small changes are made to each module. In other words, the advantage of using a second learning rate that is smaller than the first learning rate is that it allows modularity to be maintained, because the incremental changes caused by the second learning rate do not substantially change the functionality of the module set by the first learning rate, and therefore do not negate the training of the module in question. Since modularity is maintained, the user can still determine which modules introduce artifacts in the resulting output. This ensures that debuggability exists and it is easy to make changes to the jointly trained network, for example by replacing problematic modules. In addition, this orthogonality also gives the user confidence that the trained model is close to the global minimum. Therefore, in general, this achieves the effect of further refining the model ISP without compromising the modularity achieved by the individual training of each module previously performed. Furthermore, since at least one second preset learning rate is an order of magnitude smaller than the first preset learning rate, the ISP can be fine-tuned so that the processing reaches a global minimum image correction and does not move away from the optimal correction at each iteration. In this way, the fine-tuning achieves the best possible image processing and restores the degraded image signal as closely as possible to the reference image signal. This fine-tuning further ensures that the modules of the model ISP work together. This approach works because the model ISP of the present disclosure is designed to be roughly orthogonal in the sense that the tasks of the modules are well defined and do not overlap. Fine-tuning performs minor adjustments to make the model ISP work well as a whole.

[0116] The fine-tuning iteration process may continue until the calculated error is less than a threshold. This threshold may be preset by a user, or may be automatically generated by a computer. An example of this threshold may be an average pixel difference of a selected property between the second corrected image signal and the reference image signal. The selected property may be a single property or a property that considers multiple aspects of the image, such as a value representing a measure of how close each pixel as a whole is to the same pixel in the reference image signal.

[0117] In some cases, the parameters of the first module and / or the second module can be trained as part of the training method by using a numerical optimizer based on gradient descent and / or back propagation of errors (Adam optimizer). Such optimizers are well understood in the context of training neural networks and can be accelerated using available software tools (e.g., deep learning frameworks such as PyTorch or TensorFlow) and hardware (e.g., GPUs). This allows each module to be trained in a few hours rather than weeks (as is the case in conventional hardware-implemented ISPs that require manual tuning).

[0118] At least two modules (e.g., a first module and a second module) may be any two of a demosaicing module, a sharpening module, a black level subtraction module, a spatial denoiser module, a global tone mapping module, a channel gain module or an automatic white balance and colour correction module. If one of the modules includes a demosaicing module, this module may be the first module to be trained, as it may be needed to train the other modules (whose parameters are frozen).

[0119] Once the model ISP has been trained as discussed above in the training method, the final step is to implement the model ISP on the NNA (or other inference device) and use the NNA to process images captured by the image capture device imitated by the capture model. If the capture model has been constructed correctly, the image capture device has been characterized correctly, and the training of the model ISP has been performed correctly, the final processed image produced by the NNA will be an accurate representation of the scene before it was captured by the image capture device.

[0120] As briefly described above, a module can consist of a simple image processing algorithm (called a "base image processing function") for performing the task of the module and a refinement function for refining the output of the base image processing function. In some examples, the base image processing function will be a simple hand-crafted function, where the refinement function is performed by a (small) neural network. Since the base image processing function and the refinement function together comprise the module, and since they can be trained together in isolation from other modules of the differentiable model ISP, modularity is not violated. The aforementioned advantages of modularity (debuggability, interpretability, etc.) are retained. In addition, each of the base processing function and / or the refinement function is differentiable, thereby allowing the module as a whole to be differentiable. The advantages of designing the module in this way are: computational efficiency, since the hand-crafted functions provide correct results for large parts of the image, so that in difficult cases only correction by a relatively simple network is required; visual quality, since the neural network often exceeds the quality of even highly complex hand-crafted algorithms; efficient execution on the NNA in terms of power, bandwidth, and inference time, since the target hardware is specialized for neural networks and simple hand-crafted algorithms can often be implemented in a simple way on such hardware; and greatly reduced development time, since neural networks are much easier to develop and optimize than hand-crafted algorithms of equivalent quality. In summary, in this approach, the advantages offered by both neural networks (quality, development time) and hand-crafted algorithms (computational and execution efficiency) are available. Figure 5 An example of this can be seen in the Demosaicing module, which is described now.

[0121] During the image capture process, a color filter array (CFA, e.g., a Bayer pattern) transmits only light frequencies corresponding to a specific passband of the color filter, such that a single value is recorded at each pixel from incident light that may contain the full color spectrum, according to the passband of the color filter. However, a full-color image requires three values ​​per pixel, corresponding in the most common case to red, green, and blue. Therefore, a demosaicing module is typically included in the ISP to reverse this sparse sampling process by restoring the full-color image by filling in the blanks based on the context. Since demosaicing is a challenging procedure, particularly in the presence of sharp edges or high-frequency patterns (particularly in these cases where they exceed the Nyquist frequency of the sampling grid and thus introduce aliasing), the demosaicing module of the model ISP of the present disclosure can be configured to perform a basic bilinear demosaicing function, which can then be supplemented, for example, by a convolutional neural network. The bilinear demosaicing function interpolates the value of each pixel that lacks a value in a certain color channel by averaging the values ​​in that color channel of the pixels surrounding the pixel in question. This approach of using small neural networks to correct for the shortcomings of simple algorithms instead of complex manually engineered modules saves engineering and deployment time without violating the principles of modular design as described above.

[0122] Now refer to Figure 5 This approach is described in the figure, which shows the additional refinement function configuration and how such a module of the model is trained. If the demosaicing module is part of the model's ISP pipeline, it can be added to other modules (such as those discussed later). Figure 6 In this case, all aspects of the capture model 12 (except the color filter array (CFA) simulation) may be disabled and the demosaicing module then trained as discussed below, except that training of other components (modules) of the model ISP implemented on the inference device (NNA) may be disabled while the demosaicing module is being trained. This is to prevent, for example, specific lens blur from affecting the behavior of the demosaicing module. When a demosaicing module is present, the demosaicing module may be trained first because it generates an output signal of the correct shape and number of channels, which allows the other modules to be trained with the demosaicing module in place (its parameters frozen) using an autoencoder approach. In other words, without a demosaicer, the input or output of the module being trained may be in the wrong format, for example, Bayer instead of RGB. After the demosaicing module has been trained and its parameters frozen, the other modules of the NNA may be trained as generally discussed above. Figure 5 In the example of , the base image processing function is a bilinear demosaicing function that performs a base refinement of the image signal. The base bilinear demosaicing function attempts to interpolate the gaps caused by the capture model's simulation of the Bayer filtered 501 image / scene (reference image) 500. In this example, the base image processing function is a bilinear demosaicing function 503. The base bilinear demosaicing function 503 provides a base reconstruction level of the image captured by the simulated Bayer filter (degraded image) 502, which was imperfect / corrupted during capture. An example may be that the bilinear demosaicer 503 can perform most of the refinement of the image by performing a simple base image processing function. In this example, the degradation will be most noticeable at sharp edges and where high frequencies are present in the input image 500. This refinement alone may not be acceptable, so it may be beneficial to supplement this refinement with a small neural network 504, which is an example of a refinement function.

[0123] Therefore, in addition to the basic image processing function 503, one or more modules of the model ISP may also include a neural network (e.g., Figure 5 The demosaicing module of FIG. 5 comprises a refinement function 504, which may be a neural network in addition to the bilinear demosaicer 503. This refinement function 504 works together with, i.e. in parallel with, the basic processing functions performed by the module. Each neural network may be specific to the function to be performed, e.g. Figure 5 , the refinement function 504 is a neural network (NN) for the purpose of correcting the output of the bilinear demosaicer 503. The refinement function 504 may supplement the refinement of the module's base image processing function to achieve a quality threshold that is considered acceptable. The neural network within at least one of the modules that may be included in the module may be specific to the functionality of that module and the underlying image processing algorithm that refines the output of the module. In other words, the architecture of the neural network is designed for the task performed by the module. The final image of the module may then be output. Because the refinement function 504 is further refining / correcting the image from the base correction / refinement level produced by the module, less processing is required to produce an output of sufficient quality than if the refinement function 504 performed the full process itself. In . Figure 5 In the configuration shown, the output of the bilinear demosaicer 503 is combined, in this case by addition, with the output of the refinement function 504. In this configuration, the refinement function 504 can be viewed as correcting errors in the output of the bilinear demosaicer.

[0124] y=a(x)-e a y≈a(x)+b(x) Therefore b(x)≈-e a

[0125] Here, y is the reference image 500, x is the degraded image 502, a(x) is the output of the bilinear demosaicer 503, b(x) is the output of the refinement function 504, and e a is the error of the output of a(x) relative to the reference image 500. Therefore, when training this module, the refinement function 504 is trained to approximate the negative error of the base image processing function. It should be understood that this parallel configuration of the base image processing function and the refinement function is given as an example only, and other examples including but not limited to serial configurations are possible. The model ISP is therefore designed to be differentiable to ensure support for backpropagation for training of modules and pipelines, so it is important that the inclusion of small neural networks in the modules does not violate modularity by introducing the disadvantages of undifferentiated neural networks listed above. In the configuration of the present disclosure, modularity is not violated, where the small neural network exists separately in each module in parallel with the base image processing function and can be trained independently of each other.

[0126] The refinement required by refinement function 504 is much easier than correcting the input without the help of the underlying image processing function performed by the module. This in turn makes the cost (in terms of, for example, inference time, power, and bandwidth) of refinement function 504 used to perform the remaining correction / refinement lower. Figure 5Also shown is the error 506 between the initial reference image 500 and the output corrected image 505. This is important for the training of the modules as described above. Typically, the demosaicer is trained in a similar manner to the denoising and sharpening modules, except that the degraded image is produced using an analogy of a Bayer filter or the like, or the degraded image exhibits the same properties as being degraded by a Bayer filter.

[0127] Figure 6 The manner in which a denoising module may be trained is shown. An image / scene (reference image) 600 may be input to a capture process 601 and in the case of training, the denoiser module may contain at least a function that adds noise to the image to introduce a corruption. For example, such corruption may be the addition of Poisson noise to the image. Application of this method matches the noise model and its parameters to the statistics of random physical processes occurring during image capture in a real camera. The variance of the Poisson noise is proportional to the intensity of the (real) pixel, where the constant of proportionality is the gain, and the relationship is Fig.10 , where data is captured under controlled conditions. This produces a degraded image 602 having at least a noise component. The degraded image 602 is then input to the model ISP, which in this example includes at least a denoising module. The denoising module can refine the degraded image using a bilateral filter with an input 604. In this case, the learnable (trained) parameters can be modulated by gain, as discussed below. Optionally, a small neural network can also be included in the denoising module to supplement the refinement performed by the basic denoising function (e.g., bilateral filter). This refinement produces a processed image 605 with reduced noise. The processed image 605 is then compared to a reference image to calculate an error 606. The error is used to iteratively adjust the input parameters 604 of the basic denoising function (e.g., the range variance of the bilateral filter), and if there are parameters of a small refined neural network, it is used to minimize the loss given by the difference between the denoised (processed) image and the reference image.

[0128] For example, Fig.14 A neural network that can be used for a bilateral filter 603 is shown. In this example, the bilateral filter operates on an image captured with a Bayer (or similar) mosaic pattern. Execution proceeds by first separating the input image signal into component images of the same color. For example, an RG1G2B Bayer pattern is first separated into separate R, G1, G2, and B component images. According to the present reference Fig.14In the method described, each component image is denoised individually before being reassembled into a denoised Bayer image. Given a component image having a height H' and a width W' of H / 2 and W / 2, where H and W are the height and width of the degraded input image 602, respectively. The dimensions of the tensor containing this image can be represented as (H', W', 1), where 1 corresponds to the channel dimension. A convolution operation 1401 is performed on this input tensor, resulting in a tensor with K 2 A tensor of channels where the applied convolution kernel has dimensions (K, K, 1, K 2 ), and where the dimensions correspond to kernel height, kernel width, input channels, and output channels, respectively. The weights of this kernel are fixed (i.e., they are not trainable), and in the case of K=3, each output channel contains an offset delta function that follows this pattern (kernels for other values ​​of K are defined similarly):

[0129]

[0130] When convolved with the input tensor, this produces a tensor where the value in the channel at a given position corresponds to the value in the corresponding input window centered at that same point in the input tensor. The next steps are to subtract 1402 the input value from this tensor (in the case of broadcasting across channels), square it 1403, and multiply it by the learned range variance parameter σ 1404 r 2 Steps 1402, 1403, 1404, 1406, and 1410 may be performed on the element-wise operation hardware 2006 of the NNA when the trained model ISP is deployed. A function 1405 that implements the negative exponential of the input is then applied, which may be executed on the activation unit 2004 of the NNA when deployed.

[0131] Through the output of 1405, the function Implemented. We omitted the spatial term for performance reasons. Fig.14 The rest of the graph in implements the function The output of 1405 is multiplied by the input component image 1406, and K 2 channels are summed together 1407 to produce a A single channel tensor of . The K of the output of 1405 2 The channels are summed together individually 1408 to produce a denominator corresponding to Finally, to implement the division of the numerator by the denominator, the output of 1408 is processed by a reciprocal function 1409, which, when deployed, can be executed on the activation unit 2004 of the NNA, and produces the output of applying the bilateral filter to the component image through a multiplication operation 1410. The cross-channel summation 1407 and 1408 can be performed by a multiplication operation with dimensions (1, 1, K2 , 1) kernel, where all weights are fixed to 1 (i.e., not trainable). The entire bilateral denoiser is implemented in terms of a neural network operation as described, which facilitates processing by training equipment and on suitable devices (e.g. Fig.11 Deployment on an exemplary NNA 2000).

[0132] return Figure 6 , a denoising module (which may be referred to herein as a spatial denoiser) may also form part of a model ISP pipeline implemented as part of a command stream on an inference device of the present disclosure. The spatial denoiser module may be implemented by a model ISP to remove random noise (shot noise) introduced during the capture process (e.g., when photons of incident light are captured with a sensor), pseudo-capture events (thermal background noise) caused by thermal radiation in the infrared, or noise inherent to analog electronic components. The spatial denoiser may be configured to receive an image and apply a denoising function or spatial kernel to reduce the noise present in the image. This may be accomplished using a simple bilateral filter or other similar image processing function. The bilateral filter may use a single parameter to control the denoising intensity modulated by the total gain applied to each pixel (because the noise intensity in the image input to the denoiser is controlled by the applied gain). Therefore, the spatial denoiser 204 of the present disclosure may take the total gain as input.

[0133] An ideal denoiser will preserve the structure of the image (e.g., edges and features) while removing noise. A "bilateral filter," as it is known in the art, is a relatively simple and effective means of achieving this. When expressed as an equation, a bilateral filter has the following form:

[0134] in

[0135] W a is the window centered at pixel location a, and b is the pixel from window W a The pixel position within x b is the intensity of the pixel from the input image at pixel location b, and y a is the intensity of the pixel from the output image at pixel position a. and They can be called spatial term and range term respectively, and their strengths are determined by the spatial variance parameters and range variance parameter These parameters can be selected, for example, by training a bilateral filter as described below. In some cases, the spatial term can be omitted and the gain k of the original image incorporated into the range variance, e.g., as in are the trained parameters.

[0136] Where an image processing operation is definable in terms of a differentiable function, as will be appreciated for the case of a bilateral filter with reference to the above formula, the operation may be implemented, for example, in a deep learning framework such as TensorFlow or PyTorch, and its free parameters (e.g., range variance parameter 604) learned by applying conventional numerical optimization algorithms (e.g., stochastic gradient descent with back propagation of error). The image processing operation must be implementable on the target NNA, either as an existing hardware module, or as a combination of multiple hardware modules.

[0137] exist Figure 7 In this case, the module in question is the sharpening module. Figure 7 As can be seen in , the sharpening module performs a sharpening function in the model ISP pipeline, which reverses the image corruption caused by blurring during the capture process (usually by the lens). Figure 7 A scene 700 is shown in which a capture model 701 is corrupted. This corruption is in the form of fuzzy components such as Figure 7 701 is introduced to produce a degraded image 702. The sharpening module of the model ISP can be implemented as part of a command flow on the inference device (NNA) of this example, including using a convolution layer to refine the basic image processing function of the image. This convolution layer can be a depth-wise convolution including three input channels and an output channel, where the weights applied to each color channel (e.g., red, green, and blue) are shared between the channels. The parameters of the convolution layer can be trained using conventional deep learning techniques (e.g., stochastic gradient descent by back propagation of the error), where the error is defined with respect to the reference image 700 and the processed image 704. In this case, the weights will converge to a sharpening kernel that (approximately) inverts the blur kernel 701.

[0138] exist Figure 7In the case of the sharpening module seen in , the module can use a single convolution as the basic image processing function 703 to perform the sharpening function. This can be considered as a simple neural network including one convolution layer with a single input and output channel, or as a hand-crafted function. For example, if this convolution is 7×7, there are 49 parameters "w" to be learned / trained. When regarded as a hand-crafted function, this is an example of a classic spatially invariant linear deconvolution for blurred image reconstruction. The parameters can all be trained at the same time, but may not be adjusted by the same amount or in the same direction, but they are trained so that they converge to a sharpening kernel that removes (or at least reduces) a given simulated lens blur. Note that this is only an example, and other more complex sharpening functions (including independent per-channel kernels, spatially varying kernels, and small neural networks including multiple layers) can be used. This more complex sharpening function can be used when the blur function of the lens (i.e., the "point spread function") varies spatially and is different in different color channels ("chromatic aberration").

[0139] The sharpening module may optionally include additional refinement functions, such as a neural network (not shown), in the same manner as described above with reference to Figure 5 The same as the demosaicing module described above, the additional refinement function is configured to further refine the blurred image. This is another example of a base image processing algorithm being enhanced by another algorithm performed by the base image processing function. The base image processing function and the refinement function can receive inputs "w" 704. These inputs can adapt the parameters of the base image processing function. The input "w" can also be used to adapt the parameters of the refinement function (neural network) that may be present in the sharpening module, however Figure 7 This is not shown in . Once the blurred image has been refined by the sharpening module, the sharpening module is configured to output the processed image 705 to the next module of the model ISP.

[0140] Figure 7Also shown is the way in which the sharpening module can be trained. In the same manner as other modules, a reference image 700 can be input into a capture model 701. The capture model (in this case, shown as a single process of adding blur) 701 that can be used to train the sharpening module includes at least processing the image by, for example, introducing a blur effect into the image to simulate the effect of a lens on light from a scene. This blur effect / damage will be quantified and known based on the measured parameters / properties of the physical camera. The following describes a method for measuring the parameters / properties of the capture model with respect to sensor characterization. The damage introduced by the blur causes a degraded image 702 to be output by the capture process, which can be input into a model ISP pipeline, which includes a sharpening module consisting of at least a basic image processing function 703, which can include a sharpening convolution layer. Once the degraded image 702 has been processed by the model ISP pipeline, the degradation (damage) of the image (in this case, blur) produces a first processed image as described above. This first processed image 705 can be compared with the reference image to calculate the error 706 between the two. The error 706 is then used to refine the parameters within the sharpening module of the model ISP to produce a more accurate and refined processed image in subsequent iterations. This can continue for any number of desired iterations, or can continue until a refinement quality threshold is met, at which point the error 706 between the reference image and the processed image is sufficiently small.

[0141] The use of captured model-corrupted reference images is optional for the training of each module. For example, in some cases, at least one reference image and a corresponding degraded image with known properties can be used to perform training of the ISP module without having to perform degradation as part of the training process. It is important that each module is trained using a pair of perfectly aligned reference and degraded images.

[0142] Figure 8 An example of a model ISP that may be implemented on an inference device (a neural network accelerator (NNA)) is shown. Computational graph 800 depicts an exemplary representation of a model ISP pipeline that may be implemented on an inference device (e.g., an NNA) configured to receive an image signal. This will then be the subject of processing performed by implementing the model ISP on the NNA. An "image signal" may be raw sensor data representing an image, and may also include auxiliary data such as camera or sensor parameters (e.g., gain and black level), or other metadata. This may be a raw sensor data stream representing an image frame or a raw sensor data stream that is held in a buffer until all of the raw sensor data for an image frame has been received.

[0143] Figure 8Each of the modules shown performs a separate function and may be configured to receive inputs related to that function, for example, the black level subtraction module receives as input the black level input to the black level subtraction module as indicated by arrow 809. Each of these modules may receive a number of inputs related to the respective tasks they perform. Figure 8 Each module shown in is to explain how the model ISP 800 implemented on the neural network accelerator can reconstruct an image that has been corrupted by the camera's capture process. Each module can be implemented as a separate and distinct module of the model ISP 800. The model ISP is then converted into a command stream, at which point the modular nature of the model ISP is lost and the nature of the ISP is fixed. Therefore, this is only done after the model ISP is fully trained. Once the model ISP is converted into a command stream, the command stream can be implemented on an inference device, which in the case of the present disclosure is an NNA.

[0144] The model ISP may include a black level subtraction module 801 configured to receive an image signal (e.g., sensor data from a camera) and subtract a baseline black level for each pixel within the image from the image signal, thereby refining the image signal. This may be based on an input black level 809, which may represent a pixel value shift due to an underlying physical process such as dark current. The black level subtraction module may then output the refined image signal to the next module in the model ISP. The black level is typically a constant that can be measured for a given camera in a controlled capture (e.g., exposure with no incident light at the sensor), or read from the manufacturer's documentation.

[0145] The channel gain module 802 may receive the refined image signal from the black level correction module 801 or other modules, and further refine the captured image signal by taking into account the sensitivity of each pixel of the captured image. Different filters in a color filter array (CFA) may transmit different portions of light at different frequencies of the electromagnetic spectrum, which is typically manifested as a particular color (e.g., blue) appearing darker relative to other colors (e.g., green). In addition, the silicon including the sensor itself has different sensitivities to light at different frequencies (i.e., photon capture events are more likely to occur for photons of certain frequencies than for photons of other frequencies). If the pixel sensitivity of the camera used in the capture process is known, an indication of the sensitivity or equivalently a gain that should be applied to correct the sensitivity difference may be input to the channel gain module 802, otherwise this input may be a general sensitivity level. For example, the channel gain module may correct the brightness of the three color channels to offset the sensitivity of the color filter array. The sensitivity or appropriate channel gain of a given camera may be measured under controlled conditions, such as with reference to a test chart with calibrated colors.

[0146] The model ISP that can be implemented on the NNA of the present disclosure may also include information about Figure 6 Denoising module 803 described and about Figure 5 The demosaicing module 804 described above. The model ISP may also include Figure 7 Described sharpening module 805.

[0147] A color correction module 806 and an automatic white balance module 807 may also be present and configured to correct color and refine white balance, respectively. Color correction may be performed to match the human visual system by blending the color channels of the input image based on a sensor color profile corresponding to the sensor that captured the image. Automatic white balance may be performed, for example, with reference to channel averages (e.g., implementing a white balance algorithm known in the art as a "gray world").

[0148] The model ISP implemented on the NNA may also include a global tone mapping module 808 to perform global intensity manipulations, such as gamma correction or histogram equalization, which ensures that the image gives a correct or visually appealing output image when displayed. This can be achieved by mapping pixel levels from one dynamic range to another to refine the image. For example, this can be used to represent a high dynamic range input on a low dynamic range display.

[0149] It should be understood that the Model ISP of the present disclosure may include other suitable modules not described above, and may be implemented on an NNA.

[0150] Fig. 9 Another exemplary computational diagram is shown, which illustrates the configuration that can be adopted by the pipeline of the model ISP implemented on the NNA of the present disclosure. In particular, Fig. 9 An example of a model ISP pipeline 700 is shown that may be converted into a command stream for configuring an NNA. Fig. 9The model ISP and the boundaries between the modules of the model ISP are shown, and the model ISP is shown as a differentiable computation graph at the level of its constituent primitive operations. These primitive operations are selected to be implementable on the target NNA by means of a compilation or mapping process that converts the computation graph into executable machine code. It should be noted that from the perspective of the compiler, the computation graph 900 appears as a neural network, but as described in detail above, it has important structures in its modular design and training methods that distinguish it from undifferentiated neural networks. Each primitive operation can act on a tensor representing an image / image signal. For example, black level subtraction can execute a single command for performing element-by-element addition between an image input and a constant input representing a (negated) black level. These two commands constitute the black level subtraction module and can be considered in isolation from other commands of the pipeline. In the model ISP of the present disclosure, these commands can be replaced or customized without substantially changing the functionality of other commands in the pipeline. For Fig. 9 The same can be done for each of the other modules shown in . Intermediate commands between modules can also be included in the pipeline. These can perform simple conversions or transformations, such as changing the shape or size of a tensor output from one module and input to another. These intermediate commands are usually very simple compared to the commands within each module.

[0151] We will now further examine Fig. 9 The exemplary de-mosaic module 904 is shown in the exemplary pipeline of . Fig. 9 The demosaicing module 904 shown in has two branches 920, 921, each of which performs a portion of the module's functionality. Branch 921 may include: a layer that performs a "space to depth" operation (not shown) that arranges each 2×2 region of the Bayer image in the channel dimension to output a tensor with half the height and width of the input; and 4 channels (RGGB). This allows for easier processing using conventional convolution operations and transposed convolution operations. Fig. 9 In the case of , branch 921 performs a bilinear demosaicing function (base image processing function 905), for example as a transposed convolution operation that outputs non-overlapping 2×2 quadruplets of R, G, and B pixel values, and branch 920 performs a refinement function. Fig. 9 As can be seen in , the basic image processing function performed by branch 921 can be a relatively simple function, and can be a transposed convolution. The small neural network (refinement function) associated with the demosaicing module 904 shown on branch 920 of the demosaicing module 904 is a 4-layer convolutional neural network with rectified linear unit (ReLU) 907 activations between convolutions 906. Convolution 906 can be a standard convolution or a transposed convolution (also known as "deconvolution" in the art, not to be confused with deconvolution in the context of removing blur) to restore the convolved image signal to the original input resolution. As shown in Fig. 9 As can be seen in the final step of the demosaicing module 904 shown in , branch 921 and branch 920 are then combined in an element-by-element addition layer, the output of which is the final output from module 904. Therefore, these functions are executed as parallel branches within module 904 (although when deployed on an inference device, these operations can be arranged to be executed in a sequential or parallel manner). In the case of the demosaicing module 904, the neural network branch 920 refines the basic demosaicing function (basic image processing function) 921 to improve the quality of the image output by the module, which is consistent with the above reference to Figure 1 and Figure 5 The principle described is the same.

[0152] Fig. 9 The demosaicing module 904 shown in then passes the output to a sharpening module comprising 2D convolutions, and then in turn to a color correction module 909 comprising a single 1×1 convolutional layer, which implements matrix multiplications for each pixel, which perform linear color space manipulations to match the colors expected by, for example, a human observer. In a similar manner to the black level subtraction module 901, the parameters (color conversion matrix) can be measured from a physical sensor under controlled capture conditions, and can therefore be predefined rather than learned. The output of the color correction module 909 is fed into the automatic white balance 910 module, and then into the gamma correction 912 module. In Fig. 9 In the example of , modules 903, 908 and 909 do not include a refinement function, however, the image / image signal may be divided / sliced ​​between the modules. This may allow different chromatic aberrations in each color channel to be corrected independently in a module such as sharpening module 908 before being cascaded back into the 3-channel image. Different portions of the image to be sharpened may be sharpened by different amounts before combining the refined sharpened images. In this case, the cascading step may be performed in sharpening module 908. Fig. 9 In the example of , the last stage before outputting the final image is gamma (γ) correction 912. To perform gamma correction, the LRN (local response normalization) hardware implements this function:

[0153]

[0154] By setting k to 0, γ to 1 and the window size to 1, this simplifies to b = a^(1-2*β). Now, to raise a to the power of p, we set p = 1-2*β, and we get β = (1-p) / 2. Since the gamma correction is a power function, typically p = 1 / 2.2, we can substitute this into the formula and get β = 0.273. Therefore, using these parameters in the LRN hardware module on the NNA will produce the gamma correction function as desired.

[0155] Fig. 9 The boxes shown in represent operations / commands performed by the module, while arrows represent tensors. One channel of input to the model ISP is a monochrome raw image with one color value per pixel location, such as red, green, or blue. Fig. 9 The demosaicing module can be a 4-layer convolutional neural network with ReLU activations between convolutions. For example, the neural network can include a first convolution with a stride of 2 (to be consistent with a 2×2 Bayer unit), a kernel size of 10×10, a single input channel of the image signal tensor, and an output image signal tensor (at half resolution due to the stride of 2). The second convolution has a stride of 1, a kernel size of 3×3, and 32 output channels. The third convolution has the same properties as the second convolution. The final convolution is a transposed convolution (deconvolution) with a 5×5 kernel and a stride of 2 (to bring it to the original input resolution of the image signal) and 3 output channels (corresponding to RGB).

[0156] Each module of the ISP model may include a series of parameters as part of the base image processing function or refinement function. These parameters may be adjusted during the training of each module as previously described.

[0157] Each of the at least two modules can be trained independently of the other modules. This is because each module is functionally different from the other modules, so each module can be trained separately. The same is true for the operation of each module, because each module will be used in turn to process the image signal. In other words, the parameters of each module (which may include weights) can be updated and refined during training. In some cases, some modules will not include learnable parameters, for example, modules such as color correction, black level subtraction, gray world white balance, and gamma correction modules.

[0158] The inference device (in some cases, the NNA) can be configured to select and implement a model ISP from a series of model ISPs based on the requirements of the required image processing. Each of the model ISPs that the inference device can implement can have a different configuration. In other words, each of the model ISPs can have a different pipeline of functional blocks and / or a different set of one or more parameters of the functional blocks of the pipeline. This can be done by implementing one or more of the above modules that execute each functional block as part of the pipeline. In this way, the NNA of the present disclosure can implement multiple different model ISPs, each with a different configuration. Each model ISP is converted into a command stream to be implemented on the inference device of the present disclosure. Once this conversion is performed, the modularity of the model ISP is lost, so this conversion is only performed when the model ISP has been fully trained. This allows reversing and considering different capture processes, which may be the capture process of different cameras that capture images. This can also allow the inference device to implement a model ISP that is specifically configured to use these conditions when processing images that have been captured under certain conditions (e.g., low lighting conditions). In addition, the inference device of the present disclosure can be implemented as hardware on a single silicon chip or software on a single silicon chip. The model ISPs selected by the inference device from it can be considered as "virtual ISPs", and they can be implemented simultaneously or sequentially. For example, in order to process captured video in real time, a model ISP with short processing time but lower quality refinement can be run on the inference device at a high frame rate. Conversely, a model ISP with long processing time but image refinement that achieves a relatively high quality level can be occasionally run on the inference device for processing still photos. For single-core and multi-core neural network accelerators, the model ISPs can be time-division multiplexed to make them appear to be running simultaneously, or, if the neural network accelerator consists of multiple cores, multiple model ISPs can be executed simultaneously on different cores.

[0159] Due to the modular configuration of the model ISP of the present disclosure, each module that performs the set tasks of the ISP pipeline can be individually customized and refined in a different and separate manner than discussed above with respect to training. Not every module must include a refinement function (neural network), but some modules may include refinement functions if necessary, for example, the demosaicing module 510, 904 in Figure 5 and Fig. 9The ISP is shown as including a neural network, but the sharpening module or any other module may also include a neural network. The parameters of the neural network associated with each module and the hand-crafted components such as the basic image processing function can be independently customizable to provide the best refinement for that functional block of the model ISP. However, both the basic image processing function and the refinement function are represented as a single neural network in the deep learning framework for training. In other words, both the basic image processing function and the refinement function can be parts of a neural network that performs different functions. This allows the functionality of each module to be trained and changed separately from other modules before or after implementation on the hardware, allowing for easier adjustment and, importantly, maintaining modularity. In addition, the model ISP can be updated after deployment on the NNA by updating the parameters of the different modules and recompiling for the NNA.

[0160] Reference now Fig.11, which shows an exemplary inference device, in this case the NNA 2000, which can dynamically select and adjust the format of network parameters while the NNA is executing a pass process (e.g., a forward pass process or a backward pass process) of a neural network. The NNA can be configured to implement a command stream that represents a differentiable model ISP as a combination of operations from a set of basic neural network operations available on the inference device. The differentiable model ISP has a pipeline of separate (in some cases, two or more) functional blocks. The set of basic neural network operations consists of one or more of the following: convolution operations; pooling operations; element-wise operations; activation operations; local response normalization operations; tensor rescaling operations; channel permutation operations; shaping operations; cascading; reduction operations, including summing, averaging, minimization, and maximization. The NNA 2000 includes: one or more network processing hardware units 2002, 2004, 2006, 2008, 2010, 2012, 2014, 2016, each of which is configured to accelerate one or more neural network operations; one or more statistics collection hardware units 2018, 2020, 2022, 2024, each of which is configured to dynamically collect one or more statistics about a set of network parameters for a layer of an artificial neural network (ANN) while the NNA is executing a neural network pass process; and one or more format conversion hardware units 2026, 2028, 2030, 2032, each of which is configured to convert a set of network parameters into a configurable or selectable digital format. The one or more statistics collected by the statistics collection hardware unit can be used to select a digital format that can be applied to the set of network parameters from which the statistics were generated, another set of network parameters for a layer, or a set of network parameters related to the original set of network parameters in a subsequent pass process of the neural network. This allows the digital format for one or more sets of network parameters to be selected and / or adjusted on the fly (ie, while the NNA is performing a forward pass or a backward pass of the NNA).

[0161] like Fig.11 As shown in , the NNA may also include one or more buffers or units 2034, 2036, 2038 for receiving data input to the NNA, and / or an output unit 2040 for outputting data from the NNA. For example, Fig.11The exemplary NNA 2000 in includes: an input buffer 2034, which is configured to receive and store input data received from a memory; an auxiliary data input unit 2036, which is configured to receive input data received from the memory; and a coefficient buffer 2038, which is configured to receive parameters (weights) for a convolutional layer.

[0162] Each network processing hardware unit 2002, 2004, 2006, 2008, 2010, 2012, 2014, 2016, 2018 includes hardware logic that is configured to accelerate the execution of one or more neural network operations on input data. Specifically, each network processing hardware unit 2002, 2004, 2006, 2008, 2010, 2012, 2014, 2016 is configured to receive input data, perform one or more operations on the received input data using internal hardware logic, and output the results of the one or more operations, which can be referred to as output data. Since the input data for a layer can be processed by multiple hardware processing units to generate output data for the layer, the input data for the hardware unit may or may not correspond to the input data for the layer, and the output data of the hardware unit may or may not correspond to the output data of the layer.

[0163] Fig.11An exemplary NNA of 2000 includes the following network processing hardware units: a convolution processing unit 2002, which includes one or more convolution engines configured to perform convolution operations on received input data; an activation processing unit 2004, which is configured to perform nonlinear operations on received input data; an element-by-element operation processing unit 2006, which is configured to perform element-by-element operations on received input data; a normalization processing unit 2008, which is configured to perform normalization operations on received input data; a configurable pooling processing unit 2010, which includes one or more pooling engines configured to selectively perform pooling operations or depth-wise convolution operations on received input data; an interleaving processing unit 2012, which is configured to perform interleaving or rearrangement of received input data; and a tensor rescaling processing unit 2014, 2016, which is configured to scale or rescale the input data. Each of these network processing hardware units is described in more detail below. One or more of the network processing hardware units may be implemented as fixed function circuitry. It will be apparent to those skilled in the art that this is only an example set of network processing hardware units, and other NNAs may have additional network processing hardware units, fewer network processing hardware units, and / or different network processing hardware units.

[0164] Fig.11 The exemplary NNA 2000 can be configured to perform a transfer process (e.g., a forward transfer process or a backward transfer process) of the NN in one or more hardware transfer processes of the NNA. The hardware transfer process of the NNA is defined herein as loading input data into the NNA and processing the input data using one or more network processing hardware units to generate processed data or output data, which in the case of the present disclosure is a processed corrected image. The output data of the hardware transfer process can be output from the NNA to a memory via an output unit 2040, or stored in the NNA for use in a subsequent hardware transfer process. The memory used to store the output data of the hardware transfer process can be a memory external to the NNA, but inside the chip where the NNA is located (i.e., on-chip memory), or a memory external to the NNA and outside the chip where the NNA is located (i.e., off-chip memory).

[0165] The NNA may have hardware constraints (e.g., the size of the input buffer 2034, coefficient buffer 2038, and / or shared buffer 2041, the number of convolution engines, the number of pooling engines) that limit the processing that can be performed during a hardware pass, or the order or number of times a hardware pass can use network processing hardware units. In the event that all processing to implement the neural network cannot be completed in a single hardware pass of the NNA 2000, the processing may have to be divided into multiple hardware passes of the NNA 2000.

[0166] In some cases, a hardware pass process for performing or implementing a pass process of a neural network can be identified by first mapping each layer of the neural network to a sequence of one or more lower-level layers, where a lower-level layer is a set of one or more operations that can be performed by a single hardware processing unit of a neural network accelerator. In other words, each lower-level layer corresponds to a hardware processing unit of a neural network accelerator. The lower-level layer to which a layer is mapped can depend on whether a forward pass process or a backward pass process is to be performed. For example, one set of operations can be used to implement or process a forward pass process of a layer of a neural network, and a different set of operations can be used to implement or process a backward pass process of the layer.

[0167] Once the layers of the neural network have been mapped to low-level layers, the low-level layers are divided into one or more layer groups, where each layer group includes a sequence of one or more low-level layers that can be implemented on the NNA 2000. The sequence of lower-level layers that can be implemented by the NNA 2000 depends on the network processing hardware units of the NNA and how the network processing hardware units can be connected to process data. Fig.11 The exemplary NNA 2000 has convolution processing units and configurable pooling processing units that can be connected to form a pipeline, so that the NNA 2000 can perform convolution calculations for layers and pooling calculations for layers together. This means that a layer group can include a low-level convolution layer followed by a low-level pooling layer. In contrast, since the NNA 2000 includes only a single configurable pooling processing unit, it cannot perform pooling operations for two different low-level layers. Therefore, in this example, the layer group cannot include two pooling layers. In other words, since each low-level layer corresponds to a hardware processing unit, each layer group includes a series of hardware processing units that can be implemented by a neural network accelerator.

[0168] Once the low-level layers are partitioned into one or more layer groups, it is determined for each layer group whether the layer group can be implemented in a single hardware pass of a neural network accelerator. Specifically, depending on the NNA hardware constraints, it is not possible to perform all processing associated with the layer group in the same hardware pass. For example, the input tensor for the first layer of the layer group may be too large to be processed in a single hardware pass. Therefore, if it is determined that the layer group cannot be implemented in a single hardware pass of a neural network, the layer group is divided into multiple hardware pass processes. Dividing the layer groups may include dividing the input tensor in one or more dimensions. For example, the input tensor can be divided along one or more of the width, height, or input channel dimensions; and / or for a convolutional layer, the convolution kernel can be divided along one or more of the input channel dimension and the output channel dimension (which may also be referred to as a filter dimension). An example method for identifying a hardware pass process to execute a pass process of a neural network is described in the applicant's UK patent application 2209584.8, the entire contents of which are incorporated herein by reference.

[0169] exist Fig.11 In the exemplary NNA 2000 of FIG. 1 , the network processing units are connected to a crossbar switch (“Xbar”) 2042, which can connect a selected set of network processing hardware units in a selected order on a per hardware transfer process basis to form a pipeline of network processing hardware units through which input data for the hardware transfer process is transferred. Thus, at each hardware transfer process, the NNA receives information indicating which network processing hardware units are active and the order in which the active network processing hardware units will process the input data. The crossbar switch 2042 then connects the input ports and output ports of the identified network processing hardware units accordingly. The crossbar switch 2042 may include a plurality of input ports (in Fig.11 ) and multiple output ports (indicated by arrows pointing to the crossbar switch 2042 in FIG. Fig.11 2042 is indicated by an arrow pointing outward from the crossbar switch 2042).

[0170] Compared to having fixed connections between network processing hardware units, having a crossbar switch 2042 that can dynamically connect network processing hardware units in multiple different orders increases the number of different layer combinations that can be in a layer group. This can reduce the hardware transfer process used to implement the ANN. However, it will be apparent to those skilled in the art that this is just an example, and other NNAs can connect network processing hardware units in another manner. For example, in other NNAs, network processing hardware units can be connected in a fixed order (e.g., to form a fixed pipeline).

[0171] In order to perform the transfer process of the neural network on the neural network accelerator, the network parameters are represented in a digital format, such as but not limited to a floating point format or a fixed point format. The term "network parameter" is used herein to mean any tensor or part of a tensor used in the transfer process of executing the neural network, and includes but is not limited to the input data value of the low-level layer (which may or may not correspond to the input data value of the layer of the neural network), the output data value of the low-level layer (which may or may not correspond to the output data value of the layer of the neural network), the parameters for the low-level layer, and the deviation for the low-level layer. Because different low-level layers may have different input value ranges and output value ranges, it may not be effective to use a single digital format for all network parameters. For example, one low-level layer may have input data values ​​between 0 and 6, while another low-level layer may have input data values ​​between 0 and 500. Therefore, using a single digital format may not allow any set of input data values ​​to be represented effectively or accurately. Thus, NNA 2000 includes one or more format conversion hardware units 2026, 2028, 2030, 2032 that can dynamically convert a set of network parameters into a selected format. Specifically, each format conversion hardware unit 2026, 2028, 2030, 2032 is configured to: receive a set of network parameters; receive information identifying a desired digital format (in Fig.11 ) and dynamically converting a received set of network parameters into a desired digital format. This allows the digital format for the network parameters to be selected and implemented on a per-low-level layer basis, and even on a per-hardware pass basis for each low-level layer.

[0172] Each statistics collection hardware unit 2018, 2020, 2022, 2024 is configured to receive a first set of network parameters input to or output from a unit of the NNA and generate or collect one or more statistics (in Fig.11 The one or more statistical data may be used to select a digital format based on a format selection algorithm.

[0173] In addition to the above-mentioned neural network accelerator. The present disclosure also provides a method for image processing using an inference device, wherein the inference device is configured to implement a command stream that represents a differentiable model ISP as a combination of operations from a set of basic neural network operations available on the inference device. The differentiable model ISP has a pipeline of separate (in some cases, two or more) functional blocks. The set of basic neural network operations consists of one or more of the following: convolution operations; pooling operations; element-by-element operations; activation operations; local response normalization operations; tensor rescaling operations; channel permutation operations; shaping operations; cascades; reduction operations, including summation, averaging, minimization, and maximization.

[0174] In a method of image processing, an image signal may be processed using an inference device acting as an ISP, the ISP having at least a first module and a second module before conversion into a command stream. This inference device (NNA) may be the inference device described above or an alternative NNA. When processing an image signal, the inference device will perform processing of each module of the model ISP by implementing a command stream representing a differentiable model ISP pipeline. In other words, the inference device will perform image processing functions represented by each module of the model ISP, such as denoising, demosaicing, sharpening, etc. However, such modules will not be distinguishable in the command stream, and the command stream will represent the combined collective function of the ISP pipeline. Refinement of the image signal represented by the command stream will be performed until the image signal is fully refined and can be output after being processed by the inference device acting as an ISP pipeline. The first module and / or the second module of the model ISP implemented on the inference device may be any of the above-mentioned modules, or a module that performs a known alternative function for image signal processing. The first module and the second module may be the same or different from each other.

[0175] One or more of the at least two modules of the model ISP (including the first module and the second module of the model ISP) may include a basic image processing function. In the case of the above-mentioned first module and the second module, the first module may include a first basic image processing function, and the second module may include a second basic image processing function. Each of the first basic image processing function and the second basic image processing function forms a part of the first functional block and the second functional block of the model ISP, respectively. These basic image processing functions perform refinement of the image signal and can be implemented by the logic of the command flow of the inference device.

[0176] The method may include iteratively performing the first basic image processing function and / or the second image processing function. In this case, the processing performed by the first module and / or the second module may form a loop around the module processing to further refine the image signal. If a neural network is present within the module, this may be achieved by forming a loop around the neural network. Alternatively, if desired, each module as a whole may iteratively process the image signal passed to it. In this way, each of the modules may perform its corresponding image processing function on the image signal multiple times during multiple passes through the module.

[0177] The image processing method of the present disclosure may further include the step of selecting an image signal processor from a series or a group of image signal processors that can be implemented on the inference device before executing the above method.

[0178] As described above, each module of the differentiable model ISP for performing the image processing method may have been trained or may be independently trained. This allows some modules to be refined while other modules remain unchanged, thereby allowing specific parts of the model ISP on the NNA to be trained separately. The ability to train individual modules of the differentiable model ISP greatly speeds up the training process, because each module can be trained on a time scale of minutes rather than days or weeks that is usually required to manually tune the ISP. The training of these modules can be achieved by updating the parameters of the underlying image processing functions and / or neural networks. In this way, the parameters can include the weights of the functions performed as well as other parameters.

[0179] An example of sensor characterization will now be described. When characterizing a camera, properties such as channel sensitivity, color aliasing, dark noise (noise associated with photon capture), and conversion gain (the relationship between the "true" intensity of a pixel and the intensity of the noise) can be used. When considering how to characterize color calibration, the response of the three Bayer color filters will be different from the response of the human visual system (HVS) based on the three types of cone cells in the human retina.

[0180] To perform color calibration, a test chart may be used that includes areas of pure colors (e.g., pure red, pure green, and pure blue). These pure colors may be considered standard reference colors and may, for example, correspond to certain colors in a given color standard. The colors of pure red (R), pure green (G), and pure blue (B) may be represented by the following unit vectors.

[0181]

[0182] This means that in the absence of color distortions (e.g., channel sensitivity and color aliasing), pixels in an image of such a test chart would ideally be recorded by the sensor as pure red, pure green, and pure blue, up to the global scale factor. However, the colors recorded by a real-world sensor (R', G', B') are different from the displayed pure colors (R, G, B), so the recorded image will require a color correction process to restore the image of the test chart recorded by the sensor to the image of the test chart as seen by the HVS.

[0183] The reference scene is then captured with the camera, ensuring that the image is in focus and that the gain and exposure are set to default values. The capture should be set in such a way that external factors such as lighting do not change the apparent color of the red, green, and blue blocks and that the pixels are not saturated. The red, green, and blue intensities are then averaged for the nominal red, green, and blue regions of the image. Then, a mapping is found between the [R', G', B'] color recorded by the sensor and the [R, G, B] color of the HVS. That is, the parameters of the model are determined so that the average recorded color of the red region is converted to pure red, the average recorded color of the green region is converted to pure green, and the average recorded color of the blue region is converted to pure blue. A simple offline demosaicing operation such as bilinear demosaicing can be applied to interpolate the recorded values ​​to find the red value, green value, and blue value for each pixel position, thereby simplifying the implementation of this process. In order to characterize the color calibration, a simplifying assumption can be made as known in the art, that is, the recorded colors are related to pure colors by linear mapping (that is, they can be modeled by multiplication of matrix A). It should be understood that this is only an example and other (possibly more detailed) models can be used. The red, green, and blue intensities are then averaged for nominal red, green, and blue regions of the image. This produces the following vectors:

[0184]

[0185] These correspond to the measured apparent colors of the red, green, and blue regions, respectively. The matrix A that maps pure values ​​to recorded values ​​is the concatenation of these vectors, as can be appreciated, since it maps pure colors in the reference scene (represented as unit vectors as described above) to measured colors:

[0186]

[0187] In further detail, the above relationship is as follows.

[0188]

[0189] The columns of the matrix A consist of the recorded red, green and blue colors given by the following expressions (in the case of the identity matrix in the above definition of A):

[0190]

[0191] In other words, A can be constructed directly from the recorded colors. This relationship can then be inverted to find the matrix A -1 , which maps recorded colors to pure colors:

[0192]

[0193] Therefore, when A -1 When applied to a recorded image, these colors are corrected to essentially match pure colors.

[0194] When used as part of an ISP, it is desirable to perform the above calibration in two discrete stages during the pipeline, specifically as part of the channel gain block and the color correction block. To obtain the parameters of these blocks, it is necessary to transform the matrix A -1 Decomposed into two matrices M and λ corresponding to color correction and gain, respectively. M maintains the brightness of the channels in the new normalized color space (i.e., the rows sum to 1), and λ is a set of scaling factors and is a diagonal matrix. The diagonal of λ can be found by summing the rows of A. λ applies the scaling factors to each channel independently, so it is necessarily a diagonal matrix, and is therefore of the following form:

[0195]

[0196] In more detail, by first -1 The factorization is done by summing the elements of each row of A. This gives three values, corresponding to red, green, and blue. These values ​​are the channel gains. These channel gains are the diagonal of the matrix λ. The matrix A can then be calculated by -1 Each row of A is divided by the corresponding channel gain to find M. This makes the matrix A -1 The rows of are normalized so that they sum to 1, and the following expression is produced:

[0197]

[0198] The corresponding factorization of the matrix A, as might be used, for example, in a capture model, can be found by inverting the color correction matrix M to produce a “color confusion matrix”. Similarly, the channel gain matrix λ can be found by inverting the channel gain matrix λ, such that λ -1 or [1 / λ r ,1 / λ g ,1 / λ b] to find the channel sensitivity. Additionally, once λ has been found, a global scaling factor can be applied to avoid changing the overall brightness of the image. For example, all values ​​in λ can be divided by the maximum value in λ. λ can alternatively be scaled by different amounts depending on the user's needs.

[0199] With regard to the capture setup, a composite image can be created, for example, using image manipulation software. For example, the composite image can contain pure red, green, and blue patches. For the purpose of the capture setup, the composite image can be displayed on a (ideally calibrated) computer monitor and captured with a camera. The advantage of displaying the image on a monitor is that the chart is self-luminous, which means that it is uniform and less susceptible to interference from other (color) light sources. This means that color calibration can be performed in a relatively uncontrolled capture environment; however, care should be taken not to let light shine on the display. For example, red patches, green patches, and blue patches can be averaged to eliminate the effects of noise, thereby obtaining an A matrix (these averages are cascaded according to the above method). According to the above method, the A matrix can be decomposed into a color confusion matrix M and a (rescaled) sensitivity matrix λ. Then, the inverse of these can be applied to the image captured by the sensor to check whether they perform corrections, such as by checking whether the resulting image corresponds to the original composite image.

[0200] The black level of the sensor can be characterized by reducing the gain to minimum, capturing with the lens cap on, and then averaging the measured pixel intensities.

[0201] The noise of the sensor can be modeled as having two components: "dark noise", which is assumed to have a fixed intensity (variance) independent of exposure time or intensity; and photon capture noise. Photon capture can be modeled with a high degree of accuracy as a Poisson process, where the variance at each pixel is proportional to the true intensity. The proportionality constant can be called the "conversion gain" k. In order to measure the intensity of the noise component, multiple captures can be taken. According to the example, 128 captures are taken with a static camera. For each pixel position, the mean and variance are calculated. Fig.10 The graph shown in shows mean intensity versus variance, with the points colored according to their location in the Bayer array (red corresponds to red pixel locations, etc.). A noise model can then be fit to these points, expecting a straight line with an offset.

[0202]

[0203] where k is the gradient of the line and is the offset. A least squares best fit can be applied to obtain values ​​for the two parameters of this statistical model.

[0204] The above exemplary method can be used for camera characterization or as part of camera characterization. Once the properties of a camera have been characterized, the corresponding parameters of a capture model can be configured according to the characteristics of the camera so that it can be used to simulate the action of the camera on an input image.

[0205] In addition to the above-mentioned training method for training a model ISP to act as an ISP, the present disclosure also includes a training device configured to perform the method discussed above. The training device will now be described below.

[0206] The training device disclosed in the present invention is used to train a differentiable model ISP, wherein an image signal processor has a pipeline of separate image signal processing functions, wherein the differentiable model of the image signal processor includes at least two differentiable modules, and each of the differentiable modules of the differentiable model of the image signal processor is configured to execute a corresponding single image signal processing function of the pipeline. The training device includes one or more processors, wherein the one or more processors are configured to iteratively execute the above-mentioned Figure 4 The steps (S112 to S115) of the training method are used to train the first module of the differentiable model ISP to perform the first image signal processing function, while other modules of the differentiable model ISP are not trained.

[0207] The one or more processors of the training device may be configured to stop the iterative process when the error calculated in step S114 is less than a threshold. This threshold may be the same as described above with respect to the training method, but is configured to be registered by the one or more processors of the device, which then stop the iterative process. The one or more processors may then output the iterative corrected image signal as a final corrected image signal, or may convert the corrected image signal into an image that can be viewed by a user on a display.

[0208] In addition, the modules of the differentiable model ISP trained by the training device of the present disclosure can be configured as discussed above with respect to the model ISP. In other words, each module may include a basic image processing function and a refinement function (refined neural network). One or more processors may also be configured to update one or more parameters of the first image processing function by a first amount in each iteration based on at least one first preset learning rate in the update step S115 discussed above. This is the same as in the above-mentioned training method. If the module includes a refinement function, the first image processing function as discussed above may be formed by a basic image processing function and a refinement function (refined neural network). Therefore, the term first image processing function may be regarded as a general function of the module, such as a denoising function in the case of a denoising module.

[0209] One or more processors of the training device of the present disclosure may also be configured to train a second module of the model ISP to perform a second image signal processing function different from the first image signal processing function, while not training other modules of the model ISP. The one or more processors may be configured to train the second module in substantially the same manner as the first module and as discussed above with respect to training the second module in the training method. The training device may be configured to train the second module to the first module in sequence. In other words, the one or more processors may be configured to train the second module before or after the first module but not simultaneously with the training of the first module / not during the training of the first module. In this way, the training device can train each module individually and only refine the image processing functions of the corresponding modules that need to be trained. This allows only parts / modules of the model ISP pipeline to be refined without changing other parts / modules.

[0210] In some examples, one or more processors may also be configured to train two modules of the ISP simultaneously, for example, once each module has been trained independently. In this configuration, one or more processors are configured to receive a reference image. This reference image can be converted into a (reference) image signal for use in the training method. One or more processors may be configured to perform the iterative process described above with respect to the training method. In this case, the iterative process is to input the third degraded image signal into the first module, which is then processed by the first module to produce a first processed image signal. Next, the first processed image signal is input into the second module and then processed to produce a third processed image. The processor is then configured to calculate the error between the third processed image and the reference image, and update the first image processing function and / or the second image processing function performed by the corresponding first module and / or second module based on the calculated error.

[0211] The training device, and more specifically, the one or more processors may also be configured to fix the parameters of the first module or the second module that are not trained when the other of the first module or the second module is trained. This allows each module to be trained separately by the device. In addition, the one or more processors may also be configured to update the parameters by a second amount each iteration based on a second preset learning rate as described in the training method above when updating the parameters of the first image processing function and / or the second image processing function.

[0212] In the above disclosure, the first preset learning rate is efficient in providing fast convergence to an approximate solution (optimal correction), but when fine-tuning, it may be beneficial to reduce the learning rate (to a second preset learning rate) so that the solution converges to the true / minimum value. The reduction in the learning rate means that the change in the parameters will not deviate from the desired solution after converging to the minimum value.

[0213] The one or more processors of the training device of the present disclosure may also be configured to perform any of the functions described above with respect to the training method, and the modules of the ISP trained by the training device may include any of the modules discussed above.

[0214] Fig.12 A computer system is shown in which the graphics processing system described herein can be implemented. The computer system includes a CPU 1202, a GPU 1204, a memory 1206, and other devices 1214, such as a display 1216, a speaker 1218, and a camera 1222. Processing block 1210 is implemented on GPU 1204 and a neural network accelerator (NNA) 911. In other examples, processing block 910 can be implemented on CPU 1202 or in NNA 1211. NNA 1211 can be one of the NNAs described above (e.g., an inference device). The above-mentioned training device can be implemented on processing block 1210. The components of the computer system can communicate with each other via a communication bus 1220. Storage device 1212 is implemented as part of memory 1206.

[0215] Although Fig.12 A specific implementation of a graphics processing system is shown, but it will be understood that a similar block diagram can be drawn for an artificial intelligence accelerator system—for example, by replacing the CPU 1202 or GPU 1204 with a neural network accelerator (NNA) 1211, or by adding the NNA as a separate unit. In such cases, the processing block 1210 can also be implemented in the NNA.

[0216] Figure 2 The model ISP is shown as comprising a number of modules, each module performing a functional block. Each functional block may be provided in any suitable manner. It will be appreciated that intermediate values ​​described herein as being formed by the NNA need not be physically generated by the NNA at any point in time, and may simply represent logical values ​​that conveniently describe the processing performed by the NNA between its input and output terminals.

[0217] The inference device and training equipment described herein may be included in hardware on an integrated circuit. The inference device and training equipment described herein may be configured to perform any of the methods described herein. Generally speaking, any of the functions, methods, techniques, or components described above may be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. The terms "module," "function," "component," "element," "unit," "block," and "logic" may be used herein to generally represent software, firmware, hardware, or any combination thereof. In the case of software implementation, a module, function, component, element, unit, block, or logic represents a program code that performs a specified task when executed on a processor. The algorithms (functions) and methods described herein may be performed by one or more processors that execute code, which causes the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that may use magnetic, optical, and other technologies to store instructions or other data and may be accessed by a machine.

[0218] As used herein, the terms computer program code and computer readable instructions refer to any kind of executable code for a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (e.g., a hardware description language or netlist), and code expressed in programming language code such as C, Java, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, run in a virtual machine or other software environment, causes a processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0219] A processor, computer or computer system may be any kind of device, machine or special purpose circuit, or a collection or part thereof, which has processing capabilities so that instructions can be executed. A processor may be or include any kind of general or special purpose processor, such as a CPU, GPU, NNA, system on chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0220] The present invention is also intended to cover software (e.g., HDL (Hardware Description Language) software) that defines the configuration of hardware as described herein, which is used to design integrated circuits or to configure programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which is encoded a computer-readable program code in the form of an integrated circuit definition data set, which, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture an inference device or training apparatus configured to perform any of the methods described herein, or to manufacture an inference device or training apparatus including any of the devices described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.

[0221] Thus, a method of manufacturing an inference apparatus or training device as described herein at an integrated circuit manufacturing system may be provided. In addition, an integrated circuit definition data set may be provided that, when processed in an integrated circuit manufacturing system, enables the method of manufacturing an NNA or training device to be performed.

[0222] The integrated circuit definition data set may be in the form of computer code, for example as a netlist, code for configuring a programmable chip, as a hardware description language that defines hardware suitable for fabrication at any level in an integrated circuit, including as register transfer level (RTL) code, as a high-level circuit representation such as Verilog or VHDL, and as a low-level circuit representation such as OASIS (RTM) and GDSII. The higher-level representation that logically defines hardware suitable for fabrication in an integrated circuit (e.g., RTL) may be processed at a computer system configured to generate a manufacturing definition of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements to generate a manufacturing definition of the integrated circuit so defined by the representation. As is typically the case with software executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate a manufacturing definition of the integrated circuit to execute code that defines the integrated circuit to generate the manufacturing definition of the integrated circuit.

[0223] Now refer to Fig.13 An example of processing an integrated circuit definition dataset at an integrated circuit manufacturing system to configure the system to manufacture an inference device or training apparatus is described.

[0224] Fig.13An example of an integrated circuit (IC) manufacturing system 1302 is shown, which is configured to manufacture an inference device or training device as described in any of the examples in this document. In particular, the IC manufacturing system 1302 includes a layout processing system 1304 and an integrated circuit generation system 1306. The IC manufacturing system 1302 is configured to receive an IC definition data set (e.g., defining an inference device or training device as described in any of the examples in this document), process the IC definition data set, and generate an IC based on the IC definition data set (e.g., the IC includes an inference device as described in any of the examples in this document). Processing of the IC definition data set configures the IC manufacturing system 1302 to manufacture an integrated circuit that includes an inference device or training device as described in any of the examples in this document.

[0225] The layout processing system 1304 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of a circuit to be generated, such as in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining location information for the logic components, the circuit layout may be determined by the gate-level representation of the circuit. This may be done automatically or with user involvement in order to optimize the circuit layout. When the layout processing system 1304 has determined the circuit layout, it may output the circuit layout definition to the IC generation system 1306. The circuit layout definition may be, for example, a circuit layout description.

[0226] As is known in the art, IC generation system 1306 generates an IC according to the circuit layout definition. For example, IC generation system 1306 can implement a semiconductor device manufacturing process for generating an IC, which can involve a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition can be in the form of a mask that can be used in a photolithography process for generating an IC according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1306 can be in the form of a computer readable code that IC generation system 1006 can use to form a suitable mask for generating an IC.

[0227] The different processes performed by IC manufacturing system 1302 may all be performed at one location, such as by one party. Alternatively, IC manufacturing system 1302 may be a distributed system such that some processes may be performed at different locations and by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0228] In other examples, processing of an integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture an NNA or training device without processing the IC definition data set to determine a circuit layout. For example, an integrated circuit definition data set may define a configuration of a reconfigurable processor (e.g., an FPGA), and processing of the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0229] In some embodiments, when processed in an integrated circuit manufacturing system, the integrated circuit manufacturing definition data set can cause the integrated circuit manufacturing system to produce a device as described herein. For example, the integrated circuit manufacturing definition data set is described above with reference to Fig.13 An integrated circuit manufacturing system configured in the manner described may be used to manufacture the apparatus described herein.

[0230] In some examples, the integrated circuit definition data set may include software that runs on, or in combination with, the hardware defined at the data set. Fig.13 In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined at the integrated circuit definition dataset when manufacturing the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0231] The implementation of the concepts described in this application in devices, equipment, modules and / or systems (and in the methods implemented herein) can improve performance compared to known implementations. The performance improvement can include one or more of improved computing performance, increased throughput, reduced execution time and / or reduced power consumption. During the manufacture of such devices, devices, modules and systems (e.g., in integrated circuits), a trade-off can be made between performance improvement and physical implementation to improve the manufacturing method. For example, a trade-off can be made between performance improvement and layout area to match the performance of known implementations, but using less silicon. For example, this can be done by reusing functional blocks in a serial manner or sharing functional blocks between elements of the device, device, module and / or system. In contrast, the concepts described in this application that cause improvements in the physical implementation of devices, devices, modules and systems (e.g., reduced silicon area) can be weighed against performance improvement. This can be done, for example, by manufacturing multiple instances of a module within a predefined area budget. (It should be understood that the ISP can perform other aesthetic functions that are not strictly related to image reconstruction, such as "oversharpening" an image to make it look sharp.

[0232] Applicants hereby independently disclose each individual feature described herein and any combination of two or more such features, to the extent that such feature or combination can be implemented based on the specification as a whole in view of the common general knowledge of a person skilled in the art, regardless of whether such feature or combination of features solves any problem disclosed herein. In view of the foregoing description, it will be clear to a person skilled in the art that various modifications can be made within the scope of the invention.

Claims

1. A differentiable module of a differentiable model of an image signal processor, the image signal processor comprising a pipeline of functional blocks, wherein the differentiable module is configured to implement a single functional block of the pipeline, the differentiable module comprising: a base logic configured to receive an input image signal and process the received input image signal by executing a base image processing function representing a task of a functional block of the pipeline implemented by the module; a refinement function configured to receive the input image signal and process the received input image signal in parallel with the base logic processing the received input image signal; as well as Combinatorial logic configured to combine the processed image signal from the base logic and the processed image signal from the refinement function to determine an output image signal to be output from the differentiable module.

2. The differentiable module of claim 1 , wherein the differentiable module is representable in a command stream as a combination of operations from a set of elementary neural network operations available on an inference device for implementation on the inference device.

3. The differentiable module of claim 2, wherein the set of elementary neural network operations consists of one or more of: Convolution operation; Pooling operation; Element-wise operations; Activation operation; Local response normalization operation; Tensor rescaling operations; Channel permutation operations; and Integer operations.

4. A differentiable module according to claim 2 or 3, wherein the inference device is a neural network accelerator.

5. A differentiable module according to claim 1, 2 or 3, wherein the underlying image processing function that the logic of the differentiable module is configured to perform is a function that when applied to an image signal refines the image signal in one aspect.

6. The differentiable module of claim 5, wherein the refinement function is configured to supplement the refinement performed by the base image processing function.

7. The differentiable module of claim 6 , wherein in supplementing the refinement performed by the base image processing function, the refinement function is configured to process the input image signal in a manner such that the combinatorial logic corrects errors remaining in the processed image signal after the base logic has processed the received input image signal.

8. A differentiable module according to claim 1, 2 or 3, wherein the refinement of the image signal performed by the refinement function is small compared to the refinement of the image signal performed by the base logic.

9. The differentiable module of claim 1 , 2 or 3, wherein the differentiable module is any one of a demosaicing module, a sharpener module, a black level subtraction module, a spatial denoiser module, a global tone mapping module, a channel gain module, an automatic white balance or a color correction module.

10. An inference device for image processing, the inference device being configured to implement a command stream that represents a differentiable model of an image signal processor as a combination of operations from a set of elementary neural network operations available on the inference device, the image signal processor having a pipeline of two or more functional blocks, the differentiable model of the image signal processor comprising: At least two differentiable modules, each of the at least two differentiable modules is configured according to claim 1, 2 or 3.

11. A method of processing an image signal using an inference device, the inference device being configured to implement a command stream representing a differentiable model of an image signal processor, the image signal processor having a pipeline of functional blocks, wherein the model of the image signal processor includes a differentiable module configured to implement a single functional block of the pipeline, the method comprising: receiving an input image signal at the differentiable module; processing a received input image signal by executing the underlying image processing functions representing the tasks of the functional blocks of the pipeline implemented by the modules; processing the received input image signal using thinning in parallel with said processing the received input image signal by performing the basic image processing function; combining a processed image signal from performing the base image processing function and a processed image signal from using the refinement function to determine an output image signal; as well as The determined output image signal is output from the differentiable module.

12. The method of claim 11, wherein the command stream represents the differentiable model of the image signal processor as a combination of operations from a set of elementary neural network operations available on the inference device.

13. The method of claim 12, wherein the set of elementary neural network operations consists of one or more of the following: Convolution operation; Pooling operation; Element-wise operations; Activation operation; Local response normalization operation; Tensor rescaling operations; Channel permutation operations; Integer operations; cascade; Reduction operations, including summing, averaging, minimization, and maximization.

14. The method of claim 12 or 13, wherein the inference device is a neural network accelerator.

15. The method according to any one of claims 11, 12 or 13, wherein said processing the received input image signal by executing the basic image processing function refines the received input image signal in one aspect.

16. The method of claim 15, wherein said processing of the received input image signal using said refinement function complements refinement performed by said base image processing function.

17. The method of claim 16, wherein said processing of the received input image signal using said refinement function complements refinement performed by said base image processing function such that said combination corrects errors in the processed image signal from performing said base image processing function.

18. A method according to any one of claims 11, 12 or 13, wherein the refinement of the image signal performed by processing the received input image signal using the refinement function is small compared to the refinement of the image signal performed by processing the received input image signal by executing the basic image processing function.

19. The method according to claim 11, 12 or 13, wherein the task of the functional block of the pipeline implemented by the module is any one of demosaicing, sharpening, black level subtraction, spatial denoising, global tone mapping, channel gain application, automatic white balance or color correction.

20. An integrated circuit definition data set which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture the inference apparatus of claim 10.

Images