Image reconstruction and processing

Abstract

Image Reconstruction and Processing This description concerns a method for learning a neural network (202) comprising: - the generation of a modified image, by a modified data generator, based on a first image, the modified image including at least one pixel value that is an aberrant with respect to the first image; - the provision of the modified image to the network; - the generation, by the network, of a corrected image; - the provision of the corrected image, by the network, and the provision of an indication of the position of at least one modified pixel, by the generator, to a computing circuit (210); - the generation of an error value, based on the application of a cost function, by the computing circuit, taking as input the first image, the indication, and the corrected image; - the correction of parameters associated with the network by backpropagation of the error in the network. Figure for the abstract: Fig. 2

Description

Title of the invention: Image reconstruction and processing technical field

[0001] This description relates generally to methods and devices for image reconstruction and processing and in particular, to methods and devices for image demosaicing. Previous technique

[0002] Raw images acquired by image sensors, such as, for example, sensors dedicated to visible and / or infrared imaging, exhibit alterations due to sensor malfunctions. In particular, image sensors implementing canonical imaging systems, using unit pixel matrixing methods, may exhibit, during their lifetime, malfunctions on individual pixels, or on columns of pixels, and / or loss of captured data for individual pixels or columns of pixels.

[0003] It is desirable to improve the processing and reconstruction of images acquired by sensors including malfunctioning pixels. Summary of the invention

[0004] One embodiment provides a method for learning a neural network configured to perform image processing operations, the method comprising: - the generation of a modified image, by a modified data generator, on the basis of a first image, the modified image including at least one pixel value that is aberrant compared to the first image; - the provision of the modified image to the network; - the generation, by the network, of a corrected image; - the provision of the corrected image, by the network, and the provision of an indication of the position of at least one modified pixel, by the generator, to a computing circuit - the generation of an error value, based on the application of a cost function, by the calculation circuit, taking as input the first image, the indication and the corrected image; - correction of network-related parameters by backpropagation of the error in the network.

[0005] According to one embodiment, the generation of the corrected image by the network comprises: - the generation, by a first sub-module, of an intermediate data value on the basis of the modified image, the intermediate data being at least a partial reconstruction of the first image; - the generation, by a second sub-module of the network, of an attention data on the basis of one of the intermediate data and / or the modified image, the attention data including an estimate of the location of at least one modified pixel; - the generation of the corrected image, by the first sub-circuit or by a third sub-circuit, on the basis of the intermediate data and the indication data.

[0006] According to one embodiment, the generation of the attention data includes the execution of a convolutional neural network configured for image segmentation, based on the intermediate data and / or the corrected image.

[0007] According to one embodiment, the convolutional neural network is a U-net type network or a variant of a U-net type network.

[0008] According to one embodiment, the generation of attention data further includes NL1 norm type normalization, based on the data generated by the convolutional neural network.

[0009] According to one embodiment, the generation of the corrected image includes a multiplexing operation and / or a multiplication operation, for example point-to-point multiplication, based on the attention data and the intermediate data.

[0010] According to one embodiment, the generation of an error value includes the application of a focused cost function taking, as input data, the modified image, the first image and the location indication of at least one modified pixel.

[0011] According to one embodiment, the generation of an error value further includes the application: - a cost function based on an average calculation taking, as input data, the modified image and the first image; and / or - of a regularization function taking, as input data, the modified image.

[0012] According to one embodiment, the first image is included in a database.

[0013] According to one embodiment, the generation of the modified image comprises, for each pixel: - determining whether the pixel needs to be modified; and - if the pixel is to be modified, the value associated with the pixel is replaced by an outlier value.

[0014] According to one embodiment, the generation of the modified image, by the modified data generator, is further carried out on the basis of a matrixing pattern.

[0015] According to one embodiment, the modified data generator model is a neural network model previously trained in image generation, modified and configured for the implementation of so-called "style transfer" techniques.

[0016] One embodiment provides an image processing method comprising: - the capture of an image scene by an imager of an image processing device, the captured image being a matrixed image according to a matrixing pattern - the provision of the matrixed image to a neural network of the processing device trained according to the training method described above; and - the generation of a corrected image, by the network, based on the rasterized image.

[0017] According to one embodiment, the generation of the corrected image by the network comprises: - the provision of the matrixed image to a first sub-module of the network configured to generate intermediate data by performing a first dematrixing of the matrixed image; - the provision of intermediate data to a second network sub-module configured to generate attention data based on the intermediate data, the attention data including estimates of the location of malfunctioning pixels of the imager; and - the generation of the corrected image, by a third sub-module, based on the intermediate data and the attention data.

[0018] According to one embodiment, the matrix pattern corresponds to a matrix pattern used during the generation of a modified image during network training.

[0019] One embodiment provides an image processing device comprising: - an imager configured to capture image scenes, according to a matrixing pattern; - a neural network trained according to the above training method, configured to generate a corrected image based on the matrixed image.

[0020] According to one embodiment, the imager includes one or more malfunctioning pixels. Brief description of the drawings

[0021] These features and advantages, as well as others, will be described in detail in the following description of particular embodiments, given by way of non-limiting example, in relation to the accompanying figures, among which:

[0022] Fig. 1 represents examples of matrixing patterns implemented by image sensors;

[0023] [Fig.2] is a block diagram illustrating a learning architecture of a neural network, according to an embodiment of the present description;

[0024] [Fig.3] is a block diagram illustrating a network architecture configured to perform inference operations, according to one embodiment of the present description;

[0025] [Fig.4] is an example of the architecture of a processing network, according to an embodiment of the present description;

[0026] [Fig. 5] is an example of the architecture of an attention module, according to an embodiment of the present description; and

[0027] [Fig.6] is an example of the architecture of a processing network, according to an embodiment of the present description. Description of the implementation methods

[0028] The same elements have been designated by the same reference numerals in the different figures. In particular, structural and / or functional elements common to the different embodiments may have the same reference numerals and may have identical structural, dimensional and material properties.

[0029] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and detailed. In particular, the operation and implementation of different types of neural network layers, such as dense layers, convolutional layers, etc., are not described in detail and are known to those skilled in the art.

[0030] Unless otherwise specified, when referring to two elements connected together, this means directly connected without intermediate elements other than conductors, and when referring to two elements connected (in English "coupled") together, this means that these two elements can be connected or linked through one or more other elements.

[0031] In the following description, when reference is made to absolute position qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers, such as the terms "above", "below", "superior", "inferior", etc., or to orientation qualifiers, such as the terms "horizontal", "vertical", etc., reference is made, unless otherwise specified, to the orientation of the figures.

[0032] Unless otherwise specified, the expressions "approximately", "roughly", and "in the order of" mean within 10%, preferably within 5%.

[0033] Fig. 1 represents examples of matrixing patterns implemented by image sensors, or imagers.

[0034] A matrix pattern is a pattern comprising several pixels and acting as a spectral filter replicated, in mosaic, on the surface of an imager. The imager then measures, for each pixel in the mosaic, a value associated with a channel defined by the matrix pattern. The matrix pattern indicates, for each pixel, which single channel among, for example, color and / or infrared channels is being measured. By way of example, each pixel has a spectral signature defining a wavelength range encompassing the spectral response recorded by the pixel.

[0035] In the example in Figure 1, a matrix pattern 100 is a Bayer pattern. This pattern is 2x2 pixels in size and allows the measurement of the wavelengths associated with the color green (G) for the pixels in the upper left and lower right corners of the pattern. The Bayer pattern also allows the measurement of the wavelengths associated with the color red (R) for the pixel in the upper right corner, as well as the measurement of the wavelengths associated with the color blue (B) for the pixel in the lower left corner of the pattern.

[0036] A matrix pattern 102 is a QuadBayer pattern. This pattern is 4x4 pixels in size and allows the measurement of wavelengths associated with the color green (G) for the four pixels in the upper left and lower right corners of the pattern. The pattern 102 also allows the measurement of wavelengths associated with the color red (R) for the pixels in the upper right corner, as well as the measurement of wavelengths associated with the color blue (B) for the four pixels in the lower left corner of the pattern.

[0037] A pattern 104 is a 3-cell Bayer type pattern. This pattern is 6x6 pixels in size and has the same structure as patterns 100 and 102 except that the red, green and blue color channels are measured in groups of 3 x 3 pixels.

[0038] A 108 pattern is a 2x2 RGBIR pattern. This pattern is 2x2 pixels in size and allows the measurement of the wavelengths associated with the color green for the pixel in the top left corner of the pattern. The 108 pattern also allows the measurement of the wavelengths associated with the color blue for the pixel in the top right corner, as well as the measurement of the wavelengths associated with the color red for the pixel in the bottom left corner of the pattern. The 108 pattern further allows the measurement of the wavelengths associated with infrared (IR) for the pixel in the bottom right corner.

[0039] A pattern 110 is a 4x4 RGBIR pattern. This pattern is 4x4 pixels in size and allows the measurement of the wavelengths associated with the green color of the first pixel from the left on the first row, the third pixel from the left on the third row, the first and second pixels from the left on the second row, and the third and fourth pixels from the left on the fourth row. Pattern 110 also allows the measurement of the wavelengths associated with infrared of the second and fourth pixels from the left of the first and third rows.Pattern 110 also allows the measurement of wavelengths associated with the blue color of the first pixel from the left of the third row and of the first and second pixels from the left of the fourth row, as well as the measurement of wavelengths associated with the red color of the third pixel from the left of the first row and of the third and fourth pixels from the left of the second row.

[0040] These examples of die patterns are given by way of example and are of course not limiting. Other die patterns, of different sizes and arrangements, are, of course, conceivable. In other examples, the replicated die pattern The imager's surface contains a pixel configured to measure the distance between one or more targets in the captured scene and the imager. Such imagers are color image sensors that also capture depth information; they are then referred to as monolithic RGBZ imagers.

[0041] A raw image captured by the imager includes, for example, a single measured value for each pixel. Demosaicing then consists of estimating, for each pixel, the value of the unmeasured channels using, for example, the spectral correlation between the measured channels as well as the spatial correlation between the value of the same channel for two adjacent pixels.

[0042] However, it sometimes happens that certain pixels malfunction. For example, one or more isolated pixels become saturated or unusable. This is then referred to as a dead pixel. For example, the so-called Dark signal, or dark current, representing a type of noise in the absence of measured light flux for isolated pixels, deviates from the nominal statistical range, and the measurements associated with these pixels are then aberrant. Other sources of noise can cause aberrant behavior on a pixel or a group of pixels. For example, some isolated pixels are subject to random telegraphy noise (RTS). For example, following a malfunction in an imager readout circuit, columns of pixels are missing or clamped.In other examples, a digital defect, such as a loss of synchronization in measurements or a bit flip, results in a loss of data associated with individual pixels and / or columns and / or rows of pixels and / or one or more pixel frames. In the following, any malfunctioning pixel, whether isolated or within a malfunctioning column, row, or frame, will be referred to as a "bad pixel." Measurements associated with bad pixels are then aberrant and do not accurately reflect reality.

[0043] Furthermore, bad pixels will, for example, affect the estimation of adjacent pixels during demosaicing. For example, the final image, obtained after demosaicing that does not correct bad pixels, will, for example, show black spots larger than the bad pixel or group of bad pixels.

[0044] However, aberrant measurements associated with poor pixels are relatively infrequent. Indeed, the aforementioned malfunctions generally have a low occurrence, on the order of 1 / 1000 or less. However, in imagers with a very large number of pixels, these malfunctions can be significant with respect to visual rendering and image perception. For example, these malfunctions affect image rendering much more noticeably than disorders due to homoscedastic noise whose distribution is centered around zero.

[0045] When a deep neural network is trained to dematrix raw images, outliers associated with bad pixels are masked by other noise due to their infrequent occurrence. Furthermore, classical cost functions, based primarily on averaging calculations and applied canonically regardless of pixel position, fail to identify and correct this type of outlier.

[0046] Figure 2 is a block diagram illustrating a learning architecture for a deep neural network, according to an embodiment of the present description. In particular, the learning of the neural network described in relation to Figure 2 allows for the consideration of outliers, for example due to bad pixels, in image reconstruction.

[0047] According to one embodiment, an augmentation module 200 (DATA AUGM) allows, for example, the execution of a data augmentation model. For example, the module 200 is configured to receive, as input data, a Y_true image corresponding to ground truth, that is, an image containing no errors resulting from any processing. For example, each pixel of the Y_true image includes the ground truth values ​​of several channels. The module 200 is further configured to generate a modified Y_bad image based on the Y_true image. The modified Y_bad image then corresponds to the Y_true image in a matrix format, to which one or more outliers have been added.

[0048] By way of example, module 200 is configured to model noises such as pixel read noise and column read noise, photonic noise, fixed spatial noise, for example structured noise, telegraphic noise, and dead or saturated pixels.

[0049] By way of example, module 200 is configured to add an outlier to each pixel with a probability of at most 1 / 1000. In other words, module 200 performs a Bernoulli trial, pixel by pixel, with parameter p, where p is at most 1 / 1000. When the Bernoulli trial is successful, module 200 is configured to modify the associated pixel value towards an outlier. When the trial fails, in most cases, the pixel value remains the same as that of the Y_true image. In another example, when the trial fails, one or more noise values ​​are added to the pixel value. For example, the outlier is a constant value. In another example, the outlier is obtained randomly for each pixel concerned. The outlier is, for example, added to the measured value for that pixel. In another example, the outlier value is multiplied by the measured value for said pixel.In yet another example, the outlier value is substituted for the measured value for that pixel. In another example, for each pixel in question, several outliers are determined, randomly or not, and the measured value of the pixel is replaced by a combination of these. minus an addition of an outlier with the measured value and / or a multiplication of an outlier with the measured value.

[0050] By way of example, the generation of the outlier is carried out according to a strongly spiked probability distribution, that is to say having a high moment of order 4, such as for example, a Laplace law, a Cauchy law, etc.

[0051] The methods and circuits for generating such a modified image are known to those skilled in the art and the given example of generation is of course not limiting.

[0052] Module 200 is further configured to generate Y_aug data, for example in the form of a tensor, including indications of the positions of the modified pixels. As an example, the Y_aug data also includes an indication of the outliers that have been assigned.

[0053] By way of example, the field images, provided to module 200, are included in a pre-recorded database, for example in non-volatile memory, such as a server, etc.

[0054] Module 200 is configured, for example, to provide the modified image Y_bad to a neural network 202 for training. For example, network 202 is configured to perform regression tasks to reconstruct and dematrix a provided matrixed image, taking into account any bad pixels. For example, network parameters, such as weights, are initially set to initial values.

[0055] According to one embodiment, the matrixing pattern used by the module 200 to matrix the Y_true image corresponds to the matrixing pattern that will be used by an image sensor of a device comprising the driven grating 202.

[0056] In another embodiment, the module 200 includes a neural network previously trained to generate images, such as, for example, raster images. By way of example, image generation is carried out using so-called "style transfer" techniques, such as, for example, the techniques described in the publication "Unsupervised Image-to-Image Translation: A review" by Hoyer, H. et al. and published in Sensors in 2022 and / or in the popular science article on CycleGANs "A Gentle Introduction to CycleGAN for Image Translation" written by Brownlee, J. on August 17, 2017.

[0057] The network 202 includes, for example, a processing submodule 204 (FIRST DEMOSAICING MOD) configured to generate intermediate data Y_tmp, based on the modified image Y_bad. The intermediate data Y_tmp corresponds, for example, to a first demosaicing and reconstruction of the image. By way of example, module 204 is a convolutional network integrating several neural layers.

[0058] The network 202 includes, for example, in addition, an attention sub-module 206 (ATTENTION MOD) configured to receive the intermediate data Y_tmp, or a A sub-part of this image is provided by module 204 and / or the modified image Y_bad is provided by augmentation module 200. Submodule 206 is further configured to generate attention data Y_pwa, which may, for example, be in the form of a tensor. Y_pwa is generated by submodule 206 to carry information about the modified image pixel by pixel. In particular, submodule 206 is designed to first perform pixel-by-pixel image segmentation to determine, pixel by pixel, the type of demosaicing to be performed. Submodule 206 is further configured to generate Y_pwa to include information about the type of processing to be performed pixel by pixel, such as gating.Masking is, for example, performed on the Y_tmp tensor data using the Y_pwa signal in order to weight each type of reconstruction present in the Y_tmp tensor according to the Y_pwa tensor.

[0059] The network 202 further includes, for example, a refinement submodule 208 (REFINEMENT MOD). By way of example, submodule 208 is configured to generate a corrected image Y_pred based on the intermediate data Y_tmp provided by submodule 204 and the attention data Y_pwa provided by submodule 206.

[0060] In another example, submodule 208 is configured to perform masking of Y_tmp in the channel axis, based on the data Y_pwa.

[0061] During the training of network 202, the corrected image Y_pred is provided to a calculation circuit 210 (LOSS). The calculation circuit 210 is further configured to receive the ground image Y_true and the Y_aug data including indications of the positions where the outliers were added by module 200.

[0062] As an example, the data Y_aug is a tensor with values ​​in {0, 1}, pixels assigned the value 1 indicate, for example, the positions of the bad pixels.

[0063] According to one embodiment, the calculation circuit 210 is configured to calculate an error value Err by applying a cost function (or "loss function") to the modified image Y_pred, the ground image Y_true, and the data Y_aug. By way of example, the cost function applied by the calculation circuit 210 combines several fidelity cost functions, such as a cost function based on average error calculations without priors and applied between the data Y_pred and Y_true, or a focused cost function (or "Focal Loss Function" - LFF) applied to the data Y_pred, Y_true, and Y_aug, for which the focus is, for example, controlled by the Y_aug tensor. By way of example, the cost function applied by the calculation circuit 210 further includes a regularization function applied only to the corrected image Y_pred.As an example, the regularization function is determined from assumptions about the nature of the signal of the corrected image Y_pred.

[0064] By way of example, the error value is such that Err - LF M(Y _pred, Y _true) +^uf LF F {Y _pred, Y_true, Y_aug) +ÀlK x LR(Y _pred) where the LFM function is a cost function based on mean error calculations, for example, based on a mean squared error calculation, the LFF function is a focused cost function, and the LR function is a regularization function, and where Rlfm and ^lr are weighting coefficients. For example, the coefficients ^lff and ^lr are used to emphasize the importance of one cost function relative to another.

[0065] By way of example, the mean error LFM{ Y_pred, Y_true) is such that: [Math 1] LFM( Y_pred, Y_true) = |CNN( Y^pred) - CNN( Y_true) | + Â2^\CNN(Y_pred) - CNN( Y_mui) |2, where CNN(.) is an intermediate output of a neural network model that has been previously trained to perform certain inference tasks. For example, CNN(.) has been trained in a supervised manner to feed a submodel dedicated to image classification tasks. For example, the neural network relies on a latent space from a neural network model trained to perform classification operations on databases, such as the ImageNET database. For example, the CNN(.) function is a vgg(.) function where vgg(-) is a function returning a latent space, for example of type vggl6, and the coefficients Xj and X2 are weighting coefficients between L1 and L2 distance calculations. In particular, the summation is performed over all elements, that is, over all pixels and all channels.

[0066] By way of example, the focused error LFF( Y_pred, Y_truc, Y_aug^) is such that: [Math 2] LFF(Y_pred, Y_tme, Y_aug) = E((dilate^Y_aug))O(Y_pred-Y_true)y, where dilate^) defines a morphological dilation function, for example, having a 5x5 circular kernel, and where the operator O corresponds to point-by-point multiplication. In particular, the dilate^) function allows increasing the impact radius of the cost function in the vicinity of aberrant pixels, thus making it easier to correct the effects due to the large receptive fields of the convolutional neural network used to reconstruct the image.

[0067] By way of example, the regularity error LR{ Y_pred} is such that: [Math 3] LR{ Y pred) — TV( Y_pred), where TV(Y_pred) corresponds, for example, to the calculation of the total variation, or to a variant of the total variation. The use of the total variation as a regularization function is given as an example and is of course not li- mitative.

[0068] According to one embodiment, during the training of network 202, the error value obtained is backpropagated within network 202 in order to re-evaluate and update the parameters associated with network 202. By way of example, the parameter update is performed by implementing a gradient backpropagation method based on the error value. Deep learning methods based on gradient backpropagation are known to those skilled in the art and are therefore not described in further detail here.

[0069] According to one embodiment, the focused cost function serves to indicate to the network 202, during backpropagation of the error, which pixels are highly noisy. For example, the coefficient ^lff is greater than the coefficient ^lfm-

[0070] Fig. 3 is a block diagram illustrating a network architecture 202 configured to perform regression operations, according to an embodiment of the present description.

[0071] By way of example, once trained, the 202 network is implemented in an image processing device. By way of example, the processing device further includes an imager (not shown) configured to capture image scenes based on a matrix pattern such as, for example, one of the patterns 100, 102, 104, 108, or 110 described in relation to [Fig. 1]. By way of example, the imager includes one or more bad pixels, or malfunctioning pixels. By way of example, the device is a camera, a camcorder, a smartphone, etc.

[0072] Once the training of the 202 network is complete, that is, after the training described in relation to [Fig. 2] has been performed on a plurality of ground truth images Y_true, the 202 network is, for example, configured to perform regression operations on an input image Y_in, replacing the Y_bad data, which may include one or more bad pixels. As an example, the training is completed after processing a large number, for example, at least 1000 ground truth images. In another example, the training ends when the learning algorithm converges, that is, when the error value falls below a threshold value.

[0073] The regression operation, performed by the trained network 202, consists of generating a dematrixed and reconstructed image Y_out based on a matrixed input image Y_in, provided, for example, by the imager. The trained network 202 then includes, for example, sub-modules 204, 206, and 208, whose parameters are fixed following training. The matrixed image Y_in is then provided directly to the network 202, without passing through the augmentation module 200. Similarly, the dematrixed and reconstructed image Y_out is not provided to the computing circuit 210. The augmentation module 200 and the computing circuit 210 are, for example, not part of the processing device in which the driven 202 network is implemented.

[0074] The trained network 202 is then configured, for example, to perform regression, interpolation, denoising, and detection and correction of bad pixels detected in the input image Y_in, and to generate the corrected image Y_out based on these operations. In particular, following training, the processing sub-module 204 of the network 202 is configured, for example, to generate intermediate data Y_tmp by performing a first estimation of the missing or noisy information, due at least in part to the bad pixels.

[0075] Following training, submodule 206 is configured, for example, to generate an attention data point Y_pwa, based on the input image Y_in and / or the intermediate data point Y_tmp. Submodule 206 is then configured, for example, to detect, individually for each pixel, whether that pixel is a bad pixel or not. The success rate of bad pixel detection depends, of course, on the training and the level of expressiveness, linked to its internal structure, of the 202 network. Submodule 206 then generates a data point Y_pwa including information on the detected aberrations as well as their location in the input image Y_in. As an example, the detection is multi-scale detection, and the Y_pwa data point provides context, for example, a category indication among, among others, a type of contour, texture, presence and level of noise, and positions of the aberrant pixels, pixel by pixel.

[0076] Following training, submodule 208 is configured, for example, to generate the demosaiced image Y_out based on the intermediate data Y_tmp and the attention data Y_pwa. As an example, submodule 208 is configured to correct areas identified by the Y_pwa data as being outside the main statistic, that is, areas containing one or more bad pixels.

[0077] Fig. 4 is an example of the architecture of the processing submodule 204 according to an embodiment of the present description.

[0078] By way of example, submodule 204 comprises a plurality of subnetworks 400. By way of example, submodule 204 comprises a number Nd, Nd being an integer greater than 2, preferably greater than or equal to 5 and less than or equal to 12, of subnetworks 400.

[0079] By way of example, each 400 subnetwork is configured to receive the input image Y_in and to generate, each, an intermediate data Y_tmpi. The Nd intermediate data Y_tmpi are then concatenated, for example in the channel axis or in an additional axis.

[0080] Each 400 subnetwork comprises, for example, a plurality of layers. By way of example, the input image Y_in is provided to a 402 convolution layer (F- CONV KxK). As an example, layer 402 is a two-dimensional convolution layer with F output channels, F being an integer greater than or equal to the number of pixels in the matrix pattern, and comprising kernels of size K x K, where K is an integer strictly greater than the width and / or height of the matrix pattern.

[0081] By way of example, the output generated by layer 402 is provided to layer 404 (MOSAIC2CHANNEL). Layer 404 is a pixel shuffle layer configured to spatially rearrange the channels of the provided data. In particular, layer 404 is configured to perform a space-to-depth operation. Furthermore, the operation performed is parameterized by the size of the matrix pattern considered.

[0082] By way of example, the output generated by layer 404 is provided to a layer 406 (DW CONV 1x1). By way of example, layer 406 is a depth-wise convolution layer comprising 1x1 kernels, thus allowing each channel to be weighted independently.

[0083] By way of example, the output generated by layer 406 is provided to a layer 408 (MOSAIC2CHANNEL). By way of example, layer 408 is a pixel blending layer, for example similar to layer 404.

[0084] By way of example, the output generated by layer 408 is provided to a layer 410 (DW CONV KxK). By way of example, layer 410 is a depth-convolutional layer comprising kernels of size KxK.

[0085] By way of example, the output generated by layer 410 is provided to a layer 412 (MOSAIC2CHANNEL). By way of example, layer 412 is a pixel blending layer, for example similar to layers 404 and 408.

[0086] By way of example, the output generated by layer 412 is provided to a layer 414 (DW CONV 1x1) similar to layer 406.

[0087] By way of example, the output generated by layer 414 is provided to a layer 416 (CHANNEL2MOSAIC), referred to as a pixel blending layer. In particular, layer 416 is configured to perform a "depth to space" operation. Furthermore, the operation performed is parameterized by the size of the matrix pattern considered.

[0088] By way of example, the output generated by layer 416 is provided to a layer 418 (1 - CONV KxK). By way of example, layer 418 is a two-dimensional convolution layer with a single output channel and comprising kernels of size KX K.

[0089] By way of example, a 420 concatenation aggregation layer (CONCAT) is configured to perform a concatenation of the output data from layers 418 and 408. By way of example, the output data from layer 408 is provided to layer 420 via a so-called "connection hop" branch (in (English "skip connection"). The branch is then configured to provide, for example, ¾ of the output channels of layer 408 directly to layer 420.

[0090] The concatenation generated by layer 420 is, for example, provided to a convolutional layer 422 (3 - CONV KxK). For example, layer 422 is a two-dimensional, three-channel output convolutional layer comprising kernels of size K x K. For example, the three output channels are the three RGB color channels. Layer 422 is then configured to generate the output data Y_tmpi, corresponding, for example, to a first demosaicing and reconstruction of the input image.

[0091] As an example, the data Y_pwa is injected into the network model 206 in order to calculate all or part of the intermediate data Y_tmp.

[0092] According to one embodiment, the parameters of the subnetwork 400 are learned during the learning of the network 202 described in relation to [Fig.2].

[0093] Fig. 5 is an example of the architecture of the attention module 206, according to an embodiment of the present description.

[0094] Since the task of submodule 206 is to detect, pixel by pixel, the bad pixels, an architecture used for image segmentation is suitable for the implementation of submodule 206.

[0095] By way of example, submodule 206 then includes a U-net type 500 network (U-NET). In other examples, the 500 network is a type of network other than the U-net configured to perform picture-to-picture operations. By way of example, the 500 network is a variant and / or an improvement of a U-net type network. By way of example, the 500 network is further configured to implement internal attention mechanisms and / or to manipulate 2- or 3-dimensional data. By way of example, the 500 network further includes a dense interconnection structure.

[0096] By way of example, the 500 network is a U-net type network with a typical depth of 4 and comprising 5x5 receptive field convolution blocks with ReLU activation. In one example, the U-net 500 network is based on convolution blocks comprising two 3x3 convolutions per group, each followed by channel mixing. This example is specific and allows for limiting the complexity of the 500 network and applying 5x5 receptive fields. A person skilled in the art will be able to adapt and modulate the receptive field sizes as well as the number of convolution groups. By way of example, the 500 network is also configured to perform a subsampling operation, such as a so-called "stride" operation, for example, implemented by a so-called MaxPooling layer. As an example, subsampling is subsampling by 2, corresponding to a 2x2 MaxPooling operation.As an example, the 500 network also includes normalization stages to limit the impact of the absolute amplitude of the . Data on inference operations. In another example, when the absolute amplitude of the data is a factor in detecting bad pixels, the 500 network includes at least one non-normalized path. Connection skips are implemented through an aggregation operation, such as concatenation. For example, an upsampling operation is performed by a convolution with a step size of 2.

[0097] As an example, the 502 network is configured to provide the data it generates to a 504 layer (L1 NORM). The 504 layer is configured to normalize the received value, for example, by applying channel-axis normalization, denoted NL1. Applying NL1 normalization ensures that the sum of the Nd intermediate data points is equal to 1. As an example, NL1 normalization is applied via an activation function of type softmax or softargmax. Indeed, the function softargmax(.) is equal to NL1(exp(.)). The argmax function therefore also returns an output whose sum of elements is equal to 1. Thus, the different reconstructions of the intermediate data Y_tmp are summed in a normalized manner.As an example, layer 504 is configured to receive data in the form of a tensor T, where I and J are, respectively, the number of rows and columns of pixels in the image, and C is the number of channels of the tensor. For example, for modulus 204, the values ​​C and Nd are equal. Denoting the value of the tensor T at position [i,j] in the image and for a channel c, T[i,j,c], layer 504 is, for example, configured to normalize, for each position, the tensor performing the transformation: . [Math 1] Furthermore, the NL1 standardization is applied to positive data, resulting from a 500 network activation function.

[0098] In another example, in order to avoid a potential division by 0, the L1 norm of the tensor, at position [i,j] is defined such that [Math 2] VL1(T[UC]) =r[î-,y,c] / (EX1|7'[ / . / c]| + where* is a real number, strictly greater than 0.

[0099] In yet another example, the NL1 normalization of the tensor, at position [i,j] is defined such that NL1(T[i,j,c]) - 0 or NL1(T[z,j,c]) - 1 / C, if -HAS . 2V£l(T[z, j, c] ) = T[i, j, c] / £.|T[U, c] | 500 segmentation network.

[0101] The [Fig.6] is an example of the architecture of the refinement network 508, according to an embodiment of the present description. ​

[0102] As described in relation to Figures 2 and 3, submodule 208 is configured to generate the output Y_pred during the training of network 202, or Y_out during the execution of the trained network 202. The input data is then used to infer the output data. The data Y_pwa is, for example, used to select the Y_tmpi outputs of the 600 subnets. The data Y_pwa is, for example, also used to indicate the type of processing, for example masking, to be performed on the Y_tmp data resulting from the initial demosaicing and reconstruction, performed by submodule 204.

[0103] By way of example, submodule 208 includes a layer 600 (MULT) configured to perform a multiplication operation, for example point-by-point, of the tensor Y_tmp based on the values ​​of the attention tensor Y_pwa. In another example, layer 600 is configured to perform a multiplexing operation of the tensor Y_tmp based on the values ​​of the attention tensor Y_pwa. In the example where layer 600 is configured to perform a multiplexing operation, an activation function of module 206 is of type argmax. The output of layer 600 is then, for example, provided to a dense layer 602. The dense layer 602 is then configured to generate an output L, based on the data provided by layer 600.As an example, the output Y] corresponds to a list of D linear combinations of the outputs of layer 600, thus generating a tensor of size WXH x Nc XD, where W is the width in pixels of the image, H the height in pixels of the image, and where Nc refers, for example, to the three color channels and where D is the dimension in axis 4 of the tensor. The value of D depends, for example, on the topology, as well as on the hyperparameters of the 202 network.

[0104] Submodule 208 further includes a 604 convolution layer (CONV 2D) configured to perform a convolution operation on the Y_pwa tensor. The result of the 604 layer is then provided to a 606 layer (HARDSIGMOID). By way of example, the 606 layer is configured to apply an activation function, for example a HardSigmoid function, to the received data.

[0105] Layer 606 is then configured to provide its output to a layer 608 (RESHAPE). As an example, layer 608 is configured to resize the data provided by layer 606 into a Y2 tensor of size W x H x Nc x D.

[0106] By way of example, the tensor Y2 is provided to a layer 610 (x<-lx) configured to generate a tensor F3 by inverting, with respect to 1, the values ​​of the tensor Y2. The tensor Y2 and the data Y] are further provided to a layer 612 (MULT) similar to the layer 600. By way of example, the layer 612 is configured to generate a data E4 based on data Yj and Y2.

[0107] The Y4 data is provided to a 616 layer (CNN-CORR). The 616 layer is, for example, a convolution layer, or a cascade of convolution layers, configured to generate Y5 data of the defects detected using the Y_pwa tensor. Thus, the parts of the input image detected as requiring correction are smoothed.

[0108] Layer 616 is further configured to provide the ^5 data to a layer 618 (MULT). Layer 618 is further configured to receive the K3 data, provided by layer 610, and to generate Y6 data by performing an operation similar to that performed by layers 600, 612, and 614.

[0109] Submodule 208 further includes a layer 620 (ADD). As an example, layer 620 is an additional aggregation layer. Layer 620 is configured, for example, to receive the Y4 data, the output data from layer 618, and to sum them.

[0110] Submodule 208 further includes a layer 622 (CNN-COM) configured to generate the output data Y_pred or Y_out based on the output of layer 620. By way of example, layer 622 is a convolutional layer, or a cascade of convolutional layers, configured to format the output data. By way of example, the formatting includes one or more rotations and / or color adjustments and / or smoothing and / or shifting operations, etc.

[0111] According to one embodiment, the use of the activation function, of the HardSigmoid type, allows on / off type detection of bad pixels and to perform an operation similar to multiplexing using multiplicative aggregation layers and an additional aggregation layer.

[0112] Various embodiments and variants have been described. Those skilled in the art will understand that certain features of these various embodiments and variants could be combined, and other variants will be apparent to those skilled in the art. In particular, variants are possible with regard to the types of cost functions combined with the focused cost function. Similarly, the design, as well as the topology, of modules 204, 206, and 208 may vary.

[0113] Finally, the practical implementation of the described embodiments and variants is within the reach of a person skilled in the art, based on the functional specifications given above. In particular, the practical implementation of a desired type of imager is within the reach of a person skilled in the art.

Claims

Demands

1. A method for learning a neural network (202) configured to perform image processing operations, the method comprising: - the generation of a modified image (Y_bad), by a modified data generator (200), on the basis of a first image (Y_true), the modified image comprising at least one pixel value that is aberrant with respect to the first image; - the provision of the modified image to the network; - the generation, by the network, of a corrected image (Y_pred); - the provision of the corrected image, by the network, and the provision of an indication (Y_aug) of the position of at least one modified pixel, by the generator, to a computing circuit (210); - the generation of an error value, on the basis of the application of a cost function, by the computing circuit, taking as input the first image, the indication and the corrected image; - correction of network-related parameters by backpropagation of the error in the network.

2. A method according to claim 1, wherein the generation of the corrected image (Y_pred) by the network (202) comprises: - the generation, by a first sub-module (204), of an intermediate data value (Y_tmp) on the basis of the modified image (Y_bad), the intermediate data being at least a partial reconstruction of the first image; - the generation, by a second sub-module (206) of the network (202), of an attention data (Y_pwa) on the basis of one of the intermediate data and / or the modified image, the attention data comprising a location estimate of at least one modified pixel; - the generation of the corrected image (Y_pred), by the first sub-circuit or by a third sub-circuit (208), on the basis of the intermediate data and the indication data.

3. A method according to claim 2, wherein the generation of the attention data (Y_pwa) comprises the execution of a convolutional neural network configured for image segmentation (500), based on the intermediate data (Y_tmp) and / or the corrected image (Y_bad).

4. A method according to claim 3, wherein the convolutional neural network (500) is a U-net type network or a variant of a U-net type network.

5. A method according to claim 3 or 4, wherein the generation of the attention data (Y_pwa) further includes NL1 type normalization, based on the data generated by the convolutional neural network.

6. A method according to any one of claims 2 to 5, wherein the generation of the corrected image includes a multiplexing operation and / or a multiplication operation, for example point-to-point multiplication, based on the attention data (Y_pwa) and the intermediate data (Y_tmp).

7. A method according to any one of claims 1 to 6, wherein the generation of an error value includes the application of a focused cost function (LFF) taking, as input data, the modified image (Y_pred), the first image (Y_true) and the location indication of the at least one modified pixel (Y_aug).

8. A method according to claim 7, wherein the generation of an error value further comprises the application of: - a cost function based on an average calculation (LFM) taking, as input data, the modified image (Y_pred) and the first image (Y_true); and / or - a regularization function (LR) taking, as input data, the modified image (Y_pred).

9. A method according to any one of claims 1 to 8, wherein the first image is included in a database.

10. A method according to any one of claims 1 to 9, wherein the generation of the modified image comprises, for each pixel: - determining whether the pixel is to be modified; and - if the pixel is to be modified, replacing the value associated with the pixel with an outlier value.

11. A method according to any one of claims 1 to 10, wherein the generation of the modified image (Y_bad), by the modified data generator (200), is further carried out on the basis of a matrixing pattern.

12. A method according to any one of claims 1 to 11, wherein the modified data generator model (200) is a modified image generation neural network model previously trained and configured for the implementation of so-called "style transfer" techniques.

13. Image processing method comprising: - the capture of an image scene, by an imager of an image processing device, the captured image being a matrixed image according to a matrixing pattern - the provision of the matrixed image to a neural network of the processing device trained according to the training method according to any one of claims 1 to 12; and - the generation of a corrected image, by the network, on the basis of the matrixed image.

14. Image processing method according to claim 13, wherein the generation of the corrected image by the network comprises: - supplying the rasterized image to a first sub-module (204) of the network (202) configured to generate intermediate data (Y_tmp) by performing a first demosaicing of the rasterized image; - supplying the intermediate data to a second sub-module of the network (206) configured to generate attention data (Y_pwa) on the basis of the intermediate data, the attention data comprising estimates of the location of malfunctioning pixels of the imager; and - generating the corrected image, by a third sub-module (208), on the basis of the intermediate data and the attention data.

15. Image processing method according to claim 13 or 14, wherein the matrix pattern corresponds to a matrix pattern used when generating a modified image during network training (202).

16. Image processing device comprising: - an imager configured to capture image scenes, according to a matrix pattern; - a neural network (202) trained according to the training method according to any one of claims 1 to 12, configured to generate a corrected image on the basis of the matrixed image.

17. Image processing device according to claim 16 wherein the imager comprises one or more malfunctioning pixels.

Images