Dematting process and device

Abstract

Demosaicing Method and Device This description concerns a method for demosaicing an image, by a processor of an electronic device, from measurements, performed by an imager and according to a matrixing pattern, of at least one channel. The method comprises: - the determination, by the processor, for each window of a plurality of windows covering the image, of a covariance matrix from a set of covariance matrices, by performing a shift convolution operation on the entire image with a filter bank associated with the covariance matrix; - for each window, the calculation, based on the covariance matrix, of an estimator comprising estimates of at least one channel, the estimator further comprising an estimate of an average value of at least one channel over the entire window; and - the generation of a new image based on the estimator of each window. Figure for the abstract: Fig. 6

Description

Title of the invention: Dematrixing method and device Technical field

[0001] The present description relates generally to the field of imagers, and more particularly to image dematrixing methods. Prior art

[0002] Imagers are often formed of pixels of several types, configured to capture different channels. However, each pixel is generally sensitive to only one channel, and for each pixel in the raw image, captured by the imager, information from one or more other channels is therefore missing. For example, color imagers use color filters on each of the pixels separating the trichotomy information in the light. Thus, for each pixel, only one channel is measured, such as the red or blue or green channel, and each pixel therefore has only an incomplete measurement of the color. In other examples, a channel other than color, such as depth, may be measured for a pixel.

[0003] In order to reconstruct a complete image having information from one or more channels for each pixel, the values ​​of the channels not measured by certain pixels can then be estimated, in an operation called dematrixing and from the measurements carried out on the neighboring pixels.

[0004] Existing demosaicing methods are generally slow to execute and / or consume processing resources. Other existing demosaicing methods may be relatively fast, but are adapted to imagers having a specific filter configuration and are not adaptable to other configurations. Summary of the invention

[0005] One embodiment provides a method for dematrixing an image, by a processor of an electronic device, from measurements, carried out by an imager and according to a matrixing pattern, of at least one channel, among a first set of channels, the method comprising: - determining, by the processor, for each window of a plurality of windows covering the image, a covariance matrix, from a set of covariance matrices stored in a memory of the device, by performing, for each covariance matrix, a shift convolution operation on the entire image with a filter bank associated with the covariance matrix; - for each window, the calculation, on the basis of the determined covariance matrix, of an estimator comprising estimates of values ​​of at least one channel, among a second set of channels, for each pixel of the window, the estimator further comprising an estimate of an average value of the at least one channel across the entire window; and - the generation of a new image based on the estimator of each window.

[0006] According to one embodiment, the covariance matrices of the set of covariance matrices come from a Gaussian mixture model.

[0007] According to one embodiment, the windows are selected so that the matrixing pattern is identical from one window to another.

[0008] According to one embodiment, the calculation of the estimator is further carried out on the basis of a matrixing matrix dependent on the matrixing pattern.

[0009] According to one embodiment, each window is distant from its neighboring windows by a clipping step and the shift convolution operation is a step convolution, the step being equal to the clipping step of the image, the step being a multiple of the size of the matrix pattern.

[0010] According to one embodiment, for each window, the estimator x is calculated by making the matrix product: [Math 1] X = °ù m is a vector comprising the measured values ​​for the window, is an orthonormal matrix whose columns are the eigenvectors of a matrix Æ and Aj. is a diagonal matrix comprising the square roots of the eigenvalues ​​of the matrix the matrix A;. is equal to + ÿ,M being the matrix of matrixing, R being the covariance matrix associated with the measurement noise of the imager and <9 being the covariance matrix of the prior distribution of the estimator and including a covariance submatrix îz which is the determined covariance matrix, and where F is a matrix equal to MT ii

[0011] According to one embodiment, for each covariance matrix Ej, among the set of covariance matrices, the associated filter bank is a sequence of matrices originating from the elements of the matrix F^.

[0012] According to one embodiment, the above method further comprises, before determining the covariance matrix, and for each covariance matrix, calculating and storing in the memory the matrices Az, U; and Fi.

[0013] According to one embodiment, the above method further comprises, following the calculation of the estimators, the reconstruction of an image by doing: - for each pixel of each window, calculating a value of the pixel by adding the estimated mean of the at least one channel, among the second set of channels, to the estimate of the value of the at least one channel.

[0014] According to one embodiment, the above method further comprises, for each pixel covered by at least two separate windows, calculating the average of the pixel values.

[0015] According to one embodiment, the at least one channel, among the second set of channels, is spectrally correlated to the at least one measured channel, among the first set of channels.

[0016] According to one embodiment, the first and second sets of channels comprise the red, green, blue color channels.

[0017] According to one embodiment, the first set of channels further comprises a depth channel.

[0018] One embodiment provides an electronic device comprising: - a non-volatile memory configured to store a set of covariance matrices and a set of filter banks, each filter bank being associated with a covariance matrix; and - a processor configured for: determine a covariance matrix for windows covering a raw image by doing: for each covariance matrix, a shift convolution operation, from a set of measurements associated with the windows of the filter bank associated with the covariance matrix; and for each window, calculating an estimator, based on the determined covariance matrix, the estimator comprising estimates of values ​​of at least one channel, from a second set of channels, for each pixel of the window, the estimator further comprising an estimate of an average value of the at least one channel over the entire window; and generate a new image based on the estimator of each window.

[0019] According to one embodiment, the memory further comprises sets of parameters, each set being dependent on the set of covariance matrices, and wherein the processor is configured to calculate the estimator based on the sets of parameters depending on the determined covariance matrix. Brief description of the drawings

[0020] These characteristics and advantages, as well as others, will be explained in detail in the following description of particular embodiments given without limitation in relation to the attached figures among which:

[0021] [Fig.l] illustrates an example of demosaicing multiple acquired windows based on a matrixing pattern;

[0022] [Fig.2] is a block diagram showing a processing device according to one embodiment of the present description;

[0023] [Fig.3A] is an example of a matrix matrix associated with a matrix pattern;

[0024] [Fig.3B] is an example of another matrix matrix associated with another matrix pattern;

[0025] [Fig.4] illustrates operations for estimating a covariance matrix adapted to a window;

[0026] [Fig.5] illustrates an operation of reconstructing an image from the estimated covariance matrices; and

[0027] [Fig.6] is a flowchart representing different stages of image processing. Description of the embodiments

[0028] The same elements have been designated by the same references in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same references and may have identical structural, dimensional and material properties.

[0029] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and are detailed. In particular, the operation of the imagers is not described in detail. The methods and devices for measuring depth or a color channel are known to those skilled in the art.

[0030] Unless otherwise specified, when referring to two elements connected to each other, this means directly connected without intermediate elements other than conductors, and when referring to two elements connected (in English "coupled") to each other, this means that these two elements can be connected or be connected by means of one or more other elements.

[0031] In the following description, when reference is made to absolute position qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers, such as the terms "above", "below", "upper", "lower", etc., or to orientation qualifiers, such as the terms "horizontal", "vertical", etc., reference is made unless otherwise specified to the orientation of the figures.

[0032] Unless otherwise specified, the expressions "about", "approximately", "substantially", and "of the order of" mean to within 10%, preferably to within 5%.

[0033] Figure 1 illustrates an example of demosaicing several windows of an acquired image on the basis of a matrixing pattern. In particular, Figure 1 illustrates a raw image 100. By way of example, the image 100 is an image 10 pixels wide and 10 pixels high and therefore comprises [Q2 pixels. The raw image 100, also called mosaic, is for example obtained by an imager, such as for example color imagers (RGB) or color-depth imagers (RGBZ), and from a scene image. A filter configuration, or matrix pattern, 106 is for example replicated on the surface of the imager. The imager then measures, for each pixel of the mosaic, a value associated with a channel, defined by the matrix pattern 106. The matrix pattern indicates for example which single channel, for example among red (r), green (g), blue (b), is captured by each pixel. For example, the matrix pattern includes other filters than the red, green and blue color filters such as for example a filter measuring a depth channel (z). Thus, for certain pixels only a depth measurement is carried out.

[0034] In the example of Figure 1, the matrix pattern is a Bayer type pattern. This pattern is 2x2 pixels in size and allows the measurement of the wavelengths associated with the green color for the pixels at the top left and bottom right of the pattern. The Bayer pattern also allows the measurement of the wavelengths associated with the red color of the pixel at the top right as well as the measurement of the wavelengths associated with the blue color at the bottom left of the pattern. In general, by assuming the convention that the pixel at the top left of the raw image 100 is in position (1,1), the Bayer pattern allows the measurement of the green channel associated with the pixel at position (i,j) where i and j are both even or odd, of the red channel of the pixels at position (i,j) where i is odd and j is even and of the blue channel of the pixels at position (i,j) where i is even and j is odd.

[0035] Each pixel of the raw image 100 therefore comprises a measurement of the value of a single channel, measured according to the matrixing pattern used.

[0036] The demosaicing of the raw image 100, that is to say the estimation of the values ​​of other channels, for example the estimation of the color components of each pixel, is carried out window by window. A window is a sub-part of the raw image 100 of size Mj x w2 0ans in the example illustrated by FIG. 1, two windows 102 and 104, of size 5x5 are represented. The division of the raw image 100 into a plurality of windows is called matrixing. By way of example, a sub-part 108 of demosaiced image corresponds to an image resulting from the demosaicing of the windows 102 and 104.

[0037] The raw image 100 is then cut into a plurality of windows and according to a cutting step s, representing the number of pixels to be skipped, vertically or horizontally, between two successive windows. For example, the cutting step between the windows 102 and 104 is 2 pixels. For example, the cutting step is the same for the entire image, whether vertically or horizontally.

[0038] The windows 102 and 104 are, for example, neighboring windows. Generally, two successive windows are neighbors. The distance between two neighboring windows is equal to the clipping pitch. For example, the two windows 102 and 104 are 2 pixels apart from each other.

[0039] For example, each window is rectangular in shape. The window comprises a number K'i of pixel rows and a number w2 of pixel columns. When wi ~ w2, the window is square and we will note wi ~ w2 ~ vv'_ In yet another example, each window has a shape other than a square or a rectangle, for example an L shape, provided that all the windows cover the entire raw image 100. In other words, whatever the shape of the windows, the arrangement of the windows is for example chosen so that any pixel of the raw image 100 is covered by at least one window...

[0040] In the example of Figure 1, the window 102 covers the 25 pixels common to the first 5 columns and first 5 rows of the image 100, and the window 104 covers the 25 pixels common to the first 5 rows and to the 5 columns going from the 3rd column to the 7th column of the image 100. In the example of Figure 1, the two windows 102 and 104 overlap by a width of 3 pixels and therefore comprise 15 pixels in common. The clipping step being equal to 2, the window 102 therefore comprises a band of width of 2 pixels which are not in common with the window 104. For example, for windows of size by w?2 pixels, the clipping step j varies between 1 pixel and min( n'2) Pixels so that all 'cs pixels of the raw image 100 are covered by at least one window.

[0041] When the clipping step is strictly less than and at ^2 of the pixels of the raw image 100 are covered by a number of windows, which may be greater than two. Indeed, in the example illustrated by [Fig.l], the pixels of the 5th column of the window 102 are also covered by a third window (not shown) covering the pixels common to the first 5 lines and from the 5th to the 9th column of the image 100.

[0042] Other matrix patterns can of course be used, and in particular patterns whose depth channel is measured for certain pixels can be used. By way of example, matrix patterns for which the same channel is measured for all the pixels can also be used. These examples of matrix patterns are given by way of example and are of course not limiting.

[0043] In the example of [Fig.l], the clipping step between two successive windows is chosen so that each window is measured according to the same filter configuration. Indeed, if the clipping step were for example an odd integer, the green component of the top left pixel would be measured for half of the windows and the red component would be measured for the other half. Generally speaking, when the step is a multiple of the size of the matrix pattern, each window has the same filter configuration.

[0044] Demosaicing consists of estimating, for each pixel of each window, the value of the unmeasured channels using the spectral correlation between the measured channels as well as the spatial correlation between the value of the same channel for two adjacent pixels. For example, the color channels have, in fact, a spectral correlation between them. For each pixel, the value of the measured channel is also for example estimated. Indeed, the measured value can be noisy. The estimation process takes this measurement noise into account and the estimation of a measured channel acts as denoising.

[0045] During demosaicing, the depth channel is not, for example, reconstructed from the color channels. Indeed, the information from the depth channel risks polluting the color estimation. Indeed, depth measurements have particular noise statistics, and in particular depth measurements are generally noisier than color measurements.

[0046] In another example, another channel, not measured for at least some pixels, but having a spatial correlation with the measured channels, is reconstructed without having been measured. As an example, a channel representing the temperature is reconstructed.

[0047] For example, when a pixel is covered by several windows, the final estimate of the value of a channel corresponds to the average of the channel estimates over all the windows containing the pixel in question. Thus, the smaller the clipping step, the more precise the final estimate of the channel value.

[0048] By way of example, the color associated with each pixel of the image portion 108 is for example an RGB value obtained by estimating the value of the three color channels.

[0049] [Fig.2] is a block diagram showing an electronic device 200 according to an embodiment of the present description.

[0050] By way of example, the electronic device 200 comprises one or more imagers 202 (IMAGE SENSOR(S)), and in this case, the electronic device 200 is a digital camera, a smartphone comprising a photographic functionality, etc. In another example, the one or more imagers 202 are not part of the electronic device 200, and in this case the device 200 is for example an image processing device, such as a computer.

[0051] Each imager 202 is configured to measure, according to a matrix pattern, the channel value for the pixels in order to generate an image. For example, when the measurements are carried out by several imagers not having the same optical axis, the measurements taken are for example replaced in the same spatial grid. Each pixel of the grid can then have several measurements, for example taken by different imagers, or no measurements. In order to replace the measurements on this spatial grid, the images captured by all the imagers are, for example, to be readjusted. In another example, the replacement of the measurements on the grid is carried out by a calibration of an informing system, for example the processor 210, of the acquisitions of each imager with respect to the spatial grid. The methods making it possible to go from several imagers, not necessarily having the same optical axis, to a single raw image are known to those skilled in the art.For example, the imager(s) 202 com- .

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] further take one or more time-of-flight sensors configured to measure distances at which objects are located in the captured image scene. In other examples, the imager(s) 202 include temperature sensors and / or infrared sensors. The imager(s) 202 are for example connected to an integrated circuit 204, via a bus 205. The circuit 204 comprises, for example, a non-volatile memory 206 (NV MEM) and a volatile memory 208 (RAM), for example a random access memory (in English “Random Memory Access”). The circuit 204 further comprises a processor 210 (CPU), for example connected to the memories 206 and 208 via a bus 212. For example, non-volatile memory 206 includes instructions for implementing a demosaicing algorithm. For example, the imager(s) 202 are configured to store the measurements made in the volatile memory 208 or in the non-volatile memory 206 and the processor 210 is configured to execute the instructions stored in the non-volatile memory 206 and / or in the memory 208 in order to reconstruct an image from the measurements stored in the volatile memory 208. Once reconstructed, the image is for example stored in the non-volatile memory 206. For example, the raw image 100, captured by the imager(s) 202, is stored in the non-volatile memory 206. Following the matrixing of the raw image 100, for example carried out by the processor 210, a set of matrices, corresponding to the measurements of the raw image 100 per window, is stored in the non-volatile memory 208. As an example, the matrix pattern used by the imager(s) 202 is a so-called Bayer pattern 300, illustrated at the top of [Fig.3A], which is 2x2 pixels in size, the pixels comprising filters such that the pixels are respectively sensitive, in the direction of the needle, to red (r), green (g), blue (b) and green (g) colored light. For example, windows are also 2x2 pixels in size. The matrix m containing the measurements for a window takes the form of a matrix of, for example, size 2x2 where m = 11 ^12 + and where the elements i+ K? and 822 + 9? are ^21 + M / 7 <?22 + K?.

[0059] respectively the green channel measurements of the pixels at position (1,1) and (2,2) in the window. The ri2+ element is the red channel measurement of the pixel at position (1,2) of the window and the b2i+ element is the blue channel measurement of the pixel at position (2,1) of the window. According to one embodiment, the values ​​^r, IL and correspond respectively to the average values ​​of red, green and blue channels on the patch. Thus, in this For example, the values ​​i, ? 12, b2i and 822 are then centered values.

[0060] In another example, the measure m takes the form of a column vector, for example of the form P + ' • r12+^. ^21+¾ ,^22 + h?,

[0061] In yet another example, the measurement m is acquired according to a matrixing pattern comprising, for example, the measurement of the depth z channel. For example, the measurement 822 + Eg is replaced by the measurement ”22 of the depth of the pixel at position (2,2) in the window. No color component is therefore measured for the pixels at position (2,2) of each window.

[0062] The purpose of demosaicing is to estimate, for each pixel of each window, the values ​​of n channels, where 11 is equal to at least 1. For example, the number n is equal to 3 and the 3 estimated channels are the green, red, blue channels. In another example, in addition to, or instead of one or more of the color channels, other channels, having a spatial correlation with the measured channels, are estimated.

[0063] Figure 3A is an example of a matrix matrix M associated with a matrix pattern. In particular, the matrix pattern 300 illustrated in Figure 3A is the Bayer pattern and each window is of size w2 = 2x2.

[0064] According to the embodiment of Figure 3A, the measurement m for the window takes the form of a column vector of size wj2 = 4 as described in relation to Figure 2. The vector m is then equal to the product of the dematrixing matrix M, of size w2 = 4 by 3w2 + 3 = 15, with a column vector x of size 3w2 + 3 = 15.

[0065] According to one embodiment, the vector x concatenates respectively the red, green and blue values ​​of the pixels of the window as well as a vector h — [ 11 11 11 ] comprising the average values ​​for each of the red, green and blue channels respectively over the window. The vector x is unknown and the objective of the demosaicing operation is to estimate it.

[0066] The dematrixing matrix M depends entirely on the matrixing pattern used and expresses the relationship between the vector to be estimated x and the vector of measurements m.

[0067] According to one embodiment, the cutting step of the raw image 100 is a multiple of the size of the matrix pattern used, for example a multiple of 2 in the case of a Bayer pattern. Thus the matrix matrix M established as a function of the matrix pattern is the same for each window of the raw image.

[0068] Figure 3B is an example of another matrix matrix M associated with another matrix pattern 302. In the following, we assume the convention that the pixel in position (i,j) of a matrix pattern corresponds to the pixel placed on the i-th line and on the j- th column of the pattern, the first row being the top row of the pattern, and the first column being the column to the left of the pattern.

[0069] In particular, the matrix pattern 302 is of size 3x2, i.e. wq — 3 and w2 = 2. In this example, the channel measured for the pixel in position (1,2) of the pattern 302 is for example the red color (r), the channel measured for the pixels in positions (2,1) and (3,2) of the pattern 302 is for example the blue color (b), the channel measured for the pixel in position (2,2) of the pattern 302 is for example the green color (g), the channel measured for the pixels in positions (1,1) and (3,1) of the pattern 302 is for example the depth channel (z).

[0070] For example, measured depth values ​​are replaced by default by zero values ​​in the measure vector m. In the example in Figure 3B, the measure m takes the form of a column vector, for example of the form '0' ri2 + ^. ^ 22 + ^ 0 ?32 + M è , OR ^32 + 9 / , is the measure of the blue component of the pixel at position (3,2).

[0071] The vector x is of size 3 ( t-vp-is 4-1 ) =21 and then comprises a concatenation of the red, green and blue channels for each pixel with the vector M comprising the averages of each channel over the measured window. In particular, the vector x includes the red, green, and blue components for all pixels, including pixels for which depth was measured.

[0072] The dematrixing matrix M is then adapted so that the matrix-vector product Mx is equal to the measure vector m, and thus the first and fifth rows of the matrix M contain only 0s. The matrix 302 is of size Hqw2 = 6 by «(^^2+ 0 = 21-

[0073] Other die patterns than the patterns 300 and 302 of the examples of FIGS. 3A and 3B may of course be used. For example, patterns of different size and / or shape may be used.

[0074] According to one embodiment, the matrix M comprises rows, being the number of pixels making up each window, and "(wjHS +1) columns, n being the number of channels to be estimated, for example 3 in the case of a color reconstruction. The person skilled in the art will know how to adapt the matrix M to the pattern used.

[0075] Although, in the examples described in relation to Figures 3A and 3B, the size of the windows is identical to the size of the matrix patterns, it is of course possible, as described in the example in relation to [Fig.l], to choose windows of a different size than the size of the matrix pattern.

[0076] According to one embodiment, the demosaicing operation, at the scale of a window, aims to estimate an estimator x of the column vector x comprising the centered components, for each pixel of the window, of the channels to be estimated as well as the means of the channels on the window. A Maximum Posterior Estimator for x is x - argmaxxp{n^)p{x}, where p(m] x) is the likelihood of the vector x, knowing the measurement m and p(x) is the a priori distribution of x. The likelihood of the vector x is deduced from the equation m - Mx + e relating the measurement m to the unknown vector A and where E is a random vector of size wiw2 and models noise, for example linked to the imager(s) 202. As an example, the random vector e follows a centered normal distribution and covariance matrix R. As an example, the matrix R is a diagonal matrix.

[0077] According to one embodiment, the a priori distribution of the vector x follows a normal law N( 0, Cov), where Cov is a covariance matrix of size n(H'pv2 + 1), where n is the number of channels to be estimated. In particular, [Math 2] where, E is a covariance submatrix of size nw}w2 x qUj encodes the spectral and spatial correlation between the different channels to be estimated with respect to the different pixels of the window. The matrix encodes the correlation between the mean values ​​of the n channels to be estimated and assumes that these are independent of each other and that each mean value follows a centered normal distribution with variance X2

[0078] The estimator x of the Maximum Posteriori of the vector A then forms an explicit analytic: [Math 3] $ = ( MtR'M + Covxy\MTR:^

[0079] According to one embodiment, the cutting step v is chosen so that the dematrixing matrix M is the same for any window, the product ^MTR"lM + Cov^ MT R:X only depends on the window considered by the covariance matrix Cov.

[0080] The measured data of the vector ftl are not previously centered, which is why the vector of means P is also estimated. In addition, the estimation of the vector h is also based on the a priori distribution of the vector x, and is therefore better than an empirical mean traditionally subtracted from the Cov = measured data.

[0081] Although the illustrated vector x includes the red, green and blue components for each pixel of a window, it is of course possible to adapt the vector x. For example, a desired estimate is a representation of another matrixing pattern, or in another example an estimate of the components of a single channel. The person skilled in the art will know how to adapt the vector x described to a variant xJ, for example by performing a matrix-vector product = M^, where Ms is a matrix allowing the transition from x to x"s. For example, if the desired vector £ includes only the red components of the pixels while the vector x includes the estimates of the red, green and blue components, as well as the estimate of the mean values, the matrix Ms has the form M — 0 t 1 ■ 0 ^1, the matrix identity being responsible for extracting the red channel and the column vector being responsible for adding the mean to the red channel.

[0082] For example, for each window, the covariance matrix L is determined from a set of covariance matrices, stored for example in the non-volatile memory 206, before any image capture. For example, the set of covariance matrices is associated with a mixture of multivariate normal distributions (GMM, from the English “Gaussian Multivariate Mixture”). The a priori distribution of the vector x is then equal to [Math 4] p ( x ) = ]LL aN ( 0 X ) °where is the cardinality of the set of matrices of co variance ■ ■ ■, For example, k is greater than or equal to 5 and less than or equal to 50. Each coefficient 1 4 i < L weights the importance of the normal covariance law £, in the GMM model used. For example, the coefficients 1 ​

[0083] The methods for calculating and constructing GMM models, and in particular GMM models adapted to color image processing, are known to those skilled in the art and are for example described in the publication “From Learning Models of Natural Image Patches to Whole Image Restoration” published in 2011 in the journal “International Conference on Computer Vision.” by Zoran, D. and Weiss, Y.

[0084] The calculation and construction of the GMM model used are for example carried out in a design step of the device 200, for example taking place before the manufacture of the device 200.

[0085] For each window, an index J of the covariance matrix, among the set of covariance matrices, the most suitable for the measured window is the index verifying: [Math 5] / = argmax p(n^}d^e moreover, for any index i, the log likelihood log(p(^n) ) is such that: [Math 6]

[0086] 2 ) - + / og|2 tt Ad| + ^{TjA^c, °where the operator | ■ | denotes the determinant and where Ai = MTRAM + Cor* and c = MTRim, CoVj denoting the covariance matrix of the prior law of x having as covariance submatrix the matrix E,. For each window, the index j selected is therefore the one maximizing the log likelihood and the covariance matrix adapted for the window considered is Ej.

[0087] The first part of the right-hand side of the log likelihood does not depend on the measured vector m and can therefore be precalculated before any image measurement. For example, for all indices 1 < i< L, the values ​​of the term 2 log(af j - logjE,] 4- log]2 ttA}1] are calculated upstream and for example stored in the non-volatile memory 206.

[0088] The first part of the right-hand side of the log likelihood log ( p ( fyn ) ) then does not depend on the measured vector m and can therefore be precalculated before any image measurement. For example, for all indices 1 of the values a, = 2 lo^ ) - lo^ + Z^2 tt AJ1) are precalculated upstream and for example stored in the non-volatile memory 206. For example, the values ​​ai, 1

[0089] According to one embodiment, the matrix Az is a real, symmetric and positive definite matrix and is therefore decomposable so that At = U, where is an orthonormal matrix of size (n( + H) by (n( + D) whose columns are the eigenvectors of the matrix Az and where the matrix A, is a diagonal matrix comprising the square roots of the eigenvalues ​​of the matrix A,-. Thus, the term (AA^c is rewritten in the form [Math 7]​ cTA^c = (Fitn ) T F im , avec = A; 1 uTr 1 .

[0090] The matrices F^ 1 < i < L, do not depend on the measurement vector m and are therefore precalculable when the dematrixing matrix M is identical for each measured window. For example, the matrices F 1 S i < F are calculated following the construction of the GMM model and the determination of the matrixing pattern used by the sensor(s) 202 and are stored in the non-volatile memory 206.

[0091] Thus, the estimator S of the maximum a posteriori of the vector x is written: [Math 8] x = U'-^F^n, °ù ; c is the index determined by maximizing the log likelihood log(p(n^i) ) and corresponds to the index of the covariance matrix best suited to the description of the window considered. In particular, the quantity iii does not depend on the measure vector m.

[0092] According to one embodiment, the processing of the raw image 100 comprises a first phase in which, for each window, the most suitable covariance matrix is ​​first determined, then in a second phase the maximum likelihood estimators of each window are estimated, from the measurement vectors and according to the formula y = IPA^Fm- iii

[0093] According to one embodiment, the first phase, of determining the covariance matrices, is carried out not window by window but globally. Indeed, the product of each vector measured with the matrices F, is rewritten in a convolution operation with step s, also called a shift convolution operation, of the entire raw image 100 by a filter bank constructed from the matrices F^.

[0094] Figure 4 illustrates covariance matrix estimation operations for each window. In particular, Figure 4 illustrates the convolution operation at step s on the raw image 100 making it possible to determine, for each window of the raw image 100, the covariance matrix most suitable for it.

[0095] Each matrix-vector product Ftm, for any I < / < L and any vector measure m, is a vector whose each coordinate is a scalar product between a line where 1 £ k < n( + 1), of the matrix F^ and the vector m. Thus, we have the equality: [Math 9] F jT Ffn= pjn ' where each / ik, 1 < k <n( W|W9 + 1 ), est un vecteur colonne T ij of size h'iw2 including the elements of the ^-th row of the matrix Fj.

[0096] Thus, considering the set P of windows covering the raw image, the set of products between the set of matrices F, with each vector measuring a window of the image, is rewritten: [Math 10] [pm, p| = |f., *SZ. 1< k< n(u'.vtL+l)| _ou / ^cstlc d-1 i< i < L 1 u J i< L arrangement of the column vector fik into a matrix MJ x w2, Z is the raw image recorded by the imager(s) 202 and where the operator ** is a convolution with step s, s being the cutting step described in relation to [Fig.l]. The result, for a given index i of the convolution with step 5 of the filter bank ( f \ with the image Z \ ifi / ] <k<rt ( ivtK'2+1 ) is a number n( + 1 ) of matrices of size P x ​​Q where PQ is the total number of windows. For example, the image Z is decomposed into PQ windows, of size par arranged in P rows and Q columns.

[0097] Thus, the determination, for each window, of the most suitable covariance matrix is ​​carried out from the raw image Z.

[0098] According to one embodiment, all of the filter banks are previously stored in the non-volatile memory 206. For example, the filter banks are pre-calculated from the matrices Fh. The non-volatile memory 206 therefore stores, for example in association of the indices i, a number L of filter banks.

[0099] According to one embodiment, following the measurements carried out by the imager(s) 202, the raw image Z is convolved with each filter bank fï 1 . For each index i, each stepwise convolution, for example executed by processor 210, results in 0 outputs of size P x ​​Q, and such that for each 1): [Math 11] Oik = / '.^.Z.For all 1 i L, the / 1(11^2+ 1) outputs of the filter bank associated with index i for a window at a position (p, q\ l <p < P et l<q < Q, dans l’image Z sont désignées dans la suite par £ / ].

[0100] For example, the processor 210 is a processor configured to perform convolution operations, such as for example a processor used for the execution of convolutional neural networks.

[0101] According to one embodiment, for each of the L convolutions by step of the raw image Z with the filter banks, a norm 1 < i < L, is calculated by doing: [Math 12] 2Y _ a + 11 q 1|2 where a> is the coefficient described in relation to [Fig.2]. Thus, each Nj is a matrix of size P x ​​4, calculated from the n ( W]W2 + 1 ) matrices 1 < k < «( + 1 )}• In particular, the element in position ( p, q'p noted [p, g], of a matrix Nj is associated with the window at position (p, q) in the raw image. For each window, we then obtain L values rtr ,2, l^Z - (MP'#]

[0102] According to one embodiment, an optimal index J is then determined for each window associated with a position (p, p) by determining the argument of the maximum on the values ​​of the indices of the quantities Nj[p, q], 1 ? 4L.

[0103] For example, a table 400 of size PXQ represents, for each window, the determined optimal index.

[0104] The convolution operations carried out at the scale of the raw image therefore make it possible to determine the covariance matrix of the GMM model most suited to each window.

[0105] [Fig.5] illustrates an operation of reconstructing an image from the covariance matrices estimated for each window. In particular, each optimal index included in table 400 corresponds to the index of the covariance matrix, among the set of covariance matrices of the GMM model.

[0106] From table 400, for each window at a position (p, q), an estimator of the maximum posteriori of the channel values ​​for each pixel of the window is then equal to: [Math 13] xpq = w[ P» Ç[]' °ù ppq) is the index of the covariance matrix determined for the window at position ( p, q ) in the raw image Z and m [ p, q ] is the vector measured for the window at position (p, q). The estimators X(pq) are then for example vectors of size n(wjHS + 1). For example, when the channels at estimate are the red, green and blue color components of the image, for any position (p,q), [Math 14] a L a ? aaa 1 where^j, respec-[r(p,q)' g(p^)' b(p,q 1' Pr(M)' Pg(M)' ]' tively g, .and L is a vector of size ^1^2 comprising the estimates of the red, green and blue components respectively, of each pixel of the window at position ( nq}. The value û , respectively p and p , is a scalar estimating the average of the red, green and blue channels respectively, on the window at position (p,q).

[0107] In the example illustrated by Figure 5, the raw image Z comprises for example 4 lines of each 6 windows, or 24 windows in total. A table 500 represents the 24 estimators x^q, where 1 4 p 4 4 and 1 4 q 4 6. As an example, table 400 indicates that the covariance matrix Sj is for example the one which has been determined as the most suitable for the windows in position (1,1), (1,2), (2,1), (4,5) and (4,6). The calculation of the estimators Xj jXj 2' -^2 b -^4 5 ct $4 6 fad therefore calls upon the matrices Ab A{ and F

[0108] According to one embodiment, a matrix 502, for example of size vvi xw 2, includes the values ​​of the vector fq j) to which are added the estimate A 504 matrix includes the values ​​of the vector £(1,Ù to which are added the estimate. A 506 matrix includes the values ​​of the vector to which the estimate is added. Matrices 502, 504 and 506 describe res-

[0109] respectively the estimated values ​​of the red, green and blue channel components for each pixel in the window at position (1,1). The values ​​of the red, green and blue channels are estimated in a similar way, for each pixel in each window at position ( / A ?), by adding the estimate of the mean p to the estimate pca , of the estimate of the mean at the estimate g, , and the estimate of the mean p to estimate M \IW) For example, a 508 image is obtained from the set of estimates £(^). For example, when a pixel of the image is covered by at least two windows, for example when the clipping step s is less than wi, the final estimate of each channel is the average of the estimates obtained for the pixel in question on the different windows covering it.

[0110] [Fig. 6] is a flowchart representing different steps of a method for processing an image. Unless otherwise specified, the steps of the method of [Fig. 6] are for example implemented by the electronic device 200 of [Fig. 2], and in particular by instructions executed by the processor 210.

[0111] In a step 600 (PRE COMPUTATION), the matrixing parameters are for example stored in the non-volatile memory 206. For example, an indication of the width H'i and the length w2 in pixels of the windows is stored in the non-volatile memory. For example, the width and the length are greater than or equal to 2 and less than or equal to the width and the length of the raw images captured by the imager. For example, the width vvi and the length of each window are values ​​between 2 and 15 pixels, and more particularly between 4 and 9 pixels.

[0112] For example, the value of the cutting step s is further stored in the non-volatile memory 206 when performing step 600. The value of the cutting step 5 is greater than or equal to 1 and is a multiple of the size of the matrix pattern used, for example a multiple of 2 in the case of a Bayer pattern. This condition allows the configuration of the filters to be identical for each window and thus the matrix matrix M to be identical for each window.

[0113] By way of example, step 600 further comprises the storage of the covariance matrix R, modeling the correlation of the noise produced by the imager(s) 202. The matrix matrix M is further chosen, depending on the matrix pattern used and the size of the windows, and stored in memory 206 during step 600.

[0114] Step 600 further includes the choice of the dimension L of the GMM model used, as well as the calculation of the GMM model. The covariance matrix set East then stored in non-volatile memory 206.

[0115] The matrices filter banks {ÿæ 1 < / c < + 1)}^ as well as the coefficients ( , are calculated, for example by the processor 210, or by a calculation unit external to the device 200, from the matrices R and 1 < i £ L and are for example stored in the non-volatile memory 206.

[0116] In a step 601 (IMAGE ACQUISITION), a raw image is acquired according to the matrix pattern. For example, the image is acquired by the imager(s) 202. In another example, the image is acquired by a device external to the device 200 and configured to communicate the image to the device 200. The raw image is then matrixed according to the window size parameters wi and w2 and clipping step s, by example stored in memory 208. The measurements associated with each window of the image are then for example stored in the memory 208 or in the non-volatile memory 206. For example, the raw image is cut into a number PQ of windows, which may overlap and for example P lines of each Q windows.

[0117] In a step 602 (i=l), an initialization of a variable i, corresponding to the indices of the covariance matrices R l , is for example carried out. As for example, the Ab matrices the filter bank \ f VW that the coefficient are temporarily stored in the volatile memory 208.

[0118] In a step 603 (CONVOLUTION OF Z WITH f ), the image acquired in step I i 601, and more particularly, the set of measured windows, is convolved, by a step convolution operation ' for example executed by the processor 210, with the filter bank |fj < £ < n ( Wpv2 + 1 )}• Following the convolution operation, a set of data each composed of +1) matrices of sizes PXQ , is for example temporarily stored in the memory 208.

[0119] In a step 604 (Ni COMPUTATION), a matrix Nt of size P x ​​Q, described in relation to FIG. 4, is calculated from the matrices O, and the coefficient ai. The matrix N, is for example stored in the memory 208. As an example, the matrices O, are deleted from the volatile memory 208.

[0120] In a step 605 (i=L?), it is determined, for example by the processor 210, whether all the index values ​​have been scanned, for example by comparing the value of current index i to the value L. If this is not the case (branch N), the method continues in a step 606 (i+1) in which another index i is selected. For example, step 605 consists of incrementing the current value of the index i. The method then resumes in a new implementation of steps 603 to 605, resulting in the calculation and storage of a new matrix Np

[0121] When, during an execution of step 605, it is determined that all the indices have been scanned, for example if i = F the matrices Np 1 < i <L. ont été calculées pour toutes les matrices de covariance du modèle GMM. Le procédé continue alors dans une étape 607 Q DETERMINATION).

[0122] During step 607, the optimal index for each window is determined for each window associated with a position (p, q) by calculating the argument of the maximum on the values ​​of the indices of the quantities A^fp, q] ■ The index of the covariance matrix best suited to the window in position (1,1) is therefore determined by calculating the argument of the maximum on the values ​​A^[ 1,1], AA[ 1,1], •••, NL[ 1,1]- The determination of the covariance matrices best suited to the other windows is carried out in a similar manner, by calculating the maximum argument on the elements of the corresponding matrices A^.

[0123] In a step 608 (PATCH ESTIMATIONS), the values ​​of the channels to be estimated are estimated, for each window, from the covariance matrices determined in step 607. For any window in position (p, q), a vector XM = LG A; F, wfP' Q] is calculated, for example by the processor 210. As a for example, if the channels to be estimated are the three color channels red, green and blue, u, • Once the vector xp^ is obtained, the values of the n channels are determined for each pixel by adding the channel mean to the estimate of its centered value for the pixel in question. For example, the values ​​of the red component for each pixel in the window at position (p, q) are the elements of the vector r„f / + li 1 1 r rpjq

[0124] As an example, following the performance of step 608, the components of the channels of the pixels being covered by several windows are estimated several times.

[0125] In a step 609 (IMAGE RECONSTRUCTION), an image, for example a color image, is reconstructed from the components of the channels of each pixel calculated in step 607. For example, a color image is reconstructed from the estimates of the red, green and blue channels. An RGB value is for example calculated for each pixel of the reconstructed image.

[0126] According to one embodiment, in the case where a pixel is covered by several windows and for which several estimates of the same channels have been calculated in

[0127]

[0128] step 607, final estimates are calculated by, for example, averaging, per channel, each estimate. In another embodiment, the cutting step ' is not a multiple of the size of the matrixing pattern, denoted in the following by f, and in this case the matrixing matrix M is not the same for each window. Nevertheless, by making several matrixings of the raw image, each by shifting the indices of the pixels at the origin, that is to say at the top left, of the raw image by a value d = {ûy, is [0, ppcm(>y, f) / 4} and by using this same step sf for each shift value, we obtain ppcm(.y, t) Is estimates of the color image which can be averaged to obtain a final color image. The result is equivalent to a reconstruction with any step 5 while keeping the formalism of the convolution by a filter bank of the method. As an example, this embodiment can be used on the example illustrated by figure 1 in order to estimate the color image with a cutting step for example equal to 1. The size of the pattern illustrated in figure 1 is t = 2.A first matrixing and a first demosaicing are then done on the entire image 100, with a cutting step £ = 2 and for example windows of size 5 x 5. A second matrixing and a second demosaicing are then done from the second column of pixels of image 100, thus ignoring the pixels of the first column. The channel estimated for the pixel at the origin, at the top left, is then the red component. The cutting step remains equal to 2. Following these two estimations, the cutting step of the image is indeed equal to 1. In this example the matrixing matrix associated with the first demosaicing is different from that associated with the second demosaicing. Although the prior distribution of xa was given as being centered, implying that the prior mean of the mean vector is zero, it is of course possible to remove this assumption and agree that the prior distribution of x is an N( 5, Q ) • The vector ô is then of size n( WjW? + 1 ), whose first elements are zeros and the last n, i.e. the n means, are by non-zero. For example, n=3 and Oh- and the values ​​ôr, ^g and ôh are for example the <5.5g 5 b . a priori average values ​​for the window averages and The values ​​ôr, 5i and Ôb are, for example, all equal to 0.5, although any other value, included in the interval [0,1] is possible. The estimator x of the Maximum a Posteriori of x then has the form: [Math 15] 'y = (MrRlM + Q+ ô'!ô).The person skilled in the art will be able to adapt the calculations to the presence of this additional term. Indeed, simplifications are made in the development of the expression for determining the optimal index and lead to the expression of a filter bank.

[0129] The reformulation of the Maximum A Posteriori estimator including the means of each channel as unknowns has the advantage of leading to a convolution operation by a filter bank on the overall raw image. In addition, convolution operations are operations widely used in convolutional neural networks. Thus, the processors used for the execution of convolutional neural networks are particularly well suited to the execution of image processing according to the present embodiments.

[0130] Another advantage of the described embodiments is that the step of selecting, by window, the most suitable covariance matrix results from the filter bank convolution operation.

[0131] Another advantage of the described embodiments is that they adapt to any die pattern. Indeed, it is sufficient to adapt the die matrix M to the chosen pattern. This adaptation is simple and is within the reach of the person skilled in the art.

[0132] Another advantage of the described embodiments is that it is possible to adjust the time cost of image processing by adjusting the values ​​of the variables L, ■' and wi, w2.

[0133] Various embodiments and variations have been described. Those skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will occur to those skilled in the art. In particular, although the case in which the measured and estimated channels described are primarily color channels, the type of channels measured and estimated may vary. However, the channels to be estimated have a spatial correlation with the measured channels so that the estimation is meaningful.

[0134] Finally, the practical implementation of the embodiments and variants described is within the reach of the person skilled in the art from the functional indications given above. In particular, any matrixing pattern can be used, the only condition being to adapt the matrixing matrix M to the pattern and to choose the cutting step ' so that, for an estimation, each window has the same filter configuration. The size of the windows as well as the size of the GMM model can also vary.

Claims

Claims

1. Method for dematrixing an image (100), by a processor (210) of an electronic device (200), from measurements (m), carried out by an imager (202) and according to a matrixing pattern, of at least one channel, from a first set of channels (r,g,b,z), the method comprising: - the determination, by the processor (210), for each window of a plurality of windows covering the image, of a covariance matrix (Ep, from a set of covariance matrices (Er) stored in a memory (206) of the device, by performing, for each covariance matrix, a shift convolution operation (* «) on the entire image with a filter bank ( / associated with the covariance matrix;- for each window, the calculation, on the basis of the determined covariance matrix, of an estimator (xM) comprising estimates of values ​​of at least one channel, among a second set of channels (r,g,b), for each pixel of the window, the estimator further comprising an estimate of an average value (Mr, Fg, of the at least one channel over the entire window; and - the generation of a new image on the basis of the estimator of each window.;

2. The method of claim 1, wherein the covariance matrices (Ej of the set of covariance matrices, come from a Gaussian mixture model (GMM).

3. A method according to claim 1 or 2, wherein the windows are selected such that the matrix pattern is identical from one window to another.

4. Method according to any one of claims 1 to 3, wherein the calculation of the estimator (xw) is further carried out on the basis of a matrixing matrix (M) dependent on the matrixing pattern.

5. A method according to any one of claims 1 to 4, wherein each window is spaced from its neighboring windows by a clipping step (s) and the shift convolution operation is a step convolution, the step being equal to the clipping step of the image, the step being a multiple of the size of the matrix pattern.

6. A method according to any one of claims 1 to 5, wherein, for each window, the estimator x is calculated by making the matrix product: [Math 16] X = U'A^F^m, °where m is a vector comprising the measured values ​​for the window, U*, is an orthonormal matrix whose columns are the eigenvectors of a matrix Aj and A* is a diagonal matrix comprising the square roots of the eigenvalues ​​of the matrix A^, the matrix is ​​equal to MT RlM + QS M being the matrixing matrix, i R being the covariance matrix associated with the measurement noise of the imager and Q„ being the covariance matrix of the prior distribution of the estimator and i comprising a covariance submatrix E< which is the determined covariance matrix, and where F is a matrix equal to M1 R1-

7. Method according to claim 6, in which, for each covariance matrix Ej, among the set of covariance matrices, the associated filter bank is a sequence of matrices originating from the elements of the matrix F-.

8. Method according to claim 6 or 7, further comprising, before determining the covariance matrix, and for each covariance matrix, calculating and storing in the memory (206) the matrices Ui and F^

9. Method according to any one of claims 1 to 8, further comprising, following the calculation of the estimators, the reconstruction of an image by doing: - for each pixel of each window, the calculation of a value of the pixel by adding the estimated average (Pr, F b) of the at least one channel, among the second set of channels, to the estimate (rn^, à , h ) of the value of the at least one channel.

10. The method of claim 9, further comprising, for each pixel covered by at least two separate windows, calculating the average of the pixel values.

11. A method according to any one of claims 1 to 10, wherein the at least one channel, among the second set of channels, is spectrally correlated to the at least one measured channel, among the first set of channels.

12. A method according to any one of claims 1 to 11, wherein the first and second sets of channels include the red, green, blue color channels.

13. The method of claim 12, wherein the first set of channels further comprises a depth channel.

14. Electronic device (200) comprising: - a non-volatile memory (206) configured to store a set of covariance matrices and a set of filter banks, each filter bank being associated with a covariance matrix; and - a processor (210) configured to: determine a covariance matrix for windows covering a raw image by performing: for each covariance matrix, a shift convolution operation, from a set of measurements associated with the windows of the filter bank associated with the covariance matrix; and for each window, calculating an estimator, based on the determined covariance matrix, the estimator comprising estimates of values ​​of at least one channel, among a second set of channels, for each pixel of the window, the estimator further comprising an estimate of an average value of the at least one channel over the entire window;and generate a new image based on the estimator of each window.;

15. The device of claim 14, wherein the memory (206) further comprises sets of parameters F^ each set being dependent on the set of covariance matrices, and wherein the processor is configured to calculate the estimator based on the sets of parameters depending on the determined covariance matrix.

Images