Method for demosaicing a raw image, and computer program, device and system implementing such a method

Abstract

The invention relates to a method (200) for demosaicing a raw image (IMb) comprising at least one iteration of a demosaicing phase (210) iteratively modifying an image being demosaiced as a function of an error (E) corresponding to the sum of: - a deviation term relating to a deviation between the raw image (IMb) and the image being demosaiced (IMdem) obtained in the previous iteration, and - a smoothing term, dependent on the image being demosaiced (IMdem). When the error (E) does not satisfy a predetermined threshold value, the image being demosaiced (IMdem) is modified to decrease the error, and so on until the error satisfies the threshold. The invention also relates to a computer program, a device, a system and a vehicle implementing such a method.

Description

DESCRIPTION Title: Method for demosaicing a raw image, computer program, device and apparatus implementing such a method

[0001] The present invention relates to a method for demosaicing a raw image. It also relates to a computer program, a device, and an apparatus implementing such a method. Furthermore, it relates to an image processing method and an image acquisition method implementing such a demosaicing process.

[0002] The field of the invention is the field of signal processing captured by an image sensor of a camera module. State of the art

[0003] Demosaicing, also called debayering or demosaicing, is a processing technique for the raw signal from a digital camera sensor. It involves interpolating the data from each of the monochrome red, green, and blue photosites that make up the image sensor to obtain a trichromatic value for each pixel of the image.

[0004] The image sensor, or photographic sensor, of a camera module typically comprises red, blue, or green photosites. Each photosite reacts to the light it receives and converts it into an electrical charge, based on the principle of a photoelectric cell. The most well-known arrangement of photosites is called the Bayer matrix. This matrix is ​​formed by a multitude of photosite patterns, each pattern generally square and comprising two green photosites arranged diagonally, and one red and one blue photosite arranged on the opposite diagonal. The signal received from such an image sensor must be demosaiced, using widely known techniques, to estimate, for each pixel of the image, the value of each of the red, green, and blue components of that image.

[0005] Current demosaicing solutions are effective when the camera module, and in particular the optical lens, has limited chromatic dispersion. Their effectiveness decreases as the Chromatic dispersion of the camera module, which poses a significant constraint on the design of said module. The efficiency of current demosaicing solutions also decreases when, in the imaged scene, the space spatially separating the three colors red, green, and blue increases, that is to say, when the spatial chromatic dispersion of the scene increases.

[0006] One objective of the present invention is to remedy at least one of the aforementioned drawbacks.

[0007] Another objective of the invention is to provide an effective demosaicing solution even in cases of high chromatic dispersion. Description of the invention

[0008] The invention proposes to achieve at least one of the aforementioned goals by a method of demosaicing a raw image, IMb, of a scene, acquired by a camera module comprising an image sensor, said method comprising at least one iteration of a demosaicing phase iteratively modifying an image, called the image being demosaicing, and comprising the following steps: - Calculation of an error value corresponding to the sum of: ■ a first term, called the difference term, relating to a difference between said raw image and the image being demosaicing obtained in the previous iteration of said demosaicing phase, and ■ a second term, called the smoothing term, a function of the said image during demosaicing; - when said error value: ■ satisfies a predetermined threshold value, memorization of said image during demosaicing as a demosaiced image, and ■ does not satisfy said threshold value, modification of said image during demosaicing in order to reduce said error value and execution of a new iteration of said demosaicing phase.

[0009] Thus, the process according to the invention proposes to perform an iterative and progressive demosaicing of the raw image. The demosaicing is carried out by minimization of an error value depending on both the difference between the dematrixed image and the raw image, this difference having to be as small as possible and a second term allowing to avoid obtaining aberrant results. [OO1O] Consequently, the demosaicing proposed by the present invention is less sensitive to the chromatic dispersion of the optical lens used to image the scene: it remains effective even when the chromatic dispersion of the optical lens is high. Furthermore, the demosaicing proposed by the present invention is less sensitive to the spatial dispersion of the red, green, and blue colors in the imaged scene: it remains effective even when the spatial chromatic dispersion is high in the imaged scene.

[0011] Indeed, the invention performs iterative demosaicing by using, at each iteration, a smoothing term / operator that guides the reconstruction of missing information towards a likelihood of the scene being reproduced. This term / operator is calculated independently of the deviation term, which reflects the fidelity to the scene as captured, and therefore the demosaicing accuracy. In current solutions, these two concepts are either combined or considered sequentially, which reduces the effective resolution of the demosaicated image and / or the demosaicing accuracy. Thus, the invention enables more precise demosaicing while remaining as faithful as possible to the imaged scene.

[0012] Furthermore, by having access to each operator individually, it is possible to adjust these operators individually and thus customize the scene reconstruction guidance during the demosaicing of an image of said scene.

[0013] Raw image refers to a set of values ​​provided by an image sensor of an imaging module, such as a CMOS sensor or a CCD sensor or any other sensor.

[0014] In some embodiments, the image sensor includes a Bayer matrix, said Bayer matrix comprising B Bayer patterns, each Bayer pattern Bi potentially comprising P monochrome photosites. In this case, the raw image IMb can comprise a matrix of B sets of values, each set of values ​​Bi being provided by a Bayer pattern Bi. Each set of values ​​Bi comprises P values, each value being provided by a photosite of the Bayer pattern Bi supplying said set of values.

[0015] In some embodiments, a Bayer pattern may comprise P=4 photosites, for example, 2 green photosites, one red photosite, and one blue photosite. In other embodiments, the Bayer pattern may be square, with the two green photosites arranged on one diagonal of the square and the red and blue photosites arranged on the other diagonal.

[0016] The demosaiced image can be represented by N sets of values, each set of values ​​corresponding to a pixel of the demosaiced image. Each set of values ​​in the demosaiced image can contain C values. For example, in the case of an RGB image, each set of values ​​contains three values, one for each color.

[0017] Following examples of implementation, for a raw image comprising B patterns, demosaicing can provide a demosaiced image comprising N pixels such that N = B: in this case each Bayer pattern provides one pixel of the demosaiced image.

[0018] Following other realization examples, for a raw image comprising B value patterns, demosaicing can provide a demosaiced image comprising N pixels such that N = P*B: in this case each Bayer pattern provides P pixels of the demosaiced image.

[0019] Optical aberration refers to the phenomenon where the image of a point in the scene becomes a more or less large bright spot in the (generally flat) space where the image is formed. Aberration manifests itself in different types of effects: there are aberration effects characterized by a shift in the average position of the optical spot produced by a point in the scene, relative to the ideal location where it would occur. We will call these displacement aberrations. There are other effects whereby the optical spot has a spatially spread energy distribution, instead of being a point source, in the image space. We will call these spreading aberrations.

[0020] Aberrations due to an optical lens are commonly referred to as chromatic aberration, geometric aberration, and third-order aberrations. ème order.

[0021] By "aberrations of 3 ème"Order" refers to the set of aberrations introduced by an optical lens in an image acquired with said optical objective, except for chromatic aberration and geometric aberration. These aberrations of the 3 ème Third-order (and higher-order) aberrations include, for example, spherical aberration, coma, astigmatism, field curvature, distortion (of shape and energy distribution of the optical spot), etc. These aberrations of the 3rd order ème order (and subsequent order) are of the spreading type. In the following, for the sake of brevity, the expression "aberrations of the 3rd ème "Order" refers to the aberrations of the 3rd ème order and next order.

[0022] Chromatic aberration, which results from the decomposition of light into several bands of color, refers to an optical aberration that produces different positions in space for the focal point depending on the wavelength. For lateral chromatic aberration, the focal point shifts within the image location as a function of wavelength. This results in an image with iridescent edges or colored fringes around objects of uniform color, such as white objects. Axial chromatic aberration, on the other hand, involves focusing outside the image location because the focal points are shifted along an axis perpendicular to the image location (or the axis corresponding to the central optical ray arriving at the image point, which may be oblique to the image location). This results in an image with blurred colors.Lateral chromatic aberration can be corrected by digitally processing the image captured by a lens, provided it is quantified, that is, measured during the design / manufacturing of the optical lens. It is similar to displacement aberrations. Axial chromatic aberration, on the other hand, is similar to spreading aberrations, because if, for example, the color green is focused on the sensor at the image plane, the color red may focus before it, and the red optical beam will become broadened when it encounters the sensor at a distance beyond the focal point.

[0023] By "geometric aberration," we mean a discrepancy between a paraxial ray defined in the Gaussian approximation and the corresponding real ray. A geometric aberration can also be characterized by a discrepancy between the paraxial wavefront and the real wavefront. For example, geometric aberrations produce a distortion of the image shape. obtained. For example, a square can become a shape equal to the outline of a barrel (with inward-curving corners), or a seamstress's cushion (with protrusions on the corners). Thus, the straight, vertical edges of a building can become curved, rather than vertical. This aberration can be corrected by digital processing of the captured image once it is quantified, that is, measured during the design / manufacturing of the optical lens. Geometric aberrations are of the displacement type.

[0024] By "displacement aberration" or "displacement aberration", we mean lateral chromatic aberration and geometric aberration.

[0025] By "spreading aberration," or "spreading aberration," we mean axial chromatic aberration and aberrations of the 3rd ème order.

[0026] The image can be modified during demosaicing using any known method.

[0027] Depending on the embodiment, the modification of the image during demosaicing can be carried out according to a random or semi-random algorithm or rule.

[0028] Depending on the embodiment, the modification of the image during demosaicing can be carried out according to an algorithm, or a rule, taking into account the evolution of, or a derivative of, the error value or a function representing said error.

[0029] Depending on the implementation, image modification during demosaicing can be performed using the Levenberg-Marquardt algorithm. More information about this algorithm can be found, for example, on the page https: / / fr.wikipedia.org / wiki / Algorithme_de_Levenberg-Marquardt.

[0030] Depending on the embodiment, the demosaicing phase can be applied to the entire raw image.

[0031] In this case, the entire image is processed as a single block, or in its entirety. In other words, the demosaicing phase attempts to reduce the error across the entire image simultaneously. Put another way, each iteration of the demosaicing phase Demosaicing is applied to the entire image so that the gap term and the smoothing term are calculated taking into account the whole image so that the error is calculated for the whole image.

[0032] This error over the whole image is a sum of local contributions. Each local contribution to the error at a point takes into account, through the radius of the PSF function (where applicable, as described later), and / or the OP operators (as described later), a more or less local scope of the image, greater than in the prior art, and thus more consistent with both the optical spreading and the nature of the imaged objects, making the method more efficient.

[0033] Depending on the embodiment, the demosaicing phase can be applied by zone / region to the raw image, with at least two regions of the image being able to be processed in turn or in parallel.

[0034] In this case, the method according to the invention may include: - prior to the demosaicing phase, a step of dividing the raw image into several regions, the demosaicing phase being applied individually to at least one, and in particular to each, of said regions; and - an image reconstruction step with said at least one demosaiced image region.

[0035] In other words, for at least one given region of the image, the demosaicing phase is applied individually to that region and in isolation from the rest of the image. The error term and the smoothing term are calculated considering only that region of the raw image. Then, iteratively, the error is reduced, or even minimized, by performing one or more iterations of the demosaicing phase, again for that region taken individually and in isolation from the rest of the image, to demosaic that region of the image.

[0036] One or more regions of the image can be individually demosaiced. After demosaicing, each demosaiced region thus obtained from the raw image constitutes the corresponding region of the demosaiced image.

[0037] Following a non-limiting example, the raw image can be divided into k regions, Ri-Rk, with k>l. Each region Ri can be demosaiced individually and independently of other regions. After individual demosaicing, each demosaiced region Ri can be concatenated to the other regions to form the demosaiced image.

[0038] Depending on the embodiment, the division of the raw image into several regions can be carried out randomly.

[0039] Alternatively, the raw image can be divided into several regions by detecting objects in the raw image.

[0040] According to yet another alternative, the division of the raw image into several regions can be achieved by detecting different planes of different depths in the scene: in this case, the raw image can be divided into several regions, each corresponding to a given plane.

[0041] According to yet another alternative, the division of the image into several regions can be carried out according to coordinates in the image, for example cylindrical coordinates, that is to say the distance to the center of the image.

[0042] The difference term can be calculated according to any relationship providing a comparative quantity between the raw image and the image being demosaiced.

[0043] According to some embodiments, the demosaicing phase may include a convolution of the image being demosaicing with an optical transfer function associated with the camera module, and in particular with an optical lens of said camera module, the difference term being calculated between the result of said convolution and the raw image.

[0044] Thus, the image being demosaicing is first convolved with an optical transfer function of the camera module's optical lens, and the difference term is calculated using the image being demosaicing resulting from said convolution. This convolution makes it possible to take into account, during the demosaicing of the raw image and in the image being deconvolved, the optical response of the optical lens. Therefore, the method according to the invention makes it possible to take into account, and in particular to reduce or eliminate, the effect of the optical lens used to image the scene during demosaicing, which allows for more efficient, more precise and more faithful demosaicing to the real scene.

[0045] The optical transfer function can be the PSF, for "Point Spread Function", or the MTF for "Modulation transfer function", or any other optical transfer function.

[0046] Depending on the embodiment, the optical transfer function can be constant and the same for all points of the scene.

[0047] In this case, the said optical transfer function can be determined beforehand, for example during the manufacture of said optical lens or during a calibration phase of said optical lens.

[0048] Depending on the embodiment, the optical transfer function can be a function of the depth of objects in the scene.

[0049] In this case, the optical transfer function of the lens changes depending on the distance, denoted DOS, between the lens and the objects in the scene. Potentially, an optical transfer function can correspond to each plane in the scene that is at a different depth.

[0050] Depending on the embodiment, the optical transfer function can be a function of the focus used to acquire the raw image, i.e. of the distance, denoted DOC, between the optical lens and the image sensor.

[0051] In this case, the optical transfer function of the lens is different for each focus point. Potentially, one optical transfer function can correspond to each different focus point, that is, to each DOC distance.

[0052] According to some embodiments, the process according to the invention may include, before the dematting phase, a determination step of a scene depth map, the optical transfer function(s) being determined according to said depth map.

[0053] The depth map of the scene can be determined by measurement with at least one sensor, such as a LIDAR, a time-of-flight camera, etc.

[0054] The scene depth map can be determined by analyzing a previously acquired exploratory image of the scene.

[0055] The depth map can be determined from the unrasterized image where appropriate, for example by using only the brightness information deduced from the raw image, or by using the monochrome red, green or blue information deduced from the raw image.

[0056] When the image sensor includes dual-pixels, the depth map can be determined from the image formed by the data provided by said dual-pixels, possibly using the raw image.

[0057] The scene depth map indicates, for each point in the scene, the distance—and therefore for each pixel of the image being demosaiced, and thus of the demosaiced image—between the optical lens and that point in the scene. Knowing this depth map, an optical transfer function corresponding to each depth can be identified and used to calculate the gap term for all points located at that depth.

[0058] For example, for each pixel of the image being demosaiced, the optical transfer function can be represented by a function, or a matrix of values, indicating the defect, or the correction to be applied for each of the color components of said pixel.

[0059] According to non-limiting embodiments, the deviation term can be determined according to the following relationship: \PSF * IM(J — l) dem — IM b \ 2 with IM(kl)dem the image being demosaiced obtained in the previous iteration, and IMb the raw image.

[0060] Of course, the gap term can be calculated without using an optical transfer function.

[0061] According to embodiments, the process according to the invention may include a step of correcting, in the raw image, the lateral chromatic aberration of the camera module, and in particular of the optical lens of said camera module, the difference term being calculated between the image being demosaicing provided to the previous iteration of the demosaicing phase and the raw image provided by said correction step.

[0062] Thus, the lateral chromatic aberration introduced into the raw image by the optical lens is corrected, and it is the raw image obtained after this correction that is used to calculate the deviation term. Consequently, this correction allows lateral chromatic aberration to be taken into account during the demosaicing of the raw image, resulting in more efficient, more precise, and more faithful demosaicing of the actual scene.

[0063] Lateral chromatic aberration correction can be achieved using a lateral chromatic aberration correction function, denoted FCACL, previously determined for the camera module, and in particular the optical lens, used for the acquisition of the raw image.

[0064] Depending on the embodiment, the FCALC may be represented as a value field comprising, for each of several (x,y) positions of the image sensor, at least one displacement vector to be applied to correct said lateral chromatic aberration at said position. For at least one (x,y) position of the sensor, the FCALC may include: - a single displacement vector for each of the colors red, green, and blue; or - for at least one, and in particular each, of said colours, a displacement vector dedicated to said colour.

[0065] Alternatively, FCACL can be a mathematical function that takes as input the (x,y) position of the image sensor and provides one or more displacement vectors to be applied to correct lateral chromatic aberration at said (x,y) position.

[0066] Alternatively, the FCACL can be presented as a lookup table indicating, for each of several pixels of the image sensor, a corrected position of said pixel individually: - for each color, or - for at least two of the colours, for example the colour red and the colour blue, in the case where the third colour, for example the colour green, is taken as a reference.

[0067] The FCACL can be determined as a function of a lateral chromatic aberration function, denoted FACL, indicating the lateral chromatic aberration of the camera module, and in particular of the optical lens of said camera module.

[0068] For example, FCACL can correspond to the opposite, or inverse, of the FACL function so as to cancel / neutralize said FACL.

[0069] FACL can take different forms.

[0070] Depending on the embodiment, the FACL can be represented as a value field comprising, for each of several (x,y) positions of the image sensor, at least one displacement vector. In one example embodiment, for at least one (x,y) position, the value field may include: - a displacement vector measured for the color red, - a displacement vector measured for the color green, and - a displacement vector measured for the color blue; relative to a reference position. The reference position can be any previously chosen position. For example, the reference position could be the centroid of the positions of the three colors.

[0071] Following an example implementation, the position of one of the three colors can be chosen as the reference. In this case, for at least one (x,y) position, the value field can include a displacement vector for each of the other two colors. For example, the position of the color green can be chosen as the reference: in this case, for at least one (x,y) position, the value field can include a displacement vector for the color red and a displacement vector for the color blue.

[0072] Alternatively, FACL can be a mathematical function taking as input a position (x,y) of the sensor and providing one or more displacement vectors, such as those indicated above.

[0073] The FACL can be determined in different ways, for example by simulation with a numerical model of the optical lens, by prediction with a prediction model trained for this purpose, or by measurement directly on the optical lens.

[0074] The measurement / determination of FACL can be carried out in different ways.

[0075] In some embodiments, a target formed of patterns, such as dots, lines, etc., can be positioned in front of the optical lens at a measurement plane. The target is illuminated by white light, or successively by green, red, and blue light. A pattern corresponding to a position (x,y) of the image sensor generates a spot for each color on said image sensor. The position of each color spot can be detected by the image sensor. The detected positions of the three color spots are then used to calculate a displacement vector relative to a reference position, which can be the position of one of said three spots, or another position.

[0076] By taking measurements simultaneously or in turn for several positions (x,y) in the sensor plane, also called the image plane, it is then possible to determine the FACL.

[0077] Depending on the embodiment, FACL can be measured for all image sensor positions. Alternatively, FACL can be measured for only certain image sensor positions.

[0078] Depending on embodiments, the process according to the invention may include a correction step: - in the raw image, or - in the raw image previously corrected for lateral chromatic aberration as described above; for the geometric aberration of the camera module, and in particular of the optical lens of said camera module; the difference term being calculated between the image being demosaicing provided to the previous iteration of the demosaicing phase and said raw image provided by said correction step.

[0079] Thus, the geometric aberration introduced into the raw image by the optical lens can be corrected, or neutralized, at least partially, and it is the raw image obtained after this geometric aberration correction that is used to calculate the deviation term. Consequently, this correction allows the geometric aberration to be taken into account during the demosaicing of the raw image, resulting in more efficient, more precise, and more faithful demosaicing. Therefore, the demosaicing operation also provides an image corrected for geometric distortions at the end of this step, potentially saving a subsequent step of correcting these distortions.

[0080] Geometric aberration correction can be achieved using a geometric aberration correction function, denoted FCAG, previously determined for the camera module, and in particular the optical lens, used for the acquisition of the raw image.

[0081] Depending on the embodiment, the FCAG may be represented as a field of values ​​comprising, for each of several angles (0i,yi) of incidence on the optical lens of the camera module used for acquiring the raw image, with 0i the angle relative to the axis of the optical lens and yi the angle of rotation about said optical axis, at least one distance vector to be applied to correct said geometric aberration. For at least one angle (0i,yi), the FCAG may include: - a single distance correction value to be applied for each of the red, green, and blue colors; or - for at least one, in particular each, of the said colours red, green and blue: a distance correction value dedicated to said colour.

[0082] Alternatively, FCAG can be a mathematical function taking as input an angle (0i,yi) relative to the axis of the optical objection and providing one or more distance correction values.

[0083] Alternatively, FCAG can be presented as a lookup table indicating, for each of several pixels of the image sensor, a corrected position of said pixel: - individually for each color, or - for all colors.

[0084] FCAG can be determined as a function of a geometric aberration function, denoted FAG, indicating the lateral chromatic aberration of the camera module, and in particular of the optical lens of said camera module.

[0085] For example, FCAG can correspond to the opposite, or inverse, of the FAG function so as to cancel / neutralize said FAG.

[0086] The FAG can take different forms.

[0087] In some embodiments, the FAG can be represented as a value field comprising at least one displacement distance for each of several angles of incidence (0i,yi) with respect to the axis of the optical lens, where 0i is the angle with respect to the axis of the optical lens and yi is the angle of rotation about said optical axis. In one example embodiment, for at least one angle (0i,yi), the value field may include: - a measured displacement distance for the color red, - a measured displacement distance for the color green, and - a measured displacement distance for the blue color; relative to the expected position of the illuminated point on the optical sensor, the expected position in the absence of geometric distortion.

[0088] Following an example of implementation, for at least one angle (0i,yi), the field of values ​​can include a single displacement distance, obtained from the displacement distances for the three colors, for example an average of said displacement distances.

[0089] Alternatively, the FAG can be a mathematical function taking as input an angle (0i,yi) and providing a single displacement distance for all three colors, or a displacement distance individually for each color.

[0090] The FAG can be determined in different ways, for example by simulation with a numerical model of the optical lens, by prediction with a prediction model trained for this purpose, or by measurement directly on the optical lens.

[0091] The determination / measurement of FAG can be carried out in different ways.

[0092] In some embodiments, a target formed of patterns, such as dots, lines, etc., can be positioned in front of the optical lens at a measurement plane. The target is illuminated by white light, or successively by green, red, and blue light. A pattern located at an angle (θi, θi) relative to the axis of the optical lens generates a spot on the image sensor for each of the colors red, green, and blue. The position of each color spot can be detected by the image sensor. The detected positions of the three spots are then used to calculate a displacement distance for each color, relative to the center of the optical lens. Optionally, the displacement distance can be divided by the focal length to obtain an angular quantity as a function of the angle θi of the optical beam entering the optical lens relative to the axis of the optical lens.

[0093] By taking measurements simultaneously or in turn for several angles 0i relative to the optical axis of the optical objective, it is possible to determine the FAG.

[0094] Depending on the embodiment, the FAG can be measured for all image sensor positions. Alternatively, the FAG can be measured for only certain image sensor positions.

[0095] Depending on the embodiment, the FAG can be a function of, or can take into account, the depth at such and such a point in the scene.

[0096] Following non-limiting examples of implementation, the deviation term can be determined according to the following relationship: with IM(kl)dem the image being demosaiced obtained in the previous iteration, and IMb.dec the raw image having undergone lateral chromatic aberration correction and / or geometric aberration correction.

[0097] Following non-limiting examples, the raw image after lateral chromatic correction can be obtained by any one of the following relationships: IMb,dec = FCACL * IMb; IM b ,dec = FACL'l * IMb ; IMb, dec = FCAG * IMb; IMb, dec = FAG * IMb - 1 ; IMb, dec = FCAG * (FCACL * IMb), or IMb, dec = FAG * (FACL * IMb -1 ) * -1 .

[0098] According to non-limiting embodiments, the smoothing term can be a sum, possibly weighted, of several operators, OP, taking as input the image being demosaiced provided by the previous iteration of the demosaicing phase.

[0099] At least one of these operators can be pre-selected based on one or more criteria relating to the desired demosaicing, such as the accuracy of the demosaicing or the fidelity of the demosaicing to the real scene.

[0100] For example, at least one of these operators can be chosen based on the type of solutions we want to obtain because they would be encountered quite likely in real images, or the type of solution we do not want to obtain because they would be quite unlikely or not at all likely.

[0101] In general, it is possible to perform a calculation based on the pixel values ​​of a probable image or image portion such that the value of an OP does not increase, or increases only slightly, or preferably decreases, when presented with the pixel values ​​for these cases. Conversely, it is generally possible to perform a calculation based on the pixel values ​​of an unlikely or improbable image or image portion such that the value of OP increases significantly when presented with the pixel values ​​for these cases. Based on these general principles, several methods are possible for determining the OP operators.

[0102] It is possible to proceed analytically and algebraically, that is, to look for a characteristic property, preferably local at the pixel level, relative to the previously defined objectives. Then, one can write an equation whose value changes relative to this locally translated property. Finally, the result of the equation can be modified, for example, by writing it in the denominator of a division to reverse its value. Variation, that is, increasing or decreasing the value of the OP operator to achieve the property, depends on whether the goal is to avoid or achieve it, respectively. For example, the property might be the occurrence of frequent jumps in pixel hue. This frequency is considered high at the scale of the pixels or photosites of the Bayer matrix, that is, at the scale where it is difficult to recover missing color values, where the property can be achieved, even though this is generally undesirable. This example is detailed later.

[0103] Alternatively, or in addition, it is possible to proceed by learning desirable and undesirable cases without explicitly writing equations. This can be achieved using a neural network approach. This involves training a sufficiently sized network—one with enough neurons and enough inputs—by having it learn as input the image regions possessing a given property, and as output a value, or a range of values, corresponding to each image region with that property. The value is higher when the property is undesirable, and lower when it is desirable. This results in a neural network—a network of operations and coefficients—that can be replicated at all positions within the image field, for example, by translation by block or by pixel along the X and Y directions of the image.The neural network, during its initial training, can receive the values ​​of its neighboring pixels within a square, for example, 11x11 pixels. This means it receives the values ​​up to its five neighbors to the left, right, above, below, and itself, for all color components, generally red, green, and blue, or a transformation of these values. The output is a coefficient representing the value of the operator OP at the position of that pixel. This network can then be replicated by translation to all positions in the image after its training is complete (except for the edge bands, which are special cases requiring, for example, a network with fewer inputs at the edges).

[0104] Following non-limiting examples, the smoothing term can be determined according to the following relationship: with IM(kl)dem the image being demosaiced obtained in the previous iteration, OP La predetermined smoothing operator, a weighting coefficient.

[0105] At least one OPi operator can be chosen beforehand based on one or more criteria relating to the desired demosaicing, or the imaged scene, or even an imaging mode.

[0106] A specific, but by no means limiting, example of an operator is constructing a pattern in such a way as to limit the occurrence of frequent color jumps within the same motif. It is observed that a color jump is quite frequently linked to a hue jump. It is then possible, for example, to transform the (R, G, B) values ​​of the pixels in the IM(kl)dém image into HSL, that is, Hue, Saturation, Luminosity. An equation is then formed with the hue jump. This can be obtained by writing the norm 1, that is, the sum of the absolute values ​​of the X and Y components of the hue gradient. Thus, the absolute value of the hue variation with neighboring pixels is calculated, that is, at each position of the demosaicated image, and the sum is written as the absolute value of the hue of the pixel above minus the hue of the pixel at position , plus the absolute value of the hue of the pixel to the right minus the hue of the pixel at position .Therefore, the greater and more frequent the variation in hue from pixel to pixel, regardless of direction, the higher the value calculated by the corresponding OPi or OPT operator. Consequently, the resulting image will be guided, during its calculation, towards less variation in hue at each position.

[0107] It is also noticeable that a color jump is quite frequently associated with a saturation jump. Another type of OP2 or OPs operator can be created from saturation values, in a similar way, and lead to limiting saturation variations, that is to say, avoiding jumps between a "black-and-white" part, i.e., in shades of gray, and a part with more pronounced colors, within the same pattern.

[0108] To give another example, the property to avoid is a color jump, and the property to favor is a color jump around the edges of an object. This latter edge property is generally correlated with a color jump.of brightness. Thus, we write one equation for the hue jump, another for the brightness jump, and combine them. The terms of the OP1 operator previously written for the hue jump can then be divided, for each value of the sum of the absolute values ​​of the hue variations, by a constant plus the norm of the local brightness gradient, defined, for example, as the square root of the sum of the squares of the X and Y components of the brightness gradient. Thus, when this norm exceeds a certain threshold defined by the constant, the hue can vary more strongly without increasing the value of OP, which allows it to be changed more easily when crossing the contours of an object, or part of an object, since brightness jumps are generally frequent there, leading to higher brightness gradients than elsewhere.But the jumps in color remain limited, particularly in the interior of objects or patterns of these objects, thanks to the OPT operator which limits them.

[0109] As another example, we can proceed in the same way with the OPs operator to combine it with brightness jumps.

[0110] At least one weighting coefficient A t can be pre-selected based on one or more criteria related to the desired demosaicing, the imaged scene, or an imaging mode. In particular, the relative modulation of A values t The interaction between operators allows for a modification of the weighting between different types of properties to which each operator is sensitive. For example, one operator sensitive to hue jumps (OPT) was constructed, and another sensitive to saturation jumps (OPs). Furthermore, varying the set of coefficients A tby the same multiplicative coefficient, for example by increasing these values, will allow less weight to be given to the 1 er In terms of comparison to the detection matrix, this means allowing greater discrepancies between the components calculated in the demosaicated image and the corresponding values ​​detected on the photosites. In particular, this allows for compensation of more detection noise, and such a setting can prove useful for images or parts of images obtained at lower luminance, and therefore with greater sensitivity to noise. [YES] Thus, the values ​​of the weighting coefficients A t can be modified jointly according to parameters such as image brightness, exposure time, etc.

[0112] According to another example, at least one weighting coefficient A tcan be chosen individually differently from the other coefficients / ^ for k different from I. This individual choice can be made based on a trade-off between different types of properties that are generally observed in an image, or based on the type of scene observed, adapting to one or another type of scene for which this trade-off might be different.

[0113] For example, for two operators OPT and OPs sensitive to hue and saturation, and if we want hue jumps to appear less frequently than potential saturation jumps (because we would have noticed that saturation jumps are quite frequent in the real image), we could increase A T for example by doubling it to make the appearance of hue jumps less likely, and to allow saturation jumps to occur more often.

[0114] Similarly, assuming that we have constructed a light-sensitive OPL operator, we could decrease its coefficient A L compared to other hue and saturation coefficients because we would have noticed that modulations of brightness are more frequent than jumps in hue, or saturation, for example because brightness often varies on a real scene, for the same object.

[0115] As another example, for a picture taken in a very colourful environment that differs from a more homogeneous colour natural environment, one could choose a different set of coefficients depending on these cases.

[0116] For the calculation of the error, and in particular for the deviation term and / or for the smoothing term, the image being demosaiced can be represented by R, G, B values, respectively for the red color, the green color and the blue color.

[0117] Alternatively, for error calculation, and in particular for the deviation term and / or the smoothing term, the image being demosaicated can to be represented by HSL values ​​respectively the hue, denoted T, the saturation, denoted S, and the luminance, denoted L.

[0118] Using a HSL representation of the image being demosaicing, at least in the smoothing term, allows for more efficient and precise demosaicing.

[0119] The conversion from an RGB representation to an HSL representation, or from an HSL representation to an RGB representation, is well known to the person in the trade.

[0120] According to another aspect of the same invention, an image processing method is proposed comprising the following steps: - demosaicing of a raw image by the process according to the invention, providing a demosaiced image; and - digital processing of said demosaiced image.

[0121] The processing stage can perform any type of processing of the demosaiced image, such as, for example, correction of an optical aberration, correction of a geometric aberration, modification of the sharpness of the image, modification of the brightness of the image, enhancement of at least one color in the image, etc.

[0122] The processing step can be carried out by the same device as the one performing the dematting, or by another device.

[0123] According to another aspect of the same invention, a computer program is proposed comprising executable instructions which, when executed by a computer device, implement all the steps of the process according to the invention, and in particular: - the dematting process according to the invention, or - the image processing method according to the invention.

[0124] The computer program can be in any computer language, such as for example machine language, C, C++, JAVA, Python, etc.

[0125] Such a computer program can take the form of a standalone application. Alternatively, such a computer program can to be integrated into a photo or video application, or even into an image or video playback application.

[0126] The computer program can be stored in a non-transient, or non-volatile, manner in a storage medium.

[0127] According to another aspect of the invention, a device is proposed comprising means configured to implement all the steps of the process according to the invention, and in particular: - the dematting process according to the invention, or - the image processing method according to the invention.

[0128] The device according to the invention can be a computer, a processor, a computer chip, etc. programmed to implement the method according to the invention, for example by executing the computer program according to the invention.

[0129] The device according to the invention can be integrated into any type of device such as a smartphone, a tablet, a computer, a calculator, a processor, a computer chip, a medical imaging device, etc.

[0130] The device according to the invention may include, in terms of technical means, at least one, or any combination of at least two, of the characteristics described above with reference to the method according to the invention, and which will not be repeated here exhaustively for the sake of brevity.

[0131] According to another aspect of the same invention, a method for acquiring an image of a scene is proposed, comprising the following steps: - Acquisition of a raw image by a camera module, - processing of said raw image by the process according to the invention, and in particular by the demosaicing process according to the invention or the image processing process according to the invention.

[0132] The processing stage can be carried out, at least in part, in the same device that carried out the raw image acquisition stage.

[0133] Alternatively, or in addition, the processing step can be carried out, at least in part, in a device other than the one that carried out the raw image acquisition step.

[0134] According to another aspect of the invention, a device, and in particular a user device, is proposed, comprising: - a camera module for acquiring a raw image of a scene, and - a device according to the invention.

[0135] In particular, the device can be a user device such as a smartphone, tablet, etc.

[0136] The user device may also include a display screen.

[0137] In particular, the device can be a user device such as a computer.

[0138] The computer-type user device may also include a display screen.

[0139] In particular, the device could be a television.

[0140] Television may also include a display screen.

[0141] In particular, the device can be: - a virtual reality headset or glasses; or - an augmented reality headset, or glasses.

[0142] The helmet, or glasses respectively, according to the invention may further comprise at least one display screen.

[0143] In particular, the device may be a medical imaging device.

[0144] In particular, the medical imaging device can be an endoscope, an ultrasound machine, etc.

[0145] The medical imaging device may also include at least one display screen.

[0146] Of course, the device according to the invention is not limited to the examples of devices that have just been given.

[0147] According to another aspect of the present invention, a vehicle comprising a device according to the invention is proposed.

[0148] The vehicle may further include a display screen, for example arranged in a passenger compartment of the vehicle, or a projector to project at least one image onto a display surface generally known as a "head-up display".

[0149] Depending on the embodiment, the vehicle can be a land vehicle, such as a car, autonomous, semi-autonomous or non-autonomous.

[0150] Depending on the embodiment, the vehicle can be a flying vehicle, such as a drone, an airplane, a helicopter, autonomous, semi-autonomous or non-autonomous.

[0151] Depending on the embodiment, the vehicle can be a maritime vehicle, such as a boat or a submarine, autonomous, semi-autonomous or non-autonomous. Description of the figures and methods of realization

[0152] Other advantages and features will become apparent upon examination of the detailed description of non-limiting embodiments and the accompanying drawings, in which: - FIGURE 1 is a schematic representation of a non-limiting example of an image sensor implementation; - FIGURES 2 and 3 are schematic representations of non-limiting examples of implementation of a dematting process according to the invention; - FIGURE 4 is a schematic representation of a non-limiting example of an image processing method according to the invention; - FIGURE 5 is a schematic representation of a non-limiting example of an embodiment of a method for acquiring an image of a scene according to the invention; - FIGURE 6 is a schematic representation of a non-limiting example of an embodiment of a device according to the invention; - Figures 7a-7c are schematic representations of non-limiting examples of embodiments of a device according to the invention; and - FIGURE 8 is a schematic representation of a non-limiting example embodiment of a vehicle according to the invention.

[0153] It is understood that the embodiments described below are by no means exhaustive. In particular, variants of the invention may be conceived comprising only a selection of the features described below, isolated from the other features described, if this selection of features is sufficient to confer a technical advantage or to differentiate the invention from the prior art. This selection includes at least one preferably functional feature without structural details, or with only a portion of the structural details if this portion alone is sufficient to confer a technical advantage or to differentiate the invention from the prior art.

[0154] In particular, all the variants and embodiments described can be combined with each other if there are no technical obstacles to this combination.

[0155] In the figures and in the rest of the description, elements common to several figures retain the same reference.

[0156] FIGURE 1 is a schematic representation of a non-limiting example of an image sensor that can be used for the acquisition of a raw image.

[0157] The image sensor 100 in FIGURE 1 has a Bayer matrix. In particular, the sensor has n rows of Bayer patterns and m columns of Bayer patterns, so that the sensor comprises in its entirety B=nm Bayer patterns. Of course, n and m are integers and can be different or equal.

[0158] Each Bayer motif, marked Bij with the l <i<n et l<j<m, comprend un nombre P de photosites monochromes, choisis parmi des photosites rouge, vert ou bleu, avec P> 2. In the example shown, in no way limiting, each Bayer Bij pattern is square in shape and comprises four monochrome photosites, namely two green V photosites arranged on one diagonal of the Bayer Bij pattern, and one red R photosite and one blue B photosite arranged on the other diagonal of the Bayer Bij pattern.

[0159] Under these conditions, the 100 sensor provides a raw image represented by a matrix of B=nm value sets. Each value set Bij is provided by the Bayer pattern Bij and comprises four values, each provided by a monochrome photosite of said pattern. Thus, in the example shown, each value set includes two values ​​for the color green, one value for the color blue, and one value for the color red.

[0160] From a raw image provided by sensor 100 in FIGURE 1, it is possible to obtain a dematrixed digital image comprising N pixels, whose definition in terms of pixels can be N = nm: in this case, the dematrixed digital image comprises N = nm pixels, each pixel being obtained from a Bayer pattern.

[0161] Alternatively, it is possible to obtain a demosaiced digital image whose definition in terms of pixels can be P=4.(nm): in this case, the demosaiced digital image comprises P=4.(nm) pixels, a Bayer pattern providing four pixels of the demosaiced digital image.

[0162] In all cases, in the case of a color image, each pixel is represented by a set of at least three values: a value for the color red, a value for the color green, and a value for the color blue.

[0163] Of course, each pixel can also include other values ​​or be represented by other values ​​such as, for example, hue values, denoted T, saturation, denoted S, and luminance, denoted L.

[0164] FIGURE 2 is a schematic representation of a non-limiting example of an embodiment of a method according to the present invention for demosaicing a raw image.

[0165] The method 200 of FIGURE 2 can be used to dematrix a raw image, denoted IMb, of a scene obtained by a camera module comprising an image sensor and an optical lens.

[0166] The image sensor can be any type of image sensor and in particular the image sensor 100 of FIGURE 1.

[0167] The 200 process can be implemented by the device used for the acquisition of the raw IMb image.

[0168] Alternatively, the 200 method can be implemented by a device other than the one used for the acquisition of the raw IMb image, said other device being local or remote from said device used for the acquisition of the raw IMb image.

[0169] The method 200 optionally includes a step 202 for determining a depth map of the scene. This depth map indicates, for each point in the scene, the distance between said point and the model camera, and in particular the distance, denoted DOS, between the optical lens of the camera module and the point in the scene.

[0170] Generally, the depth map is a matrix of values ​​indicating, for each point (x,y) in the scene, the value of the DOS distance.

[0171] The scene depth map can be determined in various ways known to those skilled in the art. In particular, the scene depth map can be measured by one or more sensors, for example, those fitted to the device used to image the scene.

[0172] Alternatively, the scene depth map can be provided by another means or process.

[0173] It should be noted that the raw IMb image can be acquired before, after or during the optional step 202 of determining the depth map.

[0174] During step 204, the optical transfer function of the camera module, and in particular of the optical lens, is used to capture the raw image is loaded. The optical transfer function may be identical for all points in the scene and independent of the scene depth. In this case, the scene depth map may not be determined.

[0175] Preferably, the optical transfer function can be a function of the scene depth. In this case, for each point in the scene, the depth of said scene at said point is determined by consulting the depth map determined in optional step 202, and then an optical transfer function is determined as a function of said depth.

[0176] The optical transfer function(s) can be predetermined, for example, during the manufacture of the optical lens or the device, or during a calibration phase of said optical lens or device. This optical transfer function(s) can be stored in the device used for acquiring the raw image, or remotely from said device and loaded during the implementation of method 200.

[0177] The optical transfer function can be the MTF or the PSF. In the following, without loss of generality, we consider the optical transfer function to be the PSF.

[0178] During step 206, process 200 performs an initialization of an image undergoing demosaicing, noted I Mdem.

[0179] This initialization can, for example, be carried out randomly, or according to a predetermined relationship.

[0180] This initialization can, for example, be achieved by assigning the same values ​​to each pixel of said image being demosaiced I Mdem.

[0181] It is recalled that the image being demosaiced IMdem is represented by a matrix of value sets, each value set representing a pixel of said IMdem image and comprising, for example, three values, one for each of the colors red, green and blue.

[0182] Following a non-limiting example, the raw image IMb comprises B=nm Bayer patterns and the image being demosaiced I Mdem comprises P=nm pixels. Each pixel is represented by a set of three values, each for one of the colors red, green, and blue.

[0183] Alternatively, the image being demosaiced IMdem can be represented by HSL values ​​respectively for hue, denoted T, saturation, denoted S, and luminance, denoted L.

[0184] Process 200 includes a phase 210 of demosaicing the raw image, IMb.

[0185] The demosaicing phase 210 includes a step 212 of convolution of the image being demosaicated IMdem provided by the previous iteration of the demosaicing phase 210 with the PSF function of the optical lens loaded by step 204, for example according to the relation: PSF * IM(k - l) demwith IM(kl)dem the image being demosaiced obtained in the previous iteration.

[0186] Then, in step 214, a difference term is calculated between the raw image and the image being demosaicing, previously convolved with the PSF, for example according to the relationship: \PSF * IM(J — l) dem — IM b \ 2

[0187] For the calculation of the difference term, the image being demosaiced IMdem is preferably represented in RGB.

[0188] Then, in step 216, a smoothing term is calculated based on the image being demosaiced provided in the previous iteration, for example according to the relation: with IM(kl)dem the image being demosaiced obtained in the previous iteration, OP L a predetermined smoothing operator, a weighting coefficient. As mentioned above, the smoothing operators OPi and the weighting coefficients are determined beforehand, for example during a process initialization step 200.

[0189] For calculating the smoothing term, the image being demosaiced (IMdem) is preferably represented in HSL. To achieve this, an HSL representation of the image being demosaiced (IMdem) can be obtained from an RGB representation during a conversion step. not shown. This conversion step can be performed at any time before step 216.

[0190] In step 218, an error term is calculated based on the deviation term and the smoothing term, for example according to the following relationship:

[0191] In step 220, the error E is compared to a predetermined error threshold, denoted S.

[0192] If the error E is less than or equal to said threshold S, then the image being demosaiced is stored as a demosaiced image IMf during a step 222. The demosaicing process 200 can end.

[0193] If the error E is greater than said threshold S, then the image being demosaicing is modified during a step 224. The modification of the image being demosaicing IMdem can be carried out according to any known method.

[0194] Depending on the embodiment, the modification of the image during IMdem demosaicing can be carried out according to a random or semi-random algorithm or rule.

[0195] Depending on the embodiment, the modification of the image during IMdem demosaicing can be carried out according to an algorithm, or a rule, taking into account the evolution of, or a derivative of, the error value or a function representing said error.

[0196] Depending on the embodiment, the modification of the image during IMdem demosaicing can be carried out according to the Levenberg-Marquardt algorithm.

[0197] Step 224 then provides a modified image being demosaiced to be used for the next iteration of the demosaicing phase 210. A new iteration of the demosaicing phase 210 can then be carried out.

[0198] In the example described, steps 212-218 can be carried out at the same time, or in turn as shown in FIGURE 2.

[0199] FIGURE 3 is a schematic representation of another non-limiting embodiment of a method according to the present invention for demosaicing a raw image.

[0200] The 300 method of FIGURE 3 can be used to dematrix a raw image, denoted IMb, of a scene obtained by a camera module comprising an image sensor and an optical lens.

[0201] The image sensor can be any type of image sensor and in particular the image sensor 100 of FIGURE 1.

[0202] Method 300 can be implemented by the device used for acquiring the raw IMb image. Alternatively, method 300 can be implemented by a device other than the one used for acquiring the raw IMb image, said other device being local or remote from said device used for acquiring the raw IMb image.

[0203] Process 300 does not include steps 202-206 of process 200 in FIGURE 2.

[0204] In a different way, process 300 includes a step 302 of correcting aberrations in the raw image IMb. To do this, a correction is applied to said raw image in order to obtain a corrected raw image, denoted IMb, dec.

[0205] Step 302 may include a 304 step for lateral chromatic aberration correction, and / or a 306 step for geometric aberration correction.

[0206] The raw image corrected IMb, dec according to one of the following relationships: IMb, dec = FCACL * IMb; IMb, dec = FACL' 1 * IMb; IMb, dec = FCAG * IMb; IMb, dec = FAG -1 * IMb *; IMb, dec = FCAG * (FCACL * IMb), or IMb, dec = FAG' 1 *( FACL' 1 * IMb ) .

[0207] The functions FACL, FCACL, FAG and FCAG can be determined as described above.

[0208] Preferably, the corrected raw image IMb,dec can be determined according to the relationships IMb,dec = FCACL * IMb; IM b ,dec = FACL' 1 * IMb * ; or according to the relations IMb, dec = FCAG * (FCACL * IMb), or IMb, dec = FAG -1 *( FACL' 1 * IMb ) . So the corrected raw image always includes lateral chromatic aberration correction, and optionally geometric aberration correction.

[0209] Process 300 includes step 206 of initializing the image during IMdem demosaicing, as described above.

[0210] In process 300, this step 206 can be carried out at the same time as, after or before step 302.

[0211] Then, process 300 includes a demassing phase 310.

[0212] The demosaicing phase 310 includes a step 314, during which a difference term is calculated between the raw image corrected in step 302 and the image being demosaicing provided by the previous iteration of the demosaicing phase 310. For example, the difference term can be calculated according to the following relationship:

[0213] For the calculation of the difference term, the image being demosaiced can be represented in RGB form.

[0214] Then, the demosaicing phase 310 includes the steps 216-224 described above with reference to process 200 in FIGURE 2.

[0215] In the example described, steps 314, 216-218 can be carried out at the same time, or in turn as shown in FIGURE 3.

[0216] FIGURE 4 is a schematic representation of a non-limiting example embodiment of a method according to the present invention for processing an image according to the invention.

[0217] The process 400 of FIGURE 4 includes a step 402 which performs demosaicing of a raw image of a scene. This step 402 can perform the demosaicing of the raw image IMb by the demosaicing process according to the invention, and in particular any one of the processes 200 or 300 of FIGURES 2 or 3. Step 402 provides a demosaicing image, denoted IMf.

[0218] Process 400 further includes a step 404 for processing the demosaiced IMf image. This processing step 404 can perform any type of digital processing on the demosaiced IMf image.

[0219] For example, step 404 can perform the correction of an optical aberration, such as a chromatic aberration, in particular displacement aberration, a geomatic aberration, etc.

[0220] For example, step 404 can achieve image sharpness improvement, color enhancement, brightness modification, etc.

[0221] Step 404 provides an enhanced image labeled IM a

[0222] FIGURE 5 is a schematic representation of a non-limiting example embodiment of an image acquisition method according to the present invention.

[0223] The process 500 of FIGURE 5 includes a step 502 which acquires a raw IMb image with an imaging device comprising a camera module.

[0224] Process 500 includes, after step 502, step 402 which performs a demosaicing of the raw IMb image as described above with reference to process 400.

[0225] Optionally, process 500 may further include, after step 402, step 404 of processing the demosaiced image provided by step 402, as described above with reference to process 400.

[0226] FIGURE 6 is a schematic representation of a non-limiting example embodiment of a device according to the present invention.

[0227] The 600 device in FIGURE 6 can be used to dematrix a raw IMb image of a scene.

[0228] The device 600 of FIGURE 6 can be used to implement a process according to the invention, and in particular the process 200 of FIGURE 2 or the process 300 of FIGURE 3.

[0229] The 600 device optionally includes at least one 602 sensor for determining the depth map of the scene. Such a 602 sensor can be a LiDAR sensor, a time-of-flight camera, etc. This 602 sensor can be configured, in particular, to perform step 202 of process 200.

[0230] Device 600 includes a module 604 for initializing the image during demosaicing, IMdem. This module 604 can be configured, in particular, to perform step 206 of process 200.

[0231] Device 600 includes at least one module 606 to perform: - either an aberration correction in the raw image: in this case, module 606 can in particular be configured to perform step 302 of process 300; - either a convolution of the image during IMdem demosaicing with an optical transfer function, for example the PSF: in this case the module 606 can in particular be configured to carry out step 212 of process 200.

[0232] Device 600 includes a module 608 to calculate the error value, denoted E, and compare said error value to a predetermined threshold value, denoted S. This module 608 can in particular be configured to carry out steps 214-220 of process 200, or steps 314, 216-220 of process 300.

[0233] Device 600 includes a module 610 for modifying the image during IMdem demosaicing. This module 610 can be configured, in particular, to perform step 224 of process 200 or process 300.

[0234] The 600 device includes a 612 module for storing the image being demosaiced (IMdem) as a demosaiced image (IMf). This 612 module can in particular be configured to perform step 222 of process 200 or process 300.

[0235] Optionally, the 600 device can include a 614 module for digitally processing the demosaiced IMf image. This 614 module can be configured, in particular, to perform step 404 of process 400 or process 500.

[0236] Optionally, device 600 can include a camera module 616 to acquire a raw IMb image, and in particular step 502 of process 500. Of course, this camera module 616 is optional because the device in which device 600 is integrated may already have a camera module.

[0237] At least one of the 604-614 modules can be a module independent of the other 604-614 modules.

[0238] At least two of the 604-614 modules can be integrated within a single module. In particular, the 604-614 modules can be integrated within a 618 computing unit.

[0239] At least one of the 604-614 modules can be a hardware module, such as a processor, an electronic chip, etc.

[0240] At least one of the 604-614 modules can be a software module, such as a computer program.

[0241] At least one of the 604-614 modules can be a combination of at least one software module and at least one hardware module.

[0242] In particular, at least one of the 604-614 modules can be integrated into an electronic chip, or into an application installed in a user device.

[0243] In particular, the 618 computing unit can be, or can be integrated, into an electronic chip, or into an application installed in a user device.

[0244] FIGURE 7a is a schematic representation of a non-limiting example embodiment of a device according to the present invention.

[0245] The apparatus 700 of FIGURE 7a includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0246] The apparatus 700 of FIGURE 7a may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0247] In the example shown in FIGURE 7a, device 700 is a smartphone, or a tablet, comprising device 600 from FIGURE 6.

[0248] Optionally, the device 700 can also include a display screen 702, optionally equipped with a sensing surface 704, for example capacitive.

[0249] Of course, the 700 device may include other components than those indicated above.

[0250] FIGURE 7b is a schematic representation of another non-limiting embodiment of a device according to the present invention.

[0251] The apparatus 710 of FIGURE 7b includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0252] The apparatus 710 of FIGURE 7b may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0253] In the example shown in FIGURE 7b, device 710 is: - a virtual reality (VR) headset or glasses, or - an augmented reality headset, or glasses; including the 600 device of FIGURE 6.

[0254] Optionally, the 710 device may also include a 712 display screen, for example in / on / under a visor of said 710 helmet.

[0255] Of course, the 710 helmet may include other components than those listed above.

[0256] FIGURE 7c is a schematic representation of a non-limiting example embodiment of a device according to the present invention.

[0257] The apparatus 720 of FIGURE 7c includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0258] The apparatus 720 of FIGURE 7c may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0259] In the example shown in FIGURE 7c, device 720 is a medical imaging device, such as an endoscope, an ultrasound machine, etc.

[0260] Optionally, the device 720 can also include a display screen 722, optionally equipped with a sensing surface 724, for example capacitive.

[0261] Of course, the 720 medical imaging device may include other organs than those indicated above.

[0262] FIGURE 8 is a schematic representation of a non-limiting example embodiment of a vehicle according to the present invention.

[0263] The vehicle 800 of FIGURE 8 includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0264] The vehicle 800 of FIGURE 8 may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0265] In the example shown in FIGURE 8, vehicle 800 is a land vehicle, in particular a car, comprising device 600 of FIGURE 6.

[0266] Optionally, the vehicle 800 may also include a display screen 802, optionally equipped with a sensing surface 804, for example capacitive, arranged in the passenger compartment of the vehicle 800.

[0267] Of course, the 800 vehicle may include other components than those indicated above.

[0268] Of course, the invention is not limited to the examples that have just been described.

Claims

DEMANDS 1. A method (200;300) for demosaicing a raw image (IMb) of a scene, acquired by a camera module (616) comprising an image sensor, said method (200;300) comprising at least one iteration of a demosaicing phase (210;310) iteratively modifying an image (IMdem), called the image being demosaicing, and comprising the following steps: - calculation (218) of an error value (E) corresponding to the sum of: ■ a first term, called the gap term, relating to a difference between said raw image (IMb) and the image being demosaicing (IMdem) obtained in the previous iteration of said demosaicing phase (210;310), and ■ a second term, called the smoothing term, a function of said image during demosaicing (I Mdem); - when said error value (E): ■ satisfies a predetermined threshold value, storage (222) of said image being demosaiced (IMdem) as a demosaiced image (IMf), and ■ does not satisfy said threshold value, modification (224) of said image during demosaicing (IMdem) in order to decrease said error value and execution of a new iteration of said demosaicing phase (210;310).

2. Method (200;300) according to the preceding claim, characterized in that the modification (224) of the image during demosaicing (IMdem) is carried out according to the Levenberg-Marquardt algorithm.

3. Method (200;300) according to any one of the preceding claims, characterized in that the demosaicing phase (210;310) is applied to the entire raw image.

4. A method (200;300) according to any one of the preceding claims, characterized in that it comprises: - prior to the demosaicing phase, a step of dividing the raw image into several regions, the demosaicing phase (210; 310) being applied individually to at least one, and in particular to each, of the said regions; and - a step of reconstructing the demosaiced image (IMf) with at least one region of said demosaiced image.

5. Method (200) according to any one of the preceding claims, characterized in that the demosaicing phase (210) comprises a convolution (212) of the image being demosaicing (IMdem) with an optical transfer function associated with the camera module (616), and in particular with an optical lens of said camera module (616), the difference term being calculated between the result of said convolution and the raw image.

6. Method (200) according to the preceding claim, characterized in that it comprises, before the demosaicing phase (210) a step (202) of determining a depth map of the scene, the optical transfer function being determined as a function of said depth map.

7. A method (200) according to any one of claims 5 or 6, characterized in that the deviation term is determined according to the following relationship: | PS F*IM ( k- 1 )dem - IMb | 2 with IM(kl)dem the image being demosaiced obtained in the previous iteration, and IMb the raw image.

8. Method (300) according to any one of claims 1 to 4, characterized in that it comprises a step (304) of correction, in the raw image (I Mb) , of the lateral chromatic aberration of the camera module, and in particular of the optical lens of said camera module, the deviation term being calculated between the image being demosaicing (IMdem) provided at the previous iteration of the demosaicing phase and the raw image (IMb, dec) provided by said correction step (302).

9. A method (300) according to any one of claims 1 to 4 or 8, characterized in that it comprises a correction step (306). - in the raw image, or - in the raw image previously corrected for lateral chromatic aberration; for the geometric aberration of the camera module, and in particular of the optical lens of said camera module, the difference term being calculated between the image being demosaicing provided to the previous iteration of the demosaicing phase and the raw image provided by said correction step.

10. A method (300) according to any one of claims 8 or 9, characterized in that the deviation term is determined according to the following relationship: | IM (k- l )dem - I Mb,dec | 2 with IM(kl)dem the image being demosaiced obtained in the previous iteration, and IMb,dec the raw image having undergone the correction of lateral chromatic aberration and / or the correction of geometric aberration.

11. Method (200;300) according to any one of the preceding claims, characterized in that the smoothing term is a sum of several operators, possibly weighted, taking as input the image being demosaiced provided by the previous iteration of the demosaicing phase (210;310).

12. A method (200;300) according to the preceding claim, characterized in that the smoothing term is determined according to the following relationship: with IM(kl)dem the image being demosaiced obtained in the previous iteration, OP L a predetermined smoothing operator, a weighting coefficient.

13. Image processing method (400) comprising the following steps: - demosaicing (402) of a raw image (IMb) by the method (200;300) according to any one of the preceding claims, yielding a demosaiced image (IMf); and - digital processing (404) of said demosaiced image (IMf).

14. Computer program comprising executable instructions which, when executed by a computing device, implement all the steps of the process (200;300;400) according to any one of the preceding claims.

15. Device (600) comprising means configured to carry out all the steps of the process (200;300;400) according to any one of claims 1 to 13.

16. Method (500) for acquiring an image of a scene comprising the following steps: - acquisition (502) of a raw image (IMb) by a camera module (616), - processing (402,404) of said raw image by the process (200;300;400) according to any one of claims 1 to 13.

17. Device (700;710;720) comprising: - a camera module (616) to acquire a raw image (IMb) of a scene; and - a device (600) according to claim 15.

18. Vehicle (800) comprising: - a camera module (616) to acquire a raw image (IMb) of a scene; and - a device (600) according to claim 15.

Images