Method for enriching a raw image of a scene, and computer program, device and system implementing such a method

Abstract

The invention relates to a method (200) for enriching a raw image (IMBP) of a scene on the basis of one or more secondary raw images (IMBS1-IMBSl) of the scene, the method (200) comprising an enrichment phase (210) for enriching the primary raw image (IMBP) and providing an enriched raw image (IMBE), the enrichment phase (210) comprising, for at least one pattern, referred to as the target pattern, of the primary raw image (IMBP), the following steps: - identifying (212), in at least one secondary raw image (IMBS1-IMBSk), a pattern corresponding to the target pattern; and adjusting (216,218) at least one value of the target pattern according to one or more values of identified corresponding patterns. The invention also relates to a method for demosaicing a raw image, to an image processing method, to an image acquisition method, to a computer program, to a device, to a system and to a vehicle implementing such a demosaicing method.

Description

DESCRIPTION Title: Method for enhancing a raw image of a scene, computer program, device and apparatus implementing such a method

[0001] The present invention relates to a method for obtaining an enhanced raw image of a scene from several raw images of said scene. It also relates to a computer program, a device, and an apparatus implementing such a method.

[0002] The field of the invention is the field of signal processing captured by an image sensor of a camera module in order to obtain an image. State of the art

[0003] Demosaicing, also called debayering or demosaicing, is a process for processing the raw signal from a digital camera sensor. It involves interpolating the data from each of the monochrome photosites, for example red, green, and blue, that make up the image sensor to obtain a tri-color value for each pixel of the image in the case of an RGB image, for example.

[0004] The image sensor, or photographic sensor, of a camera module typically comprises red, blue, or green photosites. Each photosite reacts to the light it receives and converts it into an electrical charge, based on the principle of a photoelectric cell. The most well-known arrangement of photosites is called the Bayer matrix. This matrix is ​​formed by a multitude of photosite patterns, each pattern generally square and comprising two green photosites arranged diagonally, and one red and one blue photosite arranged on the opposite diagonal. The signal received from such an image sensor is generally dematrixed, using widely known techniques, to estimate, for each pixel of the image, the value of each of the red, green, and blue components of that image.

[0005] In an image scene, a fine pattern repeating an alternation of brightness, such as a bird's feather, and whose spatial periodicity closely matches or is specifically related to that of the color detection alternation pattern, generally within the Bayer matrix, can also produce periodic over- and underexposures of certain colored pixels within the pattern relative to others. A gray feather can thus appear with false iridescence reminiscent of the colors of the rainbow, which certainly produces a certain artistic effect, but is inaccurate in terms of accurately reproducing the real scene.

[0006] These problems can be partially solved by applying demosaicing rules involving neighboring pixels, which minimize the effects described above. However, existing solutions are generally not completely satisfactory at the edges of objects, or when the effective size of the object projected onto the detection matrix is ​​of a comparable order of magnitude to that of the detection matrix. Similarly, current demosaicing solutions implicitly favor linear patterns in certain directions, which can lead to the appearance of imaginary lines, distorted object outlines, or even make them difficult to observe due to these artifacts, and so on.

[0007] One objective of the present invention is to remedy at least one of the aforementioned drawbacks.

[0008] Another aim of the invention is to provide a solution for obtaining a raw image allowing a more precise and less erroneous, and more faithful, reproduction of the real scene. Description of the invention

[0009] The invention proposes to achieve at least one of the aforementioned goals by a method of enriching a raw image of a scene, called the main raw image, from at least one other raw image of said scene, called the secondary raw image, each of said main and secondary images being: - acquired by at least one camera module comprising an image sensor having a matrix of monochrome photosite patterns; - represented by a matrix of value patterns, each value pattern being provided by a photosite pattern of said image sensor and comprising a value provided by each monochrome photosite of said photosite pattern; said process comprising an enrichment phase of the main raw image providing an enriched raw image, said enrichment phase comprising, for at least one pattern, said target pattern, of said main raw image, the following steps: - identification, in at least one secondary raw image, of a pattern corresponding to said target pattern; and - adjusting at least one value of said target pattern based on corresponding pattern value(s) identified.

[0010] Thus, the invention proposes to obtain a raw image, that is, an image not yet demosaiced, richer than a raw image that can be obtained by a camera module. To this end, the invention proposes to combine a primary raw image, that is, an undemosaiced image, with part or all of at least one secondary raw image, that is, an undemosaiced image, to obtain the enriched raw image.

[0011] It should be noted that the enrichment step proposed by the invention is carried out before demosaicing, if applicable, of the enriched raw image.

[0012] Thus, the enriched raw image constitutes a more precise and richer representation of the scene compared to each of the primary and secondary raw images, and allows obtaining a more precise and richer dematrixed image of the scene, and therefore more faithful to the real scene.

[0013] A raw image is defined as a set of values ​​provided by an image sensor in an imaging module, such as a CMOS sensor, a CCD sensor, or any other type of sensor. Specifically, a raw image is an image composed of a multitude of image patterns, each pattern comprising the monochrome component values ​​provided by the monochrome photosites that make up that pattern.

[0014] In some embodiments, the image sensor includes a Bayer matrix, said Bayer matrix comprising B Bayer patterns, each Bayer pattern Bi potentially comprising P monochrome photosites. In this case, the raw image IMb can comprise a matrix of B value patterns, each value pattern Bi being provided by a Bayer pattern Bi of the image sensor. Each value pattern Bi comprises P values, each value being provided by a monochrome photosite of the Bayer pattern Bi providing said value pattern.

[0015] In some embodiments, a Bayer pattern may comprise P=4 photosites, for example, 2 green photosites, one red photosite, and one blue photosite. In other embodiments, the Bayer pattern may be square, with the two green photosites arranged on one diagonal of the square and the red and blue photosites arranged on the other diagonal.

[0016] The demosaiced image can be represented by N sets of values, each set of values ​​corresponding to a pixel of the demosaiced image. Each set of values ​​in the demosaiced image can contain C values. For example, in the case of an RGB image, each set of values ​​can contain three values, one for each of the colors red, green, and blue.

[0017] Following examples of implementation, for a raw image comprising B patterns, demosaicing can provide a demosaiced image comprising N pixels such that N = B: in this case each Bayer pattern provides one pixel of the demosaiced image.

[0018] Following other realization examples, for a raw image comprising B value patterns, demosaicing can provide a demosaiced image comprising N pixels such that N = P*B: in this case each Bayer pattern provides P pixels of the demosaiced image.

[0019] For at least one target pattern in the primary raw image, the adjustment can concern at least one, and in particular each, value provided by said target pattern. In particular, the adjustment can concern the value of at least one, and in particular each, monochrome component, that is, the value provided by each photosite of the pattern.

[0020] Depending on embodiments, the adjustment step may include, for at least one monochrome component of said target pattern, the following steps: - determination, for said monochrome component, of a maximum value based on the value of this monochrome component in said target pattern and in the corresponding pattern(s), and - assignment of said maximum value to said monochrome component in said target pattern.

[0021] Thus, in the enriched raw image, the monochrome component will have the maximum value, that is to say, more information concerning the point of the scene corresponding to this pattern.

[0022] The maximum value can be determined in different ways.

[0023] According to a first embodiment, the maximum value can correspond to the largest of the values ​​provided by the target pattern and the corresponding pattern(s). In other words, the maximum value can be determined according to the following relationship: VAL(McibieC P )= Max {(VAL|M k Cj)} with : - Mdbie = Mij, the target pattern in the main raw image: the main raw image potentially contains as many target patterns as there are patterns in the image sensor. Thus, if the image sensor has a matrix of 1000*1000 patterns, then there are potentially 1000 2 target patterns; -CP the p ième monochrome component, p>l, for example p=4 in the case of a Bayer pattern which has two green monochrome components, one red monochrome component and one blue monochrome component; - Mk the k ième pattern corresponding to the target pattern and originating from k ième secondary image, with k>l. For example k=10 in the case where 10 secondary images are used.

[0024] According to a second embodiment, the maximum value can correspond to a maximum value reached by a curve drawn with all the values ​​provided by the target pattern and the corresponding pattern(s).

[0025] In this case, the step of determining the maximum value may include the following operations: -plotting a curve with the values ​​from the patterns; - determination of a theoretical maximum value of said curve.

[0026] The curve can represent the value provided by each of the target and secondary motifs for the component under consideration, as a function of: - an index of each image from which said motif originates, - a spatial deviation of the pattern from the photosite providing said value, modulo the step between two photosites of the same nature, in one direction or in two directions: in the latter case we obtain a curve which is in reality a surface; - the moment at which the raw image from which said motif originates was taken, and / or - the position of this pattern in the image sensor. The plotted curve can be one-dimensional, or preferably two-dimensional, for example, when plotted based on the spatial distance between the photosite providing the value and adjacent photosites of the same type. Once the curve is plotted, through interpolation / extrapolation, and more generally through calculation, it is possible to determine the theoretical maximum value that the curve can reach, and to take this theoretical maximum value as the value for the monochrome component of the target pattern.

[0027] If it is not possible to determine the maximum value of the curve, it is still possible, as a safety measure, to choose as the maximum value the largest of the values ​​provided by the patterns, as stated above with reference to the first embodiment of the maximum value determination step. Therefore, it is possible to combine the first and second embodiments described for the maximum value determination step.

[0028] The step of determining the maximum value may further include a step of filtering the plotted curve to eliminate the noise present in said curve.

[0029] Such a filtering step thus makes it possible to provide a maximum value that is more accurate and more faithful to the scene.

[0030] According to a general definition, this filtering aims to remove one or more components, generally at high spatial frequencies.

[0031] For at least one target pattern, the identification of said target pattern in a secondary image can be achieved by any suitable technique.

[0032] According to embodiments, for at least one target pattern, the identification of said target pattern in a secondary raw image can be carried out by calculating the correlation between the primary raw image and said secondary raw image.

[0033] The correlation calculation can be performed on all of the said images.

[0034] Alternatively, the correlation calculation can be performed on a limited area of ​​the images, for example, an area centered on the target pattern, or an area that begins or ends with the target pattern, and more generally, an area encompassing the target pattern. The limited area can have a predetermined size smaller than the total image size, for example, a size between 10x10 and 50x50 patterns, and in particular 30x30 patterns. The size of the area can be fixed or variable, determined according to the displacement amplitude between the images.

[0035] Following an example implementation, the primary raw image, or the area containing the target pattern, can be spatially offset relative to the secondary raw image, and a correlation score is calculated for each offset. This operation is repeated iteratively until a maximum correlation score is obtained, or a value exceeding a predetermined matching threshold. Knowing the position of the target pattern in the primary raw image, or in the bounded area, the corresponding pattern is then identified in the secondary raw image.

[0036] The identification of the target pattern can alternatively be achieved by other techniques, for example the Lucas-Kanade method, to obtain the position of the secondary raw image relative to the primary raw image.

[0037] In some embodiments, for at least one target pattern, the identification of said target pattern in a secondary raw image can be performed based on a monochrome component of said pattern. In this case, for example, the correlation calculation can be performed based solely on the values ​​of said monochrome component.

[0038] According to some embodiments, for at least one target pattern, the identification of said target pattern in a secondary raw image can be performed based on a parameter deduced from at least two, in particular from all the monochrome components of said pattern. In this case, for example, the correlation calculation can be performed based on the values ​​of said parameter. Such a parameter could be brightness, an average of the monochrome components, etc.

[0039] As mentioned above, the enrichment phase can be performed for at least one pattern, called the target pattern, of the main raw image.

[0040] More generally, the enhancement phase can be performed for each pattern in the primary raw image, considered as the target pattern. Alternatively, the enhancement phase can be performed only for patterns located in certain areas of the primary raw image, particularly areas where there is contrast beyond a certain predetermined threshold.

[0041] The enrichment phase can be carried out for at least two motifs in turn or in parallel.

[0042] The enrichment phase can be performed for a single pattern of the main raw image at a time, or simultaneously for several patterns, or even all patterns, of the main raw image.

[0043] The enrichment phase can be carried out to: - only one pattern at a time, or - an area comprising several patterns.

[0044] In general, the process according to the invention may comprise several iterations, either sequentially or simultaneously, of the enrichment phase, each iteration being carried out to: - only one pattern at a time, or - several patterns, or an area containing several patterns; of the main raw image.

[0045] Of course, the enrichment phase can be performed with a single secondary raw image. This allows for the enrichment of the primary raw image quickly and with minimal computational resources.

[0046] Alternatively, the enrichment phase can be performed with several secondary raw images. This improves the enrichment process and results in a more detailed raw image.

[0047] Preferably, a spatial offset may exist between the primary raw image and at least one, and in particular each, secondary raw image, for at least a part of the scene.

[0048] In other words, for at least this part of the scene, there is a non-zero spatial offset between the primary raw image and the, or each, secondary raw image.

[0049] Spatial offset can affect only a portion of the scene. Alternatively, spatial offset can exist for the entire scene.

[0050] According to some embodiments, a spatial offset between a secondary raw image and the primary raw image, for at least a part of the scene, may be due to a moving object located in said part of the scene on at least one of said raw images.

[0051] In this case, the acquisition position of said raw images may be the same or different.

[0052] Depending on the embodiment, a spatial offset between a secondary raw image and the primary raw image, for at least a part of the scene, may be due to movement of the imaging device, or to the use of imaging modules positioned in different locations.

[0053] For example, two acquisition devices positioned in different locations can be used.

[0054] For example, two imaging modules equipping the same acquisition device and positioned in different locations within said acquisition device can be used, as is generally the case in a Smartphone with multiple optical lenses.

[0055] In general, for at least one secondary raw image, spatial offset can be achieved by using different viewing angles, imaging positions, etc. for the acquisition of the primary and secondary raw images.

[0056] According to advantageous embodiments, for at least one secondary raw image, the spatial shift can be obtained by controlling an image stabilizer equipping the imaging device to introduce said spatial shift.

[0057] In this case, the primary and secondary raw images are acquired from the same position for the imaging module. The image stabilizer on the imaging device is controlled to introduce the spatial offset. In other words, the stabilizer is repurposed from its original function to introduce a spatial offset between the primary and secondary raw images. This allows the same device, and the same camera module, to be used in the same position for acquiring both the primary and secondary raw images.

[0058] This embodiment allows for the rapid and simple, even completely transparent acquisition of the primary and secondary raw images, and therefore a more ergonomic implementation of the present invention.

[0059] For example, during the acquisition of said primary and secondary raw images, an offset in the positioning of the stabilizer can be added to the stabilization command applied to the image stabilizer to obtain an offset between the primary raw image and the secondary image, or each secondary raw image. Furthermore, optionally, this offset can be varied over time along at least one, and preferably both, axis of repetition of the imagesensor photosite patterns, synchronously with image acquisition, so as to obtain an offset in the pattern matrix from one image to the next. Optionally, but preferably, the spatial offset can be equal to a fraction of the pattern pitch in at least one direction of pattern repetition in the imagesensor.

[0060] According to embodiments, when several secondary IMBSI-IMBSK images can be acquired with different spatial offsets di-dx, preferably, said raw secondary IMBSI-IMBSK images can be acquired in an order other than the ascending order, or the descending order, of spatial offsets, to avoid any synchronization of said spatial offsets with a movement of an object in the scene.

[0061] For example, secondary raw IMBSI-IMBSK images can be acquired in a random order.

[0062] Following another example, the raw secondary IMBSI-IMBSK images can be acquired in a predetermined order different from ascending or descending order, and more generally different from a rectilinear order, or trajectory, or more generally according to a spatial offset different from a displacement of constant amplitude and / or sense and / or direction between each successive acquisition of the raw secondary images.

[0063] For example, assuming the use of ten secondary IMBSi-IMBSio images, each with a spatial offset from the main image, respectively di-dio, with di <d2, ..., dg<dio, alors il est préférable de prendre les images brutes secondaires IMBSi-IMBSio dans un ordre différent de l'ordre allant de 1 à 10 ou de 10 à 1. Par exemple, il est possible de prendre lesdites images dans un ordre aléatoire, dans l'ordre suivant : IMBSi, IMBSio, IMBS2, IMBS9, IMBS3, IMBSs, IMBS4, IMBSe et IMBSs. Bien entendu, cet ordre est donné à titre d'exemple non limitatif. De plus, ce qui vient d'être décrit en ce qui concerne l'ordre de prise des images peut être appliqué dans la direction u et / ou dans la direction v, et / ou dans les deux directions à la fois.

[0064] According to some embodiments, for at least one secondary image, the spatial offset between the main raw image and the secondary raw image, and in particular the offset of the secondary image relative to the main image, can be known in advance.

[0065] For example, the secondary raw image can be acquired with a known spatial offset. In other words, in this case, a spatial offset is applied before the secondary raw image is acquired. This offset can be applied and determined in various ways, which are described later.

[0066] According to embodiments, for at least one secondary raw image, the spatial offset between the primary raw image and the secondary raw image, and in particular the offset of the secondary image relative to the primary image, can be measured with at least one sensor, such as for example an inertial measurement unit.

[0067] For example, the device used to capture the primary raw image and the secondary raw image may be equipped with a displacement sensor configured to measure the displacement of said device between the acquisition position of the primary raw image and the acquisition position of the secondary raw image.

[0068] According to embodiments, for at least one secondary raw image, the spatial offset between the main image and the secondary image can be determined by calculation / analysis of said main and secondary raw images, for example during a registration step.

[0069] According to embodiments, for at least one secondary raw image, the spatial offset between said secondary raw image and the primary raw image may be less than the spatial repetition step of the photosite patterns in the image sensor.

[0070] In other words, if, in the direction u of the image sensor, the spatial repetition step of the image sensor patterns is PASu, then the offset between the secondary raw image and the primary raw image in said direction u can preferably be: of <PASu (Cl) In particular, 0 < of. Alternatively, or in addition, if, in the direction v of the image sensor perpendicular to the direction u, the spatial repetition step of the image sensor patterns is PASv, then the offset dv between the secondary raw image and the primary raw image in said direction v can preferably be: dv <PASv (C2) In particular, 0 < dv.

[0071] Depending on the embodiment, it is possible to use secondary raw images respecting only one of the Cl and C2 relationships above.

[0072] Depending on the embodiment, it is possible to use secondary raw images of which at least one respects relation Cl, and at least one respects relation C2. Depending on the embodiment, it is possible to use secondary raw images of which at least one respects relation Cl, at least one respects relation C2, and at least one respects both relations Cl and C2.

[0073] The, or at least one, secondary raw image may be an image captured by another camera module, in particular one located at a different position than the camera module used for the acquisition of the primary raw image.

[0074] In this case, it is possible to implement the invention even for a static scene, that is to say a scene including no moving objects.

[0075] Alternatively, or in addition, the, or at least one, second image may be an image captured by the same camera module at a different time.

[0076] In this case, for a static scene, it is preferable to change the imaging position between the primary raw image and the secondary raw image.

[0077] According to embodiments, the process according to the invention may further include, for at least one secondary raw image, a step of registering said secondary raw image with the primary raw image.

[0078] This registration step facilitates other steps in the process, particularly the identification, in the secondary image, of the pattern corresponding to the target pattern in the primary raw image. For example, this registration step corrects spatial discrepancies between images caused by movement of the imaging device or the presence of moving objects in the scene.

[0079] Such a registration step can be carried out using known techniques, for example by relying on several objects of interest appearing on said images.

[0080] According to another aspect of the same invention, an image demosaicing process is proposed comprising the following steps: - obtaining a raw image enhanced by the process according to the invention; and - demosaicing of said raw enriched image to obtain a demosaicated image, noted IMD in the following without loss of generality.

[0081] The dematting stage can be carried out using any known and suitable technique.

[0082] Depending on the embodiment, the demosaicing step can provide one image pixel for each pattern of the enriched raw image.

[0083] In this case, each pattern provides the values ​​of a single pixel. For example, when the image is an RGB image and each pattern is a Bayer pattern, each pattern provides a set of three values ​​(VR, VG, VB) respectively for the monochrome red, green, and blue components.

[0084] In this case, if the sensor includes an NxM pattern matrix, the demosaiced IMD image obtained after demosaicing the enriched raw image will have a definition of NxM, that is to say an image of NxM pixels.

[0085] Depending on the embodiment, the demosaicing step can provide one image pixel for each monochrome component of each pattern, or for each photosite of each pattern, of the enriched raw image.

[0086] In this case, each pattern in the enhanced raw image provides the values ​​of as many pixels as there are photosites, or monochrome components, in that pattern. For example, when the image is an RGB image and each pattern is a Bayer pattern comprising 4 photosites, each pattern provides: - four sets of three values ​​(VR, VG, VB), and therefore four image pixels, respectively for the monochrome red, green, and blue components; or -three sets of three values ​​(VR,VG,VB), and therefore three image pixels, respectively for the monochrome red, green and blue components;

[0087] In this case, according to a general definition, if the sensor comprises an NxM pattern matrix, the demosaiced image (IMD) obtained after demosaicing the enriched raw image will have a resolution of 4xNxM, i.e., an image of 4xNxM pixels, or 3xNxM, i.e., an image of 3xNxM pixels.

[0088] In this case, for each monochrome component of each pattern in the enhanced raw image, we obtain a pixel for which: - the value of said monochrome component corresponds to the value of said monochrome component in said pattern; - the value of each other monochrome component is estimated / interpolated from the value of said other monochrome component of said pattern.

[0089] Depending on the embodiment, the demosaicing step may include the following steps: - determination of a weighting map from the enriched raw image; and - deduction, for each monochrome component of the pattern, of an image pixel according to said weighting map.

[0090] Based on examples of implementation, the weighting map can be: - a brightness map; and / or - an intensity map, specifically for each individual monochrome component, or for each individual monochrome photosite: in this case, we obtain as many weighting maps as there are monochrome components in an image sensor pattern, or monochrome photosites in an image sensor pattern; and / or - a color chart, and / or - a saturation map. For example, the weighting map may have a higher resolution than the resolution of each raw image.

[0091] In this case, for each monochrome component of each pattern, we obtain a pixel for which: - the value of said monochrome component corresponds to the value of said monochrome component in said pattern; - the value of each other monochrome component is estimated / interpolated from the value of said other monochrome component of said pattern weighted by the weighting map.

[0092] According to another aspect of the same invention, an image processing method is proposed comprising the following steps: - obtaining a raw image enhanced by the process according to the invention, or a demosaiced image by the process according to the invention; and - digital processing of said raw image enhanced, or of said demosaiced image.

[0093] The digital processing stage can perform any type of processing of the enhanced raw image, or of the IMD demosaiced image, such as for example correction of an optical aberration, correction of a geometric aberration, modification of sharpness, modification of brightness, enhancement of at least one color in the image, etc.

[0094] The digital processing stage can be carried out by the same device that provides the enhanced raw image, or the demosaicing, or by another device.

[0095] According to another aspect of the same invention, a method is proposed for imaging a scene comprising the following steps: - Acquisition of a raw image, known as the main raw image, by a camera module, -processing of said primary raw image by the method according to the invention, and in particular by: ■ the method for obtaining an enhanced raw image according to the invention; or ■ the image demosaicing process according to the invention; or ■ the image processing method according to the invention.

[0096] The processing stage can be carried out, at least in part, in the same device that carried out the acquisition stage of the main raw image.

[0097] Alternatively, or in addition, the processing step can be carried out, at least in part, in a device other than the one that carried out the primary raw image acquisition step.

[0098] The image acquisition method according to the invention may further include, prior to the processing step, a step of acquiring at least one secondary raw image used during the processing of said primary raw image, in particular during the enrichment phase to obtain an enriched raw image.

[0099] At least one secondary raw image can be captured by the same camera module, or by another camera module, as the primary raw image.

[0100] At least one secondary raw image can be captured by the same camera module, or by another camera module, as another secondary raw image.

[0101] At least one secondary raw image can be obtained using any one of the non-limiting options described above.

[0102] According to another aspect of the same invention, a computer program is proposed comprising executable instructions which, when executed by a computer device, they implement all the steps of the process according to the invention, and in particular: - the method for obtaining an enhanced raw image according to the invention; or - the image demosaicing process according to the invention; or - the image processing method according to the invention; or - the method for imaging a scene according to the invention.

[0103] The computer program can be in any computer language, such as for example machine language, C, C++, JAVA, Python, etc.

[0104] Such a computer program can be presented as a standalone application. Alternatively, such a computer program can be integrated into a photo or video application, or even into an image or video playback application.

[0105] The computer program can be stored in a non-transient, or non-volatile, manner in a storage medium.

[0106] According to another aspect of the invention, a device is proposed comprising means configured to implement all the steps of the process according to the invention, and in particular: - the method for obtaining an enhanced raw image according to the invention; or - the image demosaicing process according to the invention; or - the image processing method according to the invention; or - the method for imaging a scene according to the invention.

[0107] The device according to the invention can be a computer, a processor, a computer chip, etc. programmed to implement the method according to the invention, for example by executing the computer program according to the invention.

[0108] The device according to the invention can be integrated into any type of device such as a smartphone, a tablet, a computer, a calculator, a processor, a computer chip, a medical imaging device, etc.

[0109] The device according to the invention may comprise, in terms of technical means, at least one, or any combination of at least two, characteristics described above with reference to the process according to the invention, and which will not be repeated here exhaustively for the sake of brevity.

[0110] According to another aspect of the invention, a device, and in particular a user device, is proposed, comprising: - at least one camera module to acquire at least one raw image of a scene, and - a unit of calculation; configured to implement all the steps of the process according to the invention and in particular: - the method for obtaining an enhanced raw image according to the invention; or - the image demosaicing process according to the invention; or - the image processing method according to the invention; or - the method for imaging a scene according to the invention.

[0111] The device according to the invention may comprise a single camera module, or several camera modules.

[0112] In particular, the device according to the invention can be a user device such as a smartphone, tablet, etc.

[0113] The user device may also include a display screen.

[0114] In particular, the device according to the invention can be a computer-type user device.

[0115] The computer-type user device according to the invention may further include a display screen.

[0116] In particular, the device according to the invention can be a television.

[0117] Television may also include a display screen.

[0118] In particular, the device according to the invention can be: - a virtual reality headset or glasses; or - an augmented reality headset, or glasses.

[0119] The helmet, or glasses respectively, according to the invention may further comprise at least one display screen.

[0120] In particular, the device according to the invention can be a medical imaging device.

[0121] In particular, the medical imaging device according to the invention can be an endoscope, an ultrasound device, etc.

[0122] The medical imaging device according to the invention may further include at least one display screen.

[0123] Of course, the device according to the invention is not limited to the examples of devices that have just been given.

[0124] According to another aspect of the present invention, a vehicle comprising is proposed: - at least one camera module to acquire at least one raw image of a scene, and - a unit of calculation; configured to implement all the steps of the process according to the invention, and in particular: - the method for obtaining an enhanced raw image according to the invention, or - the image demosaicing process according to the invention, or - the image processing method according to the invention; or - the method for imaging a scene according to the invention.

[0125] The vehicle according to the invention may further include a display screen, for example arranged in a passenger compartment of the vehicle, or a projector for projecting at least one image onto a display surface generally known as a "head-up display".

[0126] The vehicle according to the invention may comprise a single camera module, or several camera modules.

[0127] According to embodiments, the vehicle according to the invention can be a land vehicle, such as a car, autonomous, semi-autonomous or non-autonomous.

[0128] According to embodiments, the vehicle according to the invention can be a flying vehicle, such as a drone, an airplane, a helicopter, autonomous, semi-autonomous or non-autonomous.

[0129] According to embodiments, the vehicle according to the invention can be a maritime vehicle, such as a boat or a submarine, autonomous, semi-autonomous or non-autonomous. Description of the figures and methods of implementation

[0130] Other advantages and features will become apparent upon examination of the detailed description of non-limiting embodiments and the accompanying drawings, in which: - FIGURE 1 is a schematic representation of a non-limiting example of an image sensor implementation; - FIGURE 2 is a schematic representation of a non-limiting example of a method according to the invention for obtaining an enriched raw image; - FIGURE 3 is a schematic representation of a non-limiting example of an embodiment of a dematting process according to the invention; - FIGURE 4 is a schematic representation of a non-limiting example of an image processing method according to the invention; - FIGURE 5 is a schematic representation of a non-limiting example of an embodiment of a method according to the invention for imaging a scene; - FIGURE 6 is a schematic representation of a non-limiting example of an embodiment of a device according to the invention; - Figures 7a-7c are schematic representations of non-limiting examples of embodiments of a device according to the invention; and - FIGURE 8 is a schematic representation of a non-limiting example embodiment of a vehicle according to the invention.

[0131] It is understood that the embodiments described below are by no means exhaustive. In particular, variants of the invention may be conceived comprising only a selection of the features described below, isolated from the other features described, if this selection of features is sufficient to confer a technical advantage or to differentiate the invention from the prior art. This selection includes at least one preferably functional feature without structural details, or with only a portion of the structural details if this portion alone is sufficient to confer a technical advantage or to differentiate the invention from the prior art.

[0132] In particular, all the variants and embodiments described can be combined with each other if there are no technical obstacles to this combination.

[0133] In the figures and in the rest of the description, elements common to several figures retain the same reference.

[0134] FIGURE 1 is a schematic representation of a non-limiting example of an image sensor that can be used for the acquisition of a raw image.

[0135] The image sensor 100 in Figure 1 comprises a Bayer pattern array. In particular, the sensor has N rows of Bayer patterns and M columns of Bayer patterns, such that the sensor comprises a total of B = N x M Bayer patterns. Of course, "N" and "M" are integers and may be different or equal. Furthermore, the invention is not limited to a Bayer pattern array, and the image sensor may include an array of other types of patterns.

[0136] Each Bayer motif, noted Bi,j with the l <i<N et l<j<M, comprend un nombre P de photosites monochromes, choisis parmi des photosites rouge, vert ou bleu, avec P> 2. In the example shown, without limitation, each Bayer Bi,j motif is square and comprises four monochrome photosites, namely two green photosites V arranged on one diagonal of the Bayer Bij pattern, and one red photosite R and one blue photosite B arranged on the other diagonal of the Bayer Bij pattern.

[0137] Under these conditions, the 100 sensor provides a raw image represented by a matrix of B=nm value patterns. Each value pattern is provided by the Bayer pattern Bi,j and comprises four values, each provided by a monochrome photosite of said pattern Bi,j. Thus, in the example shown, each value pattern comprises two values ​​for the color green, one value for the color blue, and one value for the color red.

[0138] From a raw image provided by sensor 100 in FIGURE 1, it is possible to obtain a demosaiced digital image whose definition in terms of pixels can be NxM: in this case, the demosaiced digital image comprises NxM pixels, each pixel being obtained from a Bayer pattern.

[0139] Alternatively, it is possible to obtain a demosaiced digital image whose pixel definition can be P=4x(NxM): in this case, the demosaiced digital image comprises P=4x(NxM) pixels, with one Bayer pattern providing 4 pixels of the demosaiced digital image. In other words, each Bayer pattern provides as many image pixels as there are photosites in that Bayer pattern.

[0140] Alternatively, it is possible to obtain a demosaiced digital image whose pixel definition can be P=3x(NxM): in this case, the demosaiced digital image comprises P=3x(NxM) pixels, with each Bayer pattern providing 3 pixels of the demosaiced digital image. In other words, in this case, each Bayer pattern provides as many image pixels as there are different monochrome components in that Bayer pattern.

[0141] In the case of a color image, each pixel is represented by a set of at least three values: a value for the color red, a value for the color green, and a value for the color blue.

[0142] Of course, each pixel can also include other values ​​or be represented by other values ​​such as, for example, hue values, denoted T, saturation, denoted S, and luminance, denoted L.

[0143] FIGURE 2 is a schematic representation of a non-limiting example of a method according to the present invention for obtaining an enriched raw image.

[0144] The 200 method of FIGURE 2 can be used to obtain an enriched raw image of a scene, denoted IMBE, without loss of generality, and captured by an image sensor, for example the image sensor 100 of FIGURE 1.

[0145] To obtain the IMBE enriched raw image, process 200 uses a primary raw image, denoted IMBP without any generality, of the scene and one or more secondary raw images of the same scene, denoted IMBSi-IMBSk, with k>l. For example, k=5 secondary images IMBSi-IMBSs, so that the IMBE enriched raw image is obtained using five secondary raw images in addition to the primary raw image IMBP.

[0146] According to embodiments, the process 200 may include an optional step 202 of registration of at least one, and in particular of each, secondary raw image IMBSi-IMBSk with the primary raw image IMBP.

[0147] Such a registration step makes it possible, for example, to correct spatial shifts between images, due to movement of the imaging device or the fact that the scene includes moving objects.

[0148] Such a registration step makes it easier to carry out other steps of process 200, in particular the identification, in the, or each, secondary raw image, of a pattern corresponding to a target pattern of the main IMBP raw image.

[0149] Such a recalibration step can be carried out using any known technique, for example using the Lucas-Kanade method.

[0150] The 200 process includes an optional step 204 to determine a brightness map, or a hue map, or a saturation map, and more generally an image weighting map.

[0151] The weighting map can be determined from the primary raw IMBP image. This optional weighting map can be used during the demosaicing of the enhanced primary raw image, as described later, particularly when it is desired to obtain a demosaiced image pixel for each photosite of the image sensor. A non-limiting example of obtaining the weighting map is described later.

[0152] The process 200 includes a step 210 of enriching the main raw IMBP image to obtain an enriched raw image, denoted IMBE, without loss of generality. The enrichment step 210 can be performed for one or more, and in particular all, patterns of the main raw IMBP image, sequentially or simultaneously. In the following, and without loss of generality, and for clarity of description, the enrichment step 210 is considered to be performed individually for a pattern MIJ of the main raw IMBP image and can be repeated / performed for several, or even all, patterns sequentially or simultaneously.

[0153] In step 212, the MIJ motif, called the target motif, from the primary raw IMBP image is identified in a secondary IMBSi image. This identification can be performed in various ways. It can be carried out by iterative searching based on a correlation score. The correlation score can be calculated between the entire primary IMBP image and the entire secondary IMBSi image.

[0154] Alternatively, the correlation score can be calculated between an area, called the target area, Z c , of the primary IMBP image, of predetermined size, comprising the target pattern MI,J and an area of ​​the same size in the secondary image IMBSi. Without limitation, the target area Z cThe target area can be a square area of ​​81 motifs, centered on position (i,j) of the target motif MIJ, encompassing all motifs from the primary raw image IMBP located between motif Mi-4,j-4 and motif Mi+4,j+4. Of course, the target area can be a different size than 81 motifs, and / or a different shape than a square. The iterative search based on the correlation score aims to find, within the secondary image IMBSi, the area that, along with the target area Z, exhibits the same characteristics. c the highest correlation score or a correlation score greater than a predetermined correspondence threshold, for example 95% of the maximum The possible correlation is the autocorrelation of one of the images, or the average of the autocorrelation of the IMBP image and the autocorrelation of the IMBS image. For example, a correlation score is calculated between the target area Z cand a candidate area of ​​the IMBSi secondary image, of the same size and centered on position (i,j) of said raw IMBSi secondary image. Then, a new correlation score is calculated by shifting the candidate area in the raw IMBSi secondary image, for example, by one or more patterns. And so on. After identifying the area of ​​the raw IMBSi secondary image corresponding to the target area Z c , the pattern of said secondary raw image IMBSi which corresponds to the target pattern MIJ of the primary raw image IMBP is identified.

[0155] The same operation is performed for several secondary raw images, and preferably for each secondary raw image IMBSi-IMBSk, and the target MIJ motif is identified in each of said secondary images. In other words, potentially, step 212 provides, for the target MIJ motif, K motifs corresponding to the target MIJ motif, denoted M c iM c k, the corresponding pattern M ci from the secondary image IMBSi, the corresponding pattern M c 2 from the secondary image IMBS2, and so on.

[0156] Alternatively, for at least one target pattern, identification step 212 can be performed using the spatial offset between the primary raw IMBP image and the secondary raw IMBSi image. This spatial offset is either known because it is fixed during the acquisition of the secondary raw IMBSi image, or determined during the optional registration step 202.

[0157] In step 214, the target pattern MIJ is enriched using data from the k corresponding patterns M c iM c k from the K secondary raw images IMBSi-IMBSk.

[0158] Step 214 can be performed for a single monochrome component of the target pattern Mi,j. Alternatively, step 214 can be performed for several, and in particular all, monochrome components of the target pattern Mi,j, individually, in turn or simultaneously.

[0159] For one, or each, monochrome component, a maximum value is determined based on the values ​​of said component. monochrome in the Mij and M patterns c iM c k, during step 216 of enrichment step 214.

[0160] Following an example implementation, the maximum value can be the largest of the values ​​in the patterns Mij and M. c iM c k for said monochrome component.

[0161] Following another embodiment, the maximum value can be a theoretical maximum value of a curve passing through the values ​​of said monochrome component in the patterns Mij, and Mc iM c k. In this case, step 216 includes the following operations: -plotting a curve passing through the values ​​from the patterns Mij, and M c iM c k; and - determination of a theoretical maximum value for said curve. The curve can be plotted by taking into account the value provided by each of the motifs Mij, and M c iM c k for the monochrome component considered, taking into account, for each pattern: - the moment at which the raw image from which said motif originates was taken, and / or - the position of this pattern in the image sensor. The plotted curve can be a one-dimensional curve, or preferably a two-dimensional curve. Once the curve is plotted, through interpolation / extrapolation and more generally through calculation, it is possible to determine the theoretical maximum value that said curve can reach, and to take this theoretical maximum value into account as the value for the monochrome component considered in the target pattern Mij.

[0162] During step 218, the maximum value determined in step 216 is assigned to the monochrome component.

[0163] Step 214 can be carried out for at least one, and in particular each, monochrome component of the pattern Mi,j. In general, the enrichment phase 210 can be carried out for at least one, and in particular each, monochrome component of the pattern Mi,j taken individually, in turn or in parallel.

[0164] The enrichment phase 210 is repeated for at least one pattern, and preferably for several, or even all, of the patterns of the main raw IMBP image and provides the enriched raw IMBE image.

[0165] FIGURE 3 is a schematic representation of a non-limiting example embodiment of an image demosaicing process according to the present invention.

[0166] The 300 method of FIGURE 3 allows obtaining a dematrixed image, noted IMD without loss of generality, of a scene from a primary raw image IMBP and at least one less rich secondary raw image IMBSi-IMBSk of said scene.

[0167] The process 300 includes a step 302 to obtain a higher-resolution IMBE enriched raw image from the primary IMBP raw image and at least one less rich secondary IMBSi-IMBSk raw image. In particular, step 302 can implement process 200 of Figure 2, so that process 300 comprises all the steps of process 200 of Figure 1.

[0168] Next, process 300 includes a step 304 of demosaicing, of the raw enriched IMBE image, according to any known technique.

[0169] Demosaicing can be performed using any known technique to obtain a demosaiced image composed of pixels, and in particular colored pixels, each pixel being represented by a set of values. If the image is an RGB image, each pixel can be represented by a set of at least three values: one for red, one for green, and one for blue.

[0170] In step 304, demosaicing can be performed to obtain one image pixel for each pattern in the IMBE enriched raw image. In this case, when, for example, the enriched raw image contains NxM patterns, the resulting demosaiced image contains NxM pixels. In this case, for a pixel corresponding to a pattern in the IMBE enriched raw image: - the value of the color red corresponds to the value of the red monochrome component of said pattern, - the value of the blue color corresponds to the value of the blue monochrome component of said pattern, - the value of the green color corresponds to the value of the green monochrome component of said pattern, or to the average of the green monochrome components of said pattern if said pattern includes several green components.

[0171] Alternatively, in step 304, demosaicing can be performed to obtain one image pixel for each photosite of the image sensor, i.e., each monochrome component of each pattern in the raw IMBE-enriched image. In this case, process 300 may include an optional weighting step 306 using the weighting map obtained in optional step 204 of process 200. In this case, for each monochrome component of a pattern in the IMBE-enriched image, an image pixel is obtained for which: - the value of said monochrome component is assigned to the corresponding color; - the value of each of the other colours is obtained by weighting, with the weighting map, the monochrome component of the corresponding colour of said pattern. For example, for a photosite providing a red monochrome component of a pattern in the IMBE enriched raw image, we obtain a pixel for which: - the value of the color red corresponds to the value of said red monochrome component, - The value of the blue color corresponds to the value of the blue monochrome component of the pattern, weighted using the weighting map. - The value of the green color corresponds to the value of the green monochrome component of said pattern, or to the average of the green monochrome components of said pattern if said pattern includes several green components, weighted using the weighting map. The same logic applies respectively to a photosite providing a green monochrome component and to a photosite providing a blue monochrome component of the pattern.

[0172] In FIGURE 2, the optional weighting step 306 is shown independently of the demosaicing step 304 for the purposes of this description. Of course, in practice, step 306 can be carried out at the same time as demosaicing step 304, or be part of demosaicing step 304.

[0173] After demosaicing, the 300 process provides a demosaiced digital image, denoted IMD.

[0174] In one of the variants described above, process 300, and in general, the demosaicing process according to the invention, can use a weighting card.

[0175] The weighting map can be any type of map showing the evolution of light intensity in the scene.

[0176] The weighting map can be common to all monochrome components: in this case the weighting map can be a brightness map.

[0177] Alternatively, a weighting map can be created for each monochrome component: in this case, the weighting map can be obtained as an intensity map of that monochrome component. We will describe an example of obtaining the weighting map for a monochrome component, for example, the red component, R. Of course, a weighting map can be obtained for each of the other monochrome components, green, G, and blue, B, in the same way.

[0178] Each raw image—namely, the primary raw image IMBP and each secondary raw image IMBSi-IMBSk—considered individually, provides a grid of points for the scene where the intensities are known for the R component. However, by knowing the relative displacement between each of the images, for example, with reference to the primary raw image, it is possible to construct a new grid of points, called the combined grid, with much higher resolution for the R component. This combined grid is more resolved than the individual grid obtained from a single raw image because the displacement between two raw images is rarely equal to the repetition rate of the image sensor patterns, particularly the Bayer pattern repetition rate. Thus, if we have one primary raw image and K secondary raw images, the combined grid potentially has a resolution that is (K+1) times greater. to the resolution of each individual grid, both in the x direction and in the y direction (i.e. potentially (K+l) 2 resolution points times more).

[0179] Following an example implementation, the combined grid can be constructed as follows: - initialization of the over-resolved combined grid with default intensity values, for example zero, for the R component; - writing the intensities provided by the main raw image for the R component into the combined grid, for example at an arbitrary position; - for each secondary raw IMBSi image: ■determination of the displacement of said secondary raw image IMBSi relative to the primary raw image IMBP; and ■determination of the corresponding positions in the combined grid as a function of said displacement, and ■writing the intensities provided by the raw secondary image IMBSi for the R component, at said positions.

[0180] When several values ​​from several raw images correspond to a point on the combined grid, it is possible to average said values, which preserves the information from said images.

[0181] Thus, it is possible to obtain a weighting map for each of the monochrome components, for example R, G, B, more resolved than that of the original photosites.

[0182] This, or these, weighting map(s) can be used during demosaicing to obtain an image pixel for each photosite, or each monochrome component, of each pattern of the image sensor.

[0183] FIGURE 4 is a schematic representation of a non-limiting example embodiment of a method according to the present invention for processing an image according to the invention.

[0184] The 400 process allows obtaining a processed image, IMT, of a scene.

[0185] The process 400 includes a step 402 for obtaining an enriched raw image, IMBE, or a demosaiced image, IMD, from a primary raw image, IMBP, and at least one secondary raw image, IMBSi-IMBSk. In particular, step 402 can implement process 200 of FIGURE 2, or process 300 of FIGURE 3, respectively.

[0186] Process 400 further includes a step 404 for processing the raw enriched IMBE image, or the demosaiced IMD image. This processing step 404 can perform any type of digital processing on the raw enriched IMBE image, or the demosaiced IMD image.

[0187] For example, step 404 can perform the correction of an optical aberration, such as a chromatic aberration, in particular displacement aberration, a geomatic aberration, etc.

[0188] For example, step 404 can achieve image sharpness improvement, color enhancement, brightness modification, etc.

[0189] Step 404 provides the processed IMT image.

[0190] FIGURE 5 is a schematic representation of a non-limiting example of an embodiment of a method according to the present invention for imaging a scene.

[0191] The method 500 of FIGURE 5 includes a step 502 for acquiring a raw image, referred to as the master raw image, and denoted IMBP, of a scene with an imaging device comprising at least one camera module. The master raw image is acquired with a camera module comprising an optical lens associated with an image sensor, such as, for example, the image sensor 100 of FIGURE 1. Optionally, the camera module (or imaging device) may include, be equipped with, or be associated with, an image stabilizer. The function of such a stabilizer is to eliminate the effects of slight movements of the camera module, or the imaging device, used during the acquisition of the master raw image IMBP.

[0192] The 500 process of FIGURE 5 includes a step 504 which acquires one or more secondary raw IMBSI-IMBSK images of the scene.

[0193] At least one IMBSi secondary raw image can be acquired with a camera module comprising an optical lens coupled to an image sensor, such as, for example, the 100 image sensor in Figure 1. Optionally, the camera module (or imaging device) may include, be equipped with, or be coupled with an image stabilizer. Such a stabilizer is designed to eliminate the effects of slight movements of the camera module, or imaging device, used during the acquisition of the IMBSi secondary raw image.

[0194] At least one, in particular each, secondary raw IMBSi image can be acquired with the same camera module as that used for the acquisition of the primary raw IMBP image.

[0195] At least one, in particular each, secondary raw IMBSi image may be acquired with a camera module other than the one used for the acquisition of the primary raw IMBP image. This other camera module may be the same as, or different from, the one used for the acquisition of the primary raw IMBP image.

[0196] Optionally, at least one, in particular each, secondary raw IMBSi image may be acquired with a spatial offset from the primary raw IMBP image.

[0197] Following an example of an embodiment, at least one, in particular each, secondary raw IMBSi image can be acquired from a different acquisition position, and / or a different viewing angle, etc., so as to obtain a spatial offset relative to the main raw image.

[0198] Alternatively, or in addition, at least one, specifically each, secondary raw IMBSi image can be acquired by introducing a spatial offset using the image stabilizer on the imaging device. For example, when each of the secondary raw IMBSI-IMBSK images is acquired with a spatial offset, respectively di-dK, then it is possible to introduce the corresponding spatial offset into the stabilizer control. image. In this case, the stabilizer is controlled to eliminate all movement, except for the desired spatial shift.

[0199] When multiple secondary images are acquired with different spatial offsets di-dm, preferably, said raw secondary IMBSI-IMBSK images can be acquired in an order other than ascending or descending spatial offsets to avoid synchronizing said spatial offsets di-dx with the movement of an object in the scene. For example, the raw secondary IMBSI-IMBSK images can be acquired in a random order. As another example, the raw secondary IMBSI-IMBSK images can be acquired in a predetermined order other than ascending or descending order, and more generally, other than a rectilinear order or trajectory. For example, assuming the use of ten secondary IMBSI-IMBSio images, each with a spatial offset di-dio, respectively, with di <d2, ..., dg <dio, alors il est préférable de prendre les images brutes secondaires IMBSi-IMBSio dans un ordre différent de l'ordre allant de 1 à 10 ou de 10 à 1. Par exemple, il est possible de prendre lesdites images dans un ordre aléatoire, ou dans l'ordre suivant : IMBSi, IMBSio, IMBS2, IMBS9, IMBS3, IMBSs, IMBS4, IMBSe et IMBSs. Bien entendu, cet ordre est donné à titre d'exemple non limitatif.

[0200] After the acquisition of the raw images, process 500 includes a processing step 506. Step 506 can implement any one of the processes 200, 300 or 400 of FIGURES 2 to 4 to obtain, respectively, an enriched raw image IMBE, a dematrix image IMD or a processed image IMT.

[0201] FIGURE 6 is a schematic representation of a non-limiting example embodiment of a device according to the present invention.

[0202] The 600 device in FIGURE 6 can be used to implement: - a method according to the invention for obtaining a richer, enhanced raw image, IMMBE, from less rich raw images, and in particular method 200 of FIGURE 2; or - a method according to the invention for demosaicing a raw image to obtain a demosaiced image, and in particular method 300 of FIGURE 3; or - an image processing method according to the invention, and in particular method 400 of FIGURE 4; or - a method according to the invention for imaging a scene, and in particular method 500 of FIGURE 5.

[0203] The device 600 in FIGURE 6 may include an optional module 602 for registering at least one, and in particular each, secondary raw image with respect to the primary raw image. This optional module 602 may, in particular, be configured to perform the optional step 202 of the process 200.

[0204] Device 600 may include an optional module 604 to determine at least one weighting map. This module 604 can, in particular, be configured to perform step 204 of process 200.

[0205] Device 600 includes a module 606 for identifying at least one target pattern from the primary raw IMBP image in at least one secondary raw image. This module 606 can, in particular, be configured to perform step 212 of process 200.

[0206] Device 600 includes a module 608 for determining at least one maximum value of at least one monochrome component of a target pattern. This module 608 can, in particular, be configured to perform step 216 of process 200.

[0207] Device 600 further includes a module 610 for assigning at least one maximum value to at least one pattern. This module 610 can, in particular, be configured to perform step 218 of process 200.

[0208] When the device 600 is used to implement a dematting process according to the invention, and in particular the dematting process 300 of FIGURE 3, said device 600 further comprises a module 612 to demosaic the raw enriched IMBE image. This module 612 can in particular be configured to perform the demosaicing step 304 of process 300, and optionally the optional weighting step 306 of process 300.

[0209] When device 600 is used to implement an image processing method according to the invention, and in particular method 400 of FIGURE 4, said device 600 further comprises a module 614 for applying processing, in particular digital processing, to the raw enhanced IMBE image. This module 614 can, in particular, be configured to perform the digital processing step 404 of method 400.

[0210] When device 600 is used to implement a method according to the invention for imaging a scene, and in particular method 500 of FIGURE 4, said device 600 further comprises a module 616 for controlling one or more camera modules for acquiring a primary raw IMBP image and at least one secondary raw IMBSI-IMBSK image. This module 616 can, in particular, be configured to perform, or trigger the execution of, steps 502 and 504 of method 500.

[0211] Optionally, the 600 device can also include an image stabilizer 618. This image stabilizer can be controlled by the 616 module to introduce a spatial offset during the acquisition of at least one secondary raw IMBSi image. The 618 stabilizer is optional because it may already exist in the imaging device used to image the scene.

[0212] Optionally, the 600 device may also include at least one 620 camera module for acquiring the primary raw IMBP image and / or at least one secondary raw IMBSI-IMBSK image. The at least one 620 camera module can be controlled by the 616 module for acquiring the raw image(s). The at least one 620 camera module is optional because it may already be present in the imaging device used to image the scene.

[0213] At least one of the 602-616 modules can be a module independent of the other 602-616 modules.

[0214] At least two of the 602-616 modules can be integrated within a single module. In particular, the 602-616 modules can be integrated within a 622 computing unit.

[0215] At least one of the 602-616 modules can be a hardware module, such as a processor, an electronic chip, etc.

[0216] At least one of the 602-616 modules can be a software module, such as a computer program.

[0217] At least one of the 602-616 modules can be a combination of at least one software module and at least one hardware module.

[0218] In particular, at least one of the 602-616 modules can be integrated into an electronic chip, or into an application installed in a user device such as, for example, a photo application or a video application.

[0219] In particular, the 622 computing unit can be, or can be integrated, into an electronic chip, or into an application installed in a user device.

[0220] FIGURE 7a is a schematic representation of a non-limiting example embodiment of a device according to the present invention.

[0221] The apparatus 700 of FIGURE 7a includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0222] The apparatus 700 of FIGURE 7a may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0223] In the example shown in FIGURE 7a, device 700 is a smartphone, or a tablet, comprising device 600 from FIGURE 6.

[0224] Optionally, the device 700 can also include a display screen 702, optionally equipped with a sensing surface 704, for example capacitive.

[0225] Of course, the 700 device may include other components than those indicated above.

[0226] FIGURE 7b is a schematic representation of another non-limiting embodiment of a device according to the present invention.

[0227] The apparatus 710 of FIGURE 7b includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0228] The apparatus 710 of FIGURE 7b may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0229] In the example shown in FIGURE 7b, device 710 is: - a virtual reality (VR) headset or glasses, or - an augmented reality headset or glasses; including device 600 of FIGURE 6.

[0230] Optionally, the 710 device may also include a 712 display screen, for example in / on / under a visor of said 710 helmet.

[0231] Of course, the 710 helmet may include other components than those listed above.

[0232] FIGURE 7c is a schematic representation of a non-limiting example embodiment of a device according to the present invention.

[0233] The apparatus 720 of FIGURE 7c includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0234] The apparatus 720 of FIGURE 7c may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0235] In the example shown in FIGURE 7c, device 720 is a medical imaging device, such as an endoscope, an ultrasound machine, etc.

[0236] Optionally, the device 720 can also include a display screen 722, optionally equipped with a sensing surface 724, for example capacitive.

[0237] Of course, the 720 medical imaging device may include other organs than those indicated above.

[0238] FIGURE 8 is a schematic representation of a non-limiting example embodiment of a vehicle according to the present invention.

[0239] The vehicle 800 of FIGURE 8 includes means configured to implement the invention, and in particular any one of the methods 200, 300, 400 or 500.

[0240] The vehicle 800 of FIGURE 8 may include a device according to the invention, and in particular the device 600 of FIGURE 6.

[0241] In the example shown in FIGURE 8, vehicle 800 is a land vehicle, in particular a car, comprising device 600 of FIGURE 6.

[0242] Optionally, the vehicle 800 may also include a display screen 802, optionally equipped with a sensing surface 804, for example capacitive, arranged in the passenger compartment of the vehicle 800.

[0243] Of course, the 800 vehicle may include other components than those indicated above.

[0244] Of course, the invention is not limited to the examples that have just been described.

Claims

DEMANDS 1. Method (200) for enriching a raw image (IMBP) of a scene, called the main raw image, from at least one other raw image (IMBSi-IMBSi) of said scene, called the secondary raw image, each of said main (IMBP) and secondary (IMBSi-IMBSi) images being: - acquired by at least one camera module comprising an image sensor (100) having a pattern matrix (Bi,j) of monochrome photosites; - represented by a matrix of value patterns, each value pattern being provided by a photosite pattern of said image sensor (100) and comprising a value provided by each monochrome photosite of said photosite pattern; said process (200) comprising a phase (210) of enriching the main raw image (IMBP) providing an enriched raw image (IMBE), said enrichment phase (210) comprising, for at least one pattern, said target pattern, of said main raw image (IMBP), the following steps: - identification (212), in at least one secondary raw image (IMBSi-IMBSk), of a pattern corresponding to said target pattern; and - adjustment (216,218) of at least one value of said target pattern as a function of value(s) of corresponding pattern(s) identified.

2. A method (200) according to the preceding claim, characterized in that the adjustment step (216, 218) comprises, for at least one monochrome component of the target pattern, the following steps: - determination (216), for said monochrome component, of a maximum value as a function of the value of this monochrome component in said target pattern and in the corresponding pattern(s), and - assignment (218) of said maximum value to said monochrome component in said target pattern.

3. A method (200) according to any one of the preceding claims, characterized in that the step (216) of determining the maximum value comprises the following operations: -plotting a curve with the values ​​from the patterns; - determination of a maximum value of said curve.

4. Method (200) according to any one of the preceding claims, characterized in that, for at least one target pattern, the identification (212) of said target pattern in a secondary raw image (IMBSi-IMBSk) is carried out by correlation calculation between the primary raw image (IMBP) and said secondary raw image (IMBSi-IMBSk).

5. A method (200) according to any one of the preceding claims, characterized in that, for at least one target pattern, the identification (212) of said target pattern in a secondary raw image (IMBSi-IMBSk) is performed as a function of: - of a monochrome component of said pattern, and / or - of a parameter deduced from at least two monochrome components of said pattern.

6. A method (200) according to any one of the preceding claims, characterized in that the enrichment phase (210) is carried out to: - only one pattern at a time; or - an area comprising several patterns.

7. Method (200) according to any one of the preceding claims, characterized in that the enrichment phase (210) is carried out with several secondary raw images (IMBSi-IMBSk).

8. Method (200) according to any one of the preceding claims, characterized in that there is a spatial offset between the primary raw image (IMBP) and at least one, and in particular each, secondary raw image, (IMBSi_IMBSk) for at least a part of the scene.

9. Method (200) according to any one of the preceding claims, characterized in that the, or at least one, secondary raw image (IMBSi-IMBSk) is an image captured by another camera module.

10. Method (200) according to any one of the preceding claims, characterized in that the, or at least one, secondary raw image (IMBSi-IMBSk) is an image captured by the same camera module at a different time.

11. A method (300) for demosaicing a raw image of a scene comprising the following steps: - obtaining (302) an enriched raw image (IMBE) by the process (200) according to any one of the preceding claims; and - demosaicing (304) of said enriched raw image (IMBE) to obtain a demosaiced image (IMD).

12. Method (300) according to the preceding claim, characterized in that the demosaicing step (304,306) provides an image pixel for at least one, and in particular for each, pattern of the enriched raw image (IMBE).

13. Method (300) according to claim 11, characterized in that the demosaicing step (304,306) provides an image pixel for each monochrome component of at least one, and in particular of each, pattern of the enriched raw image (IMBE).

14. A process (300) according to the preceding claim, characterized in that the dematting step (304, 306) comprises the following steps: - determination (204) of a weighting map; and - deduction (306), for each monochrome component of the pattern, of an image pixel according to said weighting map.

15. Image processing method (400) comprising the following steps: - obtaining (402) an enriched raw image (IMBE) by the process (200) according to any one of claims 1 to 10, or a demosaiced image (IMD) by the process (300) according to any one of claims 11 to 14; and -digital processing (404) of said raw enhanced image (IMBE), or of said demosaiced image (IMD).

16. Method (500) for imaging a scene comprising the following steps: - acquisition (502) of a raw image (IMBP), called the main raw image, by a camera module, -processing (506) of said primary raw image (IMBP) by the process (200;300;400) according to any one of claims 1 to 15.

17. Computer program comprising executable instructions which, when executed by a computing device, implement all the steps of the process (200;300;400) according to any one of the preceding claims.

18. Device (600) comprising means configured to carry out all the steps of the process (200;300;400;500) according to any one of claims 1 to 16.

19. Device (700;710;720) comprising: - at least one camera module (620) to acquire at least one raw image of a scene; and - a calculation unit (622); configured to implement all steps of the process (200;300;400;500) according to any one of claims 1 to 16.

20. Vehicle (800) comprising: - at least one camera module (620) to acquire at least one raw image of a scene; and - a calculation unit (622); configured to implement all steps of the process (200;300;400;500) according to any one of claims 1 to 16.

Images