Image correction method

Abstract

Disclosed is a method for rendering at least one source image of a scene obtained by an optical sensor of an imaging system. The method comprises a phase of processing a reconstructed image, at least some of the pixels of which includes depth information and / or speed information. At the end of the processing phase, a rendered image R is obtained for each processed reconstructed image. The processing phase includes an iterative modification of the reconstructed image so as to minimize a function E comprising a term D, called difference term, determined by comparing the reconstructed image convolved by a point spread function (PSF) with the source image and a term A, called penalty term, which is representative of defects or anomalies in the reconstructed image and is determined from the reconstructed image.

Description

[0001] DESCRIPTION

[0002] TITLE: Image correction process

[0003] Technical field

[0004] The present invention relates to the field of image processing and correction methods. These methods aim to improve the overall quality or overall rendering of an image containing defects.

[0005] The invention relates to a method for correcting and improving the quality of any type of image.

[0006] The invention relates in particular, but not exclusively, to any device comprising cameras or optical sensors capable of acquiring images. For example, it may be photographic devices, cameras, smart phones (or smartphones), tablets, laptops or systems, vehicles or infrastructures equipped with cameras.

[0007] State of the prior art

[0008] The most well-known and widespread image processing method is so-called stereo vision, derived from binocular human vision, or more broadly animal vision, from two images that are compared. The biological processing of the images obtained is extremely efficient, since it provides in real time a notion of depth of the observed scene, allowing, for example, to move knowing the relative distance of the observed objects.

[0009] Despite these remarkable performances, this biological stereo vision system does not allow to simultaneously restore a sharp image over a large depth of field, ranging from a few tens of centimeters to infinity. In particular, this blur comes partly from the limited depth of field of the human eye. The ocular optics does not allow to obtain a focus simultaneously at all distances, which is well explained by the conjugation law in geometric optics. This well-known phenomenon is linked to the notion of accommodation of the eye. But also, human vision involves a rotational displacement of the eyes in particular to match the zone of best sharpness of each eye (fovea), which depends on the distance of the observed objects. As a result, objects observed outside the distance of best focus may not overlap correctly and give rise to an imperfect 3D interpretation.Also known in the state of the art are methods for reconstructing images comprising relief or depth information from 2D images. These methods allow depth information associated with the images to be deduced from two images. A widely used reconstruction technique is photogrammetry. However, there are several problems and limitations to the reconstruction methods of the state of the art. Indeed, all these techniques require identifying as many common pixels as possible on the 2D images from which the depth is determined. This identification is necessary to be able to extract the depth information.

[0010] Therefore, to obtain information on a large part of an imaged scene and / or to obtain the most reliable depth information possible, state-of-the-art methods require images with the highest possible definition as well as the highest possible resolution.

[0011] However, despite good image quality and / or definition, when the 2D images from which the depth is determined include smooth areas, i.e. areas with little or no relief, or areas with little or too much texture or areas with little or too much contrast, the depth information determined is at best erroneous or even impossible to determine.

[0012] Furthermore, the raw images obtained by optical systems equipping image acquisition devices, and in particular consumer devices such as cameras and smartphones, are imperfect and contain defects inherent to image acquisition devices. Although there are image processing methods aimed at improving their quality, defects persist in the images processed using state-of-the-art methods and / or the resolution of the corrected image is reduced compared to the raw image obtained by the sensor. These two effects combine with the problems mentioned above and induce additional errors in the extracted depth information and prevent the depth information from being determined on a large number of pixels in the reconstructed images.Finally, state-of-the-art image processing methods, when implemented by processing means with "standard" computing power, often have a high processing time that is not compatible with the extraction of depth information in real time, in particular from a continuous image stream. The majority of optical systems equipping cameras and smartphones comprise a single focusing system and a single photographic sensor. These optical systems generate mosaicked raw images due to the use of a matrix of colored filters, the best-known example being the Bayer matrix, allowing color separation. This technique requires that the different color channels, typically R, G, B (Red, Green, Blue), be obtained at different positions of the photographic sensor, i.e. via different photosites, typically photodiodes, forming a mosaic.As a result, the colors reproduced on the raw image from the sensor are not reproduced at all points (photosites) of the sensor. In order to maintain the best possible resolution, i.e. the native resolution of the sensor corresponding to the tiling of the photosites, it is necessary to process the raw image to extract the two missing colors from the photosites, for example blue and green for the red photosite. Methods are known in the state of the art to eliminate this mosaic effect on raw images. However, these methods lead to the appearance of color fringes and the erroneous reproduction of colors not present in the imaged object.

[0013] Other cameras and smartphones are equipped with optical systems comprising several separate focusing lenses and a stack of several photosite planes aligned in the sensor. Thus, each pixel of the raw image includes the three color channels and no mosaic effect needs to be processed. However, these optical systems generate aberrations in the raw image linked to focusing effects and defects. In practice, this generates geometric distortions and blurring effects. Methods are known in the state of the art to eliminate these optical and chromaticity aberrations. However, these methods lead to the appearance of rendering errors and defects within the processed image.

[0014] One aim of the invention is to propose an image processing method enabling:

[0015] - to improve the quality of the images, and / or

[0016] - to compensate for focusing defects inherent in any optical lens, and / or

[0017] - to limit artifacts related to the projection of the image onto the color detection matrix, and / or

[0018] - to limit the effects of noise, and / or

[0019] - to increase the dynamic range in amplitude, and / or - to extract a depth, i.e. the distance to the objective, and / or a speed at each point of the image from images not including depth or speed information, and / or

[0020] - to improve the depth and / or speed information contained in an image, and / or

[0021] - to increase the depth of field of the image, without having to adjust the focus of the optical image acquisition system,

[0022] - to process and / or enrich images and / or extract depth and / or speed information more precisely than the state of the art.

[0023] Presentation of the invention

[0024] For this purpose, a method is proposed for restoring at least one image from one or more source images. Each source image is an image of a scene obtained by an optical sensor of an imaging system.

[0025] The method comprises a phase of processing a reconstructed image, associated with a source image, at least some of the pixels of which comprise depth information and / or speed information. At the end of the processing phase, a restored image is obtained for each reconstructed image processed. The processing phase comprises an iterative modification of the reconstructed image so as to minimize a function E.

[0026] The function E includes a term D, called the difference term, determined by comparing the reconstructed image convolved by a point spread function (PSF) with the source image, to which the reconstructed image is associated. The PSF describes the response of the imaging system from which the source image used for comparison is obtained.

[0027] The function E includes a term A, called anomalies or penalties, representative of defects or anomalies within the reconstructed image, determined from the reconstructed image.

[0028] Preferably, the method comprises, prior to and / or, respectively, during the processing phase, a step of reconstructing the reconstructed image or respectively the reconstructed image being processed from at least one source image and / or at least one reconstructed image and / or at least one reconstructed image being processed and / or at least one restored image.Preferably, the step of reconstructing the reconstructed image, or respectively the image being reconstructed, comprises or is carried out by or consists of an addition of depth and / or speed information and / or a substitution of the depth information and / or the speed information contained in at least a portion of the pixels of the at least one source image and / or of the at least one reconstructed image and / or of the at least one reconstructed image being processed and / or of the at least one restored image, or respectively in at least a portion of the pixels of the at least one reconstructed image being processed.

[0029] Preferably:

[0030] - the depth information and / or the speed information added or substituted is obtained from or by comparing the reconstructed image or the reconstructed image being processed with at least one source image and / or at least one reconstructed image and / or at least one reconstructed image being processed and / or at least one restored image, and / or

[0031] - the depth information and / or the speed information added or substituted is obtained by comparing at least one source image and / or at least one reconstructed image and / or at least one reconstructed image being processed and / or at least one restored image with at least one source image and / or at least one reconstructed image and / or at least one reconstructed image being processed and / or at least one restored image, and / or

[0032] - depth information and / or speed information is added or substituted by moving a pixel or a set of pixels or one or more blocks of an image by an operator, for example differential or vector, and / or according to a function, for example a scalar or vector field.

[0033] In this application, the term "image being processed" may be understood to mean the reconstructed image being processed or being restored. In this application, the term "processing phase" may be understood to mean the iterative modification of the reconstructed image.

[0034] In the present application, the term "image" used alone may mean a source image, a reconstructed image, a reconstructed image being processed or a restored image. In the present application, "image block" may be understood to mean a set of pixels, preferably contiguous and / or adjacent to each other, of an image. In the present application, "imaging system" may be understood to mean an optical system capable of acquiring images. For example, the imaging system may be a camera.

[0035] The processing phase can be defined as minimizing the function E by modifying the reconstructed image. In other words, the processing phase can be defined as comprising at least one iteration of a step of modifying the reconstructed image aimed at minimizing the function E.

[0036] The reconstruction step can be defined as consisting of adding depth information and / or speed information, for example when the latter is missing, in at least a part of the pixels of an image not containing depth information and / or speed information and / or substituting depth information and / or speed information contained in a source image, a reconstructed image, a reconstructed image being processed or a restored image and / or moving a pixel or a set of pixels or one or more blocks of an image by one and / or according to a function.

[0037] The image obtained by such a displacement of a pixel or a set of pixels or one or more blocks of an image can be considered as a reconstructed image in that the image before displacement sees, among other things, the depth information and / or the speed information contained in one or more pixels modified in the image obtained after displacement. For example, the reconstruction step can comprise a displacement of one or more pixels or a set of pixels or one or more blocks of an image according to a velocity field. Preferably, the velocity field corresponds to or is representative of a relative movement of one, several or each of the objects of the scene with respect to the imaging system as a function of time.

[0038] The PSF can be substituted by any known function describing the response of an imaging system. For example, the PSF can be substituted by an optical transfer function (OTF) of the imaging system.

[0039] Preferably, the function E, more preferably the term D and / or the term A, depends on the speed information and / or the depth information.

[0040] Preferably, the method or step of reconstructing the reconstructed image comprises an extraction of the depth information and / or the speed information of the imaged scene by comparing at least two source images or at least two reconstructed images or at least two reconstructed images being processed or at least two restored images or at least one source image with at least one restored image or with a reconstructed image or with a reconstructed image being processed.

[0041] In the present application, an extraction of depth and / or speed information by image comparison may consist of or comprise a comparison of at least two source images or at least two reconstructed images or at least two reconstructed images being processed or at least two restored images or a comparison of at least any two images from at least one source image, at least one restored image, at least one reconstructed image and at least one reconstructed image being processed.

[0042] The reconstruction method or step may include:

[0043] - during the processing phase, an addition of depth information and / or speed information in at least a portion of the pixels of the reconstructed image being processed not containing depth information and / or speed information and / or a substitution of the depth information and / or speed information contained in the at least a portion of the pixels of the reconstructed image being processed from the depth information and / or speed information extracted from an image comparison, and / or

[0044] - prior to the processing phase, an addition of depth information and / or speed information in at least a portion of the pixels of an image not containing depth information and / or speed information and / or a substitution of the depth information and / or speed information contained in the at least a portion of the pixels of the image from the depth information and / or speed information extracted from an image comparison.

[0045] Preferably, the term D is determined by comparing:

[0046] - the reconstructed image or the reconstructed image being processed convolved with the PSF, with

[0047] - the source image, to which the reconstructed image is associated.

[0048] Preferably, the step of determining the term D comprises, prior to the comparison of the reconstructed image convolved with the PSF or of the reconstructed image being processed convolved with the PSF with the source image, with which the reconstructed image is associated, a spreading, at least locally, according to a velocity field and / or according to an acquisition duration of said source image, with which the reconstructed image or the reconstructed image being processed is associated, of pixel(s) or group(s) or block(s) of pixels of said reconstructed image convolved with the PSF or of said reconstructed image being processed convolved with the PSF.

[0049] Preferably, the spreading, preferably local, of pixel(s) or group(s) or block(s) of pixels of the reconstructed image convolved with the PSF or of the reconstructed image being processed convolved with the PSF is carried out according to a velocity field.

[0050] Preferably, the processing phase comprises, prior to the convolution of the reconstructed image or the reconstructed image being processed with the PSF, a selection of the PSF.

[0051] Preferably, the reconstruction step comprises a convolution of the source image, to which the reconstructed image is associated, by an inverse function of the PSF, called iPSF, describing the response of the imaging system from which the source image, convolved by the iPSF, is obtained.

[0052] Preferably, the iterative modification of the reconstructed image ends when the function E, or a combination of partial derivatives of the function E with respect to the reconstructed image being processed, is less than a minimization threshold, or when a certain number of iterations of the iterative modification of the reconstructed image is reached; the at least one reconstructed image thus modified is restored.

[0053] Preferably, the reconstruction step comprises or is implemented by or is carried out by or consists of a displacement of blocks, according to a velocity field, of a source image, of a reconstructed image, of a reconstructed image being processed or of a restored image.

[0054] Preferably, the reconstruction step comprises or is implemented by or is carried out by or consists of a displacement of blocks, according to a depth field, of a reconstructed image, of a reconstructed image being processed or of a restored image. Preferably, the reconstructed image is obtained by merging at least two reconstructed images.

[0055] Preferably, the processing phase comprises a step of merging at least two rendered images to provide at least one rendered image.

[0056] Preferably, for a considered imaging system, the PSF depends on:

[0057] - an orientation of an optical axis of the imaging system considered relative to at least one other optical axis of another imaging system, and / or

[0058] - a relative position of the imaging system considered with respect to at least one other imaging system, and / or

[0059] - a distance between the sensor of the imaging system considered and an object imaged by the sensor of the imaging system considered, and / or

[0060] - a distance between a part of the imaging system considered and the at least one sensor, and / or

[0061] - a distance between a part of the imaging system considered and an object imaged by the sensor of the imaging system considered, and / or

[0062] - a state of the imaging system, such as a zoom or focus or numerical aperture setting of the imaging system, and / or

[0063] - of the pixel or of a group or of a block of pixels of a source image or of a reconstructed image or of a restored image and / or of a photosite of the sensor of the imaging system considered, and / or

[0064] - one or more pieces of information contained in a pixel or a group or block of pixels of an image, and / or

[0065] - one or more angles between the optical sensor of the imaging system considered and the imaging system.

[0066] Preferably, the PSF is selected from a set of PSFs. Preferably, the PST is selected based on one or more of the parameters on which the PSF depends.

[0067] Preferably, the at least one source image I comprises a set of source images comprising or consisting of a temporal succession of source images (Itl, It2...) acquired, preferably by the same imaging system, at different times (tl, t2...), each processing phase (Pl, P2...), each reconstruction step (Ml, M2...), each reconstructed image (Ml, M2...) and each restored image (Cl, C2...) are associated with a source image obtained at a given time t.

[0068] Preferably, the reconstructed image Mt is obtained from the source image It to which the reconstructed image Mt is associated.

[0069] The reconstructed image Mt may not be reconstructed from the source image It with which it is associated. For example, the reconstructed image Mt may be obtained or reconstructed from at least one reconstructed image and / or at least one reconstructed image currently being processed and / or at least one restored image, associated with a source image (It-1, It-2...) obtained at a previous time (t-1, t-2...).

[0070] Preferably, the reconstructed image Mt, associated with the source image It, is obtained, or the step of reconstructing the image Mt, associated with the source image It, comprises or is carried out, by comparison:

[0071] - of an image obtained by displacements, preferably according to a velocity field and / or according to a depth field, of blocks of the restored image Ct-1, or of the reconstructed image Mt-1 or of the reconstructed image Mt-1 during processing, at the processing phase Pt-1, with

[0072] - the source image It, to which the reconstructed image Mt is associated, convolved by the inverse function of the PSF, called iPSF, describing the response of the imaging system from which the source image It is obtained.

[0073] Preferably, the reconstructed image mt, associated with the source image It, is obtained from the restored image Ct-1 obtained in the processing phase Pt-1.

[0074] Preferably, the reconstructed image Mt at the processing phase Pt is obtained by moving blocks, according to a velocity field, of:

[0075] - the restored image Ct-1 obtained in the processing phase Pt-1 or of the reconstructed image Mt-1 in the processing phase Pt-1 or of the reconstructed image Mt-1 being processed in the processing phase Pt-1, with which the source image It-1 is associated, or

[0076] - of the source image It, to which the reconstructed image Mt is associated.

[0077] Preferably, at least one step, preferably each step, of the method, and in particular the processing phase and the reconstruction step, is implemented for each source image (Itl, It2...) acquired at each of the different times (tl, t2...). Preferably, the at least one source image I comprises a set of source images acquired by different imaging systems (d, g...), preferably at the same given time or instant t, each processing phase (Pd, Pg...), each reconstructed image (Md, Mg...) and each restored image (Cd, Cg...) are associated with a source image I, of the set of source images (Id, Ig...), acquired by a different imaging system; the imaging systems being arranged so as to acquire, at least in part, preferably mainly, the same scene.

[0078] Preferably, at least one step, preferably each step, of the method, and in particular the processing phase and the reconstruction step, is implemented for each source image (Id, Ig...) acquired by a different imaging system (d, g...).

[0079] Preferably, the step of reconstructing the reconstructed image M is carried out from:

[0080] - at least one source image Id or at least one reconstructed image Md or at least one restored image Cd, and

[0081] - at least one source image Ig or at least one reconstructed image Mg or at least one restored image Cg.

[0082] Preferably, the temporal succession of source images (Itl, It2...) and the source images acquired by the different imaging systems (d, g...) can form two distinct sets of source images.

[0083] Preferably, the temporal succession of source images (Itl, It2...) and the source images acquired by the different imaging systems (d, g...) form the same set of source images (Idtl, Idt2, Igtl, Igt2...). Preferably, each of the source images of the set of source images (Idtl, Idt2, Igtl, Igt2...) is acquired by a separate imaging system (d, g...) at a given time (tl, t2...). Each processing phase (Pdtl, Pdt2, Pgtl, Pgt2...), each reconstructed image (Mdtl, Mdt2, Mgtl, Mgt2...) and each restored image (Cdtl, Cdt2, Cgtl, Cgt2...) are associated with a source image obtained by a given imaging system and at a given time t. According to the invention, there is also provided a device or apparatus or a data processing system comprising means, for example a processing unit, arranged and / or programmed and / or configured to implement the method according to the invention.

[0084] According to the invention, there is also provided a computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to the invention.

[0085] According to the invention, there is also provided a medium, for example a recording medium, readable by a computer comprising instructions which, when executed by a computer, cause the latter to implement the method according to the invention.

[0086] According to the invention, a computer-readable data carrier is also provided, on which the computer program according to the invention is recorded.

[0087] Description of figures

[0088] Other advantages and features of the invention will become apparent upon reading the detailed description of implementations and embodiments which are in no way limiting, and the following appended drawings: [Fig. 1] FIGURE 1 is a diagram illustrating the implementation of the iterative loop, consisting of successive modifications or successive iterations of modifications made to the reconstructed image, during the processing phase, for each of the source images acquired successively, by an imaging system, [Fig. 2] FIGURE 2 is a schematic representation of the iterative loop of the processing phase implemented from the source images of the left imaging system,

[0089] [Fig. 3] FIGURE 3 illustrates a global and simplified view of the image restitution method according to the invention,

[0090] [Fig. 4] FIGURE 4 illustrates an embodiment of the initialization step of the method, [Fig. 5] FIGURE 5 illustrates an embodiment of the initialization step of the method,

[0091] [Fig. 6] FIGURE 6 illustrates one embodiment of depth extraction from two images.

[0092] Description of the embodiments

[0093] The embodiments described below being in no way limiting, it will be possible in particular to consider variants of the invention comprising only a selection of the described characteristics, isolated from the other described characteristics (even if this selection is isolated within a sentence comprising these other characteristics), if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection comprises at least one characteristic, preferably functional without structural details, or with only a part of the structural details if this part only is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.

[0094] With reference to FIGURES 1 to 6, an embodiment of the method 1 for restoring images from source images I according to the invention is presented. Each source image I is an image of a scene obtained by an optical sensor of an imaging system 2, for example a camera 2. Preferably, the method 1 is implemented for a set of source images I. The method can be implemented from source images I stored on a storage medium or can be implemented, in real time, from a stream of source images I coming from one or more image acquisition devices 2, such as cameras 2.

[0095] With reference to FIGURE 1, the method 1 is implemented for a set of source images consisting of a temporal succession of source images ItO, Itl, ..., It-1, It, It+1... acquired, by the same imaging system 2, at different times tO, tl ... t-1, t, t+1.... In FIGURE 1, the times at which each source image I is acquired are discretized tO, tl, ..., t-1, t, t+1... so that each new source image ItO, Itl, ..., It-1, It, It+1 acquired is incremented successively. However, the method 1 can also be implemented for a set of source images I acquired by different imaging systems 2 d, g... at the same time t. According to the invention, the imaging systems 2 are arranged so as to acquire the same scene.

[0096] The method 1 according to the invention is particularly suitable for processing a set of source images IdtO, Idtl, ..., Idt-1, Idt, Idt+1..., IgtO, Igtl, ..., Igt-1, Igt, Igt+1... acquired by a distinct imaging system 2 d, g... at a given time tO, tl, t-1, t, t+1.... Thus, the method 1 makes it possible to process in real time a stream of source images Idt, Igt each coming from several imaging systems 2 (d, g). This embodiment is particularly suitable, among other things, for image processing which is implemented by autonomous vehicles. However, it is also particularly suitable for any type of system or device, such as, among others, smartphones and tablets, which have several distinct cameras. In the case of autonomous vehicles, several cameras 2 are arranged in different positions to acquire images of the surroundings of the vehicle.In general, several cameras are oriented and arranged so as to acquire the same scene. For the sake of simplicity and understanding, the description of the embodiment is based on the use of two imaging systems 2 called right, noted d, and left, noted g. The two imaging systems 2 right and left are arranged so as to acquire, at least in part, the same scene.

[0097] Also, with reference to FIGURE 3, and in accordance with the non-limiting embodiment presented, the image restitution method 1 comprises a processing phase Pt of a reconstructed image Mdt, Mgt. As illustrated in FIGURE 2 and detailed below, the processing phase Pt is implemented independently for each reconstructed image Mdt, Mgt and each reconstructed image Mdt, Mgt is associated with a distinct source image Idt, Igt; each source image Idt, Igt being obtained by a distinct imaging system 2 d, g. At the end of the processing phase Pt, a restored image Cdt, Cgt is obtained for each reconstructed image Mdt, Mgt processed. As illustrated in FIGURES 1 and 2, the processing phase Pt comprises an iterative modification of the reconstructed image Mdt, Mgt so as to minimize a function E.

[0098] The processing phase can therefore be described as consisting of iterating the processing loop 3 or iterative loop 3 illustrated in FIGURE 2. With reference to FIGURE 1, the number of iterations is denoted k. For the processing loop Pt-1, the reconstructed image Mt-1 is modified iteratively until the function E, or a combination of partial derivatives of the function E with respect to the reconstructed image Mt-1 being processed, is less than a minimization threshold. In other words, the reconstructed image M will be modified at each iteration k of the iterative loop 3 so that the reconstructed image M being processed (obtained after each iterative modification), and ultimately the restored image C, best matches or best conforms to or best restores the observed scene.The reconstructed image M before being modified for the first time in the first iterative loop 3 (or during the first iterative modification of the processing phase P) can be called reconstructed image M or reconstructed image M at iteration k=0, the reconstructed image M before being modified for the second time (i.e. after the first iterative loop 3 of the processing phase P and before and before the second iterative modification of the processing phase P) can be called reconstructed image M being processed at iteration k>0 (in the case described at iteration k=1). In other words, the reconstructed image M (at iteration k=0) corresponds to the reconstructed image that has never been modified in the iterative loop 3 and the reconstructed image M being processed (at iteration k>0) corresponds to the set of reconstructed images M being processed at iterations k>0 that are being restored or are being iteratively modified.It may also be provided that the processing phase P is iterated until a certain number of iterations k is reached in the event that the minimization threshold cannot be reached in a satisfactory time, and therefore number of iterations. In the processing phase Pt-1, the reconstructed image Mt-1 thus modified after kl iterations is restored Ct-1. In the processing phase Pt, the reconstructed image Mt as modified after k2 iterations is restored (restored image Ct). Also, the method 1 comprises a succession of processing phases P, each implemented for a different source image I. In other words, the method 1 comprises the implementation of as many processing phases P as there are processed source images I.

[0099] The function E includes a term D, called difference or anomaly, determined by comparing the reconstructed image Mdt, Mgt convolved by a point spread function (PSF) with the source image Idt, Igt, to which the reconstructed image Mdt, Mgt is associated. Thus, each processing phase Pt (and therefore each reconstructed image Mt, each reconstructed image Mt being processed and each restored image Ct is associated with the source image It to which the reconstructed image Mt convolved by the PSF is compared during the processing phase Pt. The PSF describes the response of the imaging system d or g from which the source image Idt, Igt used for the comparison is obtained. The comparison step consists of detecting any information or value or data contained in the pixels of the pixel matrix of the compared images.According to an advantageous but non-limiting embodiment, the comparison step consists of detecting the colors within a color matrix, for example a Bayer matrix, of the reconstructed image Mt and of the source image It and comparing them in order to find the best match. In order to provide improved and / or precise depth information, it is advantageous to take into account only the pixels R (Red), VI (Green, or G (Green)), V2 (Green, or G (Green)), B (Blue) of the Bayer matrix corresponding to the reconstructed image Mt, or on pixels without a color filter with a particular geometry, called "dual pixels", to give sensitivity to the depth of the image for the auto-focusing of the objective of the imaging system 2.However, the comparison step may consist of detecting one or more data contained in the pixels of the pixel matrix of the compared images among a color data and / or texture data, and / or brightness data, and / or hue data, and / or saturation data, and / or RGB data.

[0100] The function E includes a term A, called penalty, representative of defects or anomalies within the reconstructed image Mdt, Mgt, determined from the reconstructed image Mdt, Mgt.

[0101] The P processing phase is described in detail below.

[0102] Each image (source I, reconstructed M, reconstructed M during processing and restored C) can be represented in a table of values. It contains color values ​​at discretized positions, for example according to the three colors R (red), G (green), B (blue), or in another reference frame. We can see these elements as juxtaposed square or rectangular elementary objects, comprising three color properties, in addition to width and length properties of the elements common to all these elementary objects, as well as the position property on the optical sensor implicitly known by a row arrangement in the table and at a constant pitch in each direction x and y (x and y being two orthogonal directions in the image plane). These objects can be called pixels.Also, with the exception of the source images I which do not contain depth information, each reconstructed image Mdt, Mgt, each reconstructed image Mdt, Mgt being processed and each restored image Cdt, Cgt is representable by a pixel table or pixel matrix containing (xd or xg, yd or yg, Z). The reconstructed images Mdt, Mgt, reconstructed images Mdt, Mgt being processed and restored images Cdt, Cgt containing depth information can be described as images enriched with depth information. In addition, the objects in the scene imaged by the imaging systems 2 are at different distances from one or more optical sensors (of the imaging systems 2) and therefore at a different depth.Also, a group or block of pixels corresponding to one of the objects of the imaged scene will have depth information different from one or more other groups or blocks of pixels corresponding to the other or other objects of the imaged scene. Consequently, each image will be enriched by a depth field, that is to say that each pixel or group or block of pixels (of the matrix or table of pixels representing the image of the scene) corresponding to an object of the scene will have depth information which corresponds to the distance between the object and the optical sensor from which the source image I was acquired; the depth of the different objects of the imaged scene being, in most cases, different for at least some of the objects of the scene.The person skilled in the art will know how to convert or transpose the coordinates or positions of the pixels (or any other associated operator, matrix or grid) from the x and y space (sensor reference frame) to another spatial reference frame U and W (in physical position in space) to represent the images in a matrix (U, W, Z). Thus, the convolution of the reconstructed image Mt or reconstructed Mt being processed by the PSF can be carried out from any of the representation matrices chosen by the person skilled in the art.

[0103] This table represents elementary objects preferably according to a resolution equal to or greater than that of the union of all the photosites of all the colors of a considered optical sensor. It can have a scanning step and positions offset relative to the photosites of the considered sensor, but preferably in spatial agreement with the latter.For example, for an elementary color filtering matrix (or "Color Filter Array" in English or CFA) of 4 photosites R (Red), VI (Green 1), V2 (Green 2), B (Blue), arranged on 2 x 2 rows and columns, it can have a digital representation, for each of the for example three color properties R (red), V (green), B (blue), of a pixel opposite the R, then of a pixel opposite the VI, V2, B, that is to say in total 4 pixels comprising a channel R, V, B, in spatial correspondence of the 4 photosites R, VI, V2, B of detection, and repeated as many times as necessary in the U and W directions to describe the entire optical sensor considered associated with a given image. Each source image I includes the intensities of the R (Red), G (Green), B (Blue) photosites (or according to another possible choice of color matrix such as R (Red), J (Yellow), W (White or White), B (Blue)) of the optical sensor associated with this source image I.As known to those skilled in the art, the source image I may, in addition, include other information on the gain and black level of the photosites, photosite by photosite (or region by region where appropriate), that is to say that the intensities detected by the photosites will preferably have been corrected to have the same digital level of black restitution, and a level of restitution of a homogenized reference white, for example uniform, or according to a reference in accordance with the subsequent point spread correction processing. In the event of non-linearity or saturation, the method may comprise a step of correcting this source image I.

[0104] It would also be possible to downsample, although this would not result in an improvement in image quality in terms of sharpness, but would allow for good quality of a compressed image (in terms of number of pixels).

[0105] This table can also be completed, or replaced by a list of objects describing the image. Each object can thus be the square or rectangular pixel, or objects with more properties such as an outline, average color components, but also brightness and color gradient components for example, and a particular shape. This second representation allows more freedom in representing positions, with possibly coordinates represented by numbers having more resolution than a collection of pixels at predefined locations. In addition, one can authorize the addition of colors and intensities of objects that would be superimposed at certain locations.

[0106] The processing phase is now described, i.e. the iterative modification of the reconstructed image M (i.e. the iterations of the processing loop 3 shown in FIGURE 3). Information is also generated from relatively precise knowledge of the optical focusing defects of the imaging system 2 or of one of the optical elements of the imaging system 2 (for example the objective).This information may include reading a table providing access to specific information related to the defects of the imaging system 2 or one of the optical elements of the imaging system 2 (for example the lens), as well as the distance between each lens or system and the scene observed through this lens or system, either considered according to a single distance, or differentiated into object-by-object distance of the scene, and the distance between the lens or system and the sensor associated with this lens or system, and possibly variations in angle between these elements. From this information, a restitution of the point spread functions (called PSF) is synthesized, which can be materialized by matrices of coefficients capable of producing by convolution with the intensities ideally emitted by each point of the scene, the intensities obtained at the level of the optical sensor, for each color channel.In the case where displacement defects are known, for example by a representation of the residual movement of non-compensation of the camera module with respect to the scene, the PSF matrices at different positions weighted by the time which has elapsed there are summed, to calculate a composite PSF including these residual movements not compensated, and to use this form of the PSF for the processing phase. To obtain the movement trajectory of a sensor or imaging system 2, it is sufficient for example that an acceleration sensor records the movements of the device which contains the camera module 2 during the image capture, and if the image is mechanically stabilized with respect to the movements of the device comprising the camera module 2, it would be sufficient that the absolute residual of non-stabilization is calculated with respect to the acceleration sensor and to the stabilization signal.In another implementation, in the absence of direct motion measurement data, the image is analyzed to determine the result of the motion trajectory and to deduce the composition to be applied to the PSFs to obtain compensation for this motion. In this way, the restored image C will also be compensated for the blurring effects related to the uncompensated residual motions.

[0107] Thus, the processing phase allows compensation for optical projection defects on the optical sensor, and reduces noise in the detection of light intensities. In particular, it also compensates for mosaic effects with spatially separated detection of several colors, and can also effectively compensate for residual movements of a camera module relative to the scene. The processing phase does not consist of processing the image in successive stages but allows a single processing including all the restitution operations.

[0108] Also, taking into account the fact that the processing phase P implemented from a reconstructed image M containing depth information, it is appropriate to calculate the PSFs as a function of the distances to the different objects on the scene, in order to obtain compensation for these optical aberrations taking into account the real focusing distance of each photographed object, which will increase the effective depth of field when restoring the image.

[0109] Thus, the processing phase provides as output a so-called restored image C comprising an absence of mosaicking, a significant reduction of noise according to a setting to be adjusted if necessary, a correction of position distortion and blurring of the image adapted to each region of the image, a compensation of relative movements between the objective or the imaging system 2 and the observed scene. The processing phase P therefore does not include a succession of correction steps dissociated from each other. Thus, the effects of conjugations and / or additions of the defects inherent in each step are avoided. It also makes it possible to limit the interpolations of values ​​which would have had to be detected in the absent photosites (because at the location of one of the other detected colors).First of all, it is worth noting that, in general, it is advantageous, but not necessary and in certain cases to be avoided, to implement the steps of the method (whether, for example, in the image processing or image reconstruction phase or for any step, in particular of image comparison or comprising an image comparison) from blocks of images rather than on all the pixels considered individually. A block of an image corresponds to an object (or a part of an object) of the scene for which one or more data of the pixels (among the pixel table of the image) constituting the block (representative of the object of the scene) of the image has an identical, similar or equivalent value.Also, it is advantageous to consider as an image block a set of pixels for which one or more of the data which are contained in each of the pixels of the set of pixels constituting the block (and in particular the depth and / or speed information) have identical, neighboring, similar, close or similar values. Such regionalization of the image makes it possible to considerably reduce the time of the calculations carried out on the basis of these images. Furthermore, when the processing or the step implemented from the image comprises or consists of moving (or translating) the position (in the x, y space) of all or part of the pixels within the pixel grid (which is for example the case during the processing phase, during the reconstruction step or during spreading), no loss of resolution is generated in this case.

[0110] It should be noted that the steps of the method could be implemented, in part, from blocks of images but also, in part, from (isolated) pixel(s) which could not have been grouped in the form of blocks. The term "block" extends in the broad sense, it can, for example, be a group of pixels forming the outline (or part of the outline) of another block. Also, in most cases, it will be possible to constitute images formed from a set of blocks.

[0111] It is also possible to homogenize the data(s) having similar or equivalent values ​​by assigning to all the pixels a single and same value for this or these data(s) to the detriment of the image resolution. In other words, it is possible to give an identical value to one or more data(s) contained in the pixels forming a block of the image, when these values ​​have been identified as neighboring, similar, close or similar. A person skilled in the art will know, depending on the intended application, the required image quality and the resources to be allocated to the processing (or the maximum tolerated processing time), how to adapt a threshold difference from which the data values ​​are no longer considered equivalent.

[0112] The processing phase P must take into account the entire model of optical defects by calculating, for each iteration k of the processing loop 3 and for a given imaging system 2 (from which the source image It associated with the reconstructed image Mt is acquired), from the reconstructed image Mt, the optical defects induced on each pixel or group of pixels (or blocks) of a source image It (represented by the values ​​of the different color components of the photosites).

[0113] D is determined by comparison of:

[0114] - the reconstructed image Mt convolved with the PSF then spread, at least locally, according to a velocity field and / or according to an acquisition duration of the source image It, to which the reconstructed image Bt is associated, with

[0115] - the source image It, to which the reconstructed image Bt is associated.

[0116] The term D can include the sum of several terms Di. Obtaining the distances Di obtained by comparing the reconstructed image Mt convolved by the PSFs with the source image It, can be, among other things, can be carried out by an evaluation of the local colors in the pixels or groups of pixels in front of the discretization grid of the PSFs, so as to know the colors calculated at the positions of the photosites of the reconstructed image Mt to make the distance comparisons at the appropriate places, without shift, allows maximum restitution precision.

[0117] The optical defect model is taken into account by a calculation, at each iteration k, of the effects that it induces from the reconstructed image M, by composing this image M during processing with the optical defect model, according to the embodiment by a convolution product, noted MCt, between the matrix of the luminous values ​​of the reconstructed image M (at each point of a fairly fine grid), and the PSF matrix on this same grid, which at output provides a matrix of values ​​MCt according to a grid (grid which allows comparison with the photosites of the optical sensor from which the source image I was obtained).

[0118] Of course, a convolution calculation can be done by multiplication in the frequency domain, and it is also possible to take into account the point spreading effects by the spatial Fourier transforms of the PSF matrices, generally called OTF (Optical Transfer Function), which then simply need to be multiplied by the Fourier transforms of the reconstructed image being rendered, then returned to the spatial domain by inverse transform, if this operating mode is of interest. The point spreading effects can be taken into account by using the TG operator which can be the identity, or a geometric transformation, which consists of locally moving the elements, either by translation of pixels in such a representation, or by equivalent modification of the PSF by recalculation of the set of coefficients to take into account this transformation, or possibly a mixture of the two.

[0119] Knowledge of the velocity field within the source images I and / or the reconstructed images M makes it possible to predict and anticipate the movement of objects from a source image It and / or, respectively, from a reconstructed image Mt associated with a successive source image It and / or, respectively, from a successive reconstructed image Mt+1. Thus, point spread effects can be applied to the reconstructed image M, prior to comparison with the source image I, using the velocity field. This makes it possible, among other things, to eliminate the blur linked to the aperture time of the imaging system, when taking each image and offers a precise and more faithful restitution of the image of the objects present on the imaged scene (in particular when they are moving in the imaged scene).In addition to predicting or anticipating the movement of objects from a source image It and / or, respectively, from a reconstructed image Mt associated with a successive source image It and / or, respectively, from a successive reconstructed image Mt+1, the velocity field also makes it possible to calculate and predict the future position of moving (or non-moving) objects in the imaged scene and therefore to know the depth information associated with the objects in the imaged scene. Also, it is possible by analyzing, for example by comparing the position of identical objects within successive images, to know the position of objects hidden in the source image It.

[0120] According to a non-limiting embodiment, it is possible to express the result, noted MCt, of the convolution product of the reconstructed image Mt being processed at iteration k by the PSF, and possibly processed by the geometric transformation TG:

[0121] [Math. 4]

[0122] MCt = reconstructed image Mt (*) PSF, or

[0123] MCt = TG(reconstructed image Mt) (*) PSF, or

[0124] MCt = TG(reconstructed image Mt (*) PSF), where (*) means convolution product.

[0125] From this comparison, a first indicator or term called distance Di, possibly completed with other terms Di, resulting from a comparison between the reconstructed image Mt convolved by the PSFs (possibly taking into account the point spreading effects) with the source image It, which we note Di(Bt, It), is obtained:

[0126] [Math. 5]

[0127] D= Dl(Bt, It) + D2 + ... + Dnd.

[0128] The term A, called penalty, representative of defects or anomalies within the reconstructed image Mdt, Mgt, is determined from the reconstructed image Mdt, Mgt, is calculated concomitantly with the term D. The second term A comprises: at least one component Ai whose effect is minimized for small differences in intensity between neighboring pixels of the reconstructed image M being processed at iteration k, and / or at least one component A3 whose effect is minimized for small differences in hue between neighboring pixels of the reconstructed image M being processed at iteration k, and / or at least one component A2 whose effect is minimized for small frequencies of changes in direction between neighboring pixels of reconstructed image M being processed drawing a contour.

[0129] The second term A can thus include a sum of several terms Ai.

[0130] As for the term D, for each iteration k and for a given imaging system (from which the source image It associated with the reconstructed image Mt is acquired), one or more penalties called Pi, P2, ..., P np , are calculated directly from the reconstructed image Mt or from the reconstructed image Mt being processed.

[0131] A global deviation indicator E is obtained which, according to the non-limiting embodiment, can be expressed as: [Math. 6] where nd is the number of distances Di, np the number of penalties Pi, with for example: nd = l, np = l, ai = 1 and 0i which can be adjusted to give more or less weight between the distance which forces to conform to the source image I, and the penalty which forces to make the reconstructed image M, at each iteration, conform to another property, or to have a certain coherence with respect to the images that the imaging system is supposed to observe. In other words, this penalty indicator also makes it possible to select the images which are most likely to be observed among those which will be restored. For example, it is unlikely to observe a reconstructed image M containing red, green, blue "snow" within its textures, and we can say that the penalty indicator P is constructed to make the appearance of such restored images C less likely, when it is constructed to be sensitive to color noise in the reconstructed image M.The penalty function Pi will guide towards a preferred solution for example by preventing neighboring pixels from having values ​​with a large difference between them (except, for example, for neighboring block / contour pixels) and thus reduce the noise of the rendered image C compared to the potentially noisy source image I or even limit the effects of slots on the contours of the blocks or objects of the rendered image C. The penalty function Pi will, among other things, also improve the depth and speed information by preventing neighboring pixels from having values ​​that are too different, and by removing aberrant values ​​within the same block.

[0132] In particular, according to the embodiment, the function E comprises or consists of the sum of the first term D and the second term A. Also, at the first iteration k=0, the processing loop at the processing phase PO compares the source image 10 and the result MCt of the convolution product of the reconstructed image MO at iteration 0 (i.e. unmodified or as resulting from the reconstruction step) by the PSF, and possibly processed by the geometric transformation TG so as to take into account a maximum of elements precisely modeling the geometry of all the parts of the optical sensor, the optical focusing effects throughout the sensor, any movements, the distance effects modifying the focusing effects.

[0133] The implementation of the method aims to obtain a restored image C which minimizes the indicator E.

[0134] Except in special cases where the expression of the distance Di would allow a direct calculation to minimize it, and where the penalty PI would also allow it, or would be taken as zero, the general method of resolution can for example be obtained by a method called Newton's gradient, which makes it possible to find the minimum of a function, by varying the parameters or input data inversely to the gradient of a function E, to minimize its value.

[0135] Thus, at each iteration, we calculate the sum of the derivatives of each term of E with respect to all the parameters or data of the reconstructed image Mt-1 at the previous iteration k-1, then we evolve the elements of the reconstructed image Mt at iteration k with respect to this gradient. The aim of the iterations is to modify the reconstructed image M at each step, so as to minimize the indicator E. The parameters or data of the image, and therefore of E, are, depending on the embodiment, the values ​​R, G, B and the depth information (Z) and / or the speed information (V). It may also be advantageous to consider the position (v, w) of any pixel or any image block (or the physical coordinates (x, y)) as a parameter.

[0136] Although the particular embodiment described is based on the use of partial derivatives of the function E with respect to the reconstructed image M being processed to minimize the function E, any other techniques for minimizing the function E are applicable. With regard to the non-limiting embodiment described, it is possible to calculate a local matrix of partial derivatives dE / di( ,w') where i(v,w) is one of the data or a parameter contained in a pixel or a block, at the coordinates (v,w), of the reconstructed image M being processed. Also, the minimization of E will amount, depending on the embodiment, to calculating dE to find its optimal value. dE can be expressed as the sum of the partial derivatives dE / di(y,w).

[0137] As detailed above, it is preferable to implement the method on image blocks. In addition to the R, G, B, V and Z parameters, it may be advantageous to use other partial derivatives with respect to other parameters such as, for example, a block dilation parameter and / or a block contour parameter representative of the dilation and contour effects of objects in the scene (contour which may, in some cases, be considered as a block). It may also be advantageous to consider the position of any pixel or any image block as a parameter, which would add to dE the partial derivatives dE)dv and dE / dw.

[0138] Also, depending on the embodiment, dE can be written:

[0139] [Math dE = d

[0140] [Math dE = d

[0141] [Math dE = (

[0142] Finally, it is recalled that this minimization of E is carried out at each iteration k of each processing phase Pt. Also, the derivative dE, as well as each of the partial derivatives dE / di(y,w) can be indexed by the iteration k at which it is calculated as well as by the time t at which the source image It associated with the current processing phase t was acquired d(E(R,G,B,Z,V,v,w)k,t) and dE / di(y,w') k t . The description of the embodiment using partial derivatives for the minimization of the function E is transposed, directly, to the term D of differences. Also, the minimization of E will tend to ensure that each of the terms Di = dD / di( ,w) kt and Pi = dP / di( ,w k t be minimized, that is to say to reduce the disparities between the different parameters of the reconstructed image Mt being processed at iteration k compared to the source image It and guide the restitution of the image towards what is expected.

[0143] The convolution of the reconstructed image M by the PSF will contribute to reducing the distance term Di. As such, everything described in the application regarding PSFs, discretization of the PSF grid, geometric transformation (GT) (of the image before convolution or of the convolution product) obviously applies here.

[0144] In order to further improve the image quality, and in particular the restitution of the depth and speed parameters, it is possible to implement the step of updating the depth and speed information during the processing phase, i.e. between two iterations k of the processing phase. In this case, the update step is carried out from the reconstructed image Mgt being processed in the iterative loop associated with the source image Igt and the reconstructed image Mdt being processed in the iterative loop associated with the source image Idt. A geometric transformation (GT) may sometimes be necessary before the comparison. This step is described in detail below.

[0145] In addition, preferably, contour search or extraction operations are to be carried out on the reconstructed image M or reconstructed M being processed at least at certain iterations, in order to create, move elements of the reconstructed image M or reconstructed M being processed.

[0146] The distance D or the distances Di therefore come from the convolved reconstructed images Mt (*) PSF (or TG (reconstructed image Mt (*) PSF) while P or the Pi come directly from the reconstructed images Mt.

[0147] At the end of a Pt processing phase, either the value of E is judged sufficiently small, or an estimate of the gradients relative to it is sufficiently small, or the number of iterations is limited to estimate that the image obtained is restorable.

[0148] The calculations of the distances Di are preferably made on all the positions of the photosites. It is therefore appropriate to calculate the values ​​of the different intensities of the colors from [Math. 4]. The penalties Pi can be calculated at positions not necessarily linked to the positions of the photosites. They can be calculated from pixels or groups of pixels of the reconstructed image Mt-1 at the previous iteration k-1, or from properties of larger objects, without necessarily showing the color properties according to the precise image restitution grid or the PSF for example, but by a more global calculation from properties of the object.

[0149] In general, and for each imaging system considered, the function describing the response of the at least one imaging system, the PSF according to the non-limiting embodiment, may depend on:

[0150] - a distance Z between the optical sensor and an object in the scene imaged by the optical sensor, and / or

[0151] - a relative position of the imaging system considered with respect to at least one other imaging system, and / or

[0152] - a distance between a part of the imaging system and the optical sensor of the imaging system, and / or

[0153] - a distance between a part of the imaging system and an object in the scene imaged by the optical sensor, and / or

[0154] - a state of the imaging system, such as a zoom or focus or numerical aperture setting of the imaging system, and / or

[0155] - of the pixel of the image being reconstructed M or of the reconstructed image M being processed and / or of the restored image C and / or of the photosite of the sensor, and / or

[0156] - of the information contained in each pixel or group or block of pixels of the convolved image, and / or

[0157] - of each position in the image field, in particular depending on the depth of the object, and / or

[0158] - of each pixel or group or block of pixels, in particular depending on the depth information contained in each pixel or group of pixels), and / or

[0159] - one or more angles between the optical sensor and the imaging system.

[0160] One of the objectives of the invention being to improve and make reliable the depth information contained in the enriched images, it is important that the PSF is a function, among other things, of the depth field so that the restored image C is sharp, has a maximum effective depth of field and accurately reflects the distance between the objects in the scene and the lens and / or the sensor. Thus, for a given reconstructed image M, there will therefore be at least as many PSFs as there are pixels or image block(s) comprising different depth information.

[0161] The distance separating the 2 imaging systems must be known. Preferably, the 2 imaging systems are stationary relative to each other. The orientation, relative and / or relative to each other, of the sensor and / or the lens and / or the 2 imaging systems is known. In the case of a smartphone or a car, all these parameters are known and do not change.

[0162] The distance Z between the optical sensor and an object in the scene imaged by the optical sensor is known or, preferably, determined or calculated according to the invention. Also, the determined or calculated depth information can be used to feed the PSF during successive implementations of the steps of the method according to the invention.

[0163] Although the processing phase P may be sufficient to obtain an improved image C, in particular comprising solidified depth and speed information. The reconstruction step is not necessarily implemented before or during each processing phase. It is even possible that the method does not include a reconstruction step. It is possible, for example, to assign a predetermined or default depth and / or speed information value to the pixels of the source image It. The processing phase P described above will make it possible to improve the depth and / or speed information as described above. It is even possible to enrich the source image It with several depth and / or speed information values, some blocks being, for example, assigned a depth and / or speed value different from some other blocks.However, to further improve the quality of the restored images C, it is preferable that the method includes a reconstruction step.

[0164] With reference to FIGURES 3 to 5, the method 1 comprises, prior to and / or, respectively, during the processing phase, a step of reconstructing the reconstructed image Mdt, Mgt or respectively the reconstructed image Mdt, Mgt being processed from at least one source image I and / or at least one reconstructed image M and / or at least one reconstructed image M being processed and / or at least one restored image C. Also, at least a portion of the pixels of the reconstructed image Mdt, Mgt comprises depth information. A reconstructed image is understood to mean an image in which at least a portion of the pixels comprises depth information. In other words, apart from the source images I, all of the images of the method are enriched with depth information, in particular are enriched with a depth field.

[0165] The method is of particular interest when the imaging systems, the imaged scene and / or objects of the imaged scene are in relative motion with respect to one (or more) another. Indeed, in this case, it is advantageous for the reconstruction step to comprise an extraction of the depth information from the imaged scene. However, the depth information may not be extracted or may not be extracted prior to or during each processing phase p. Indeed, by way of non-limiting example, the reconstruction step may be implemented from depth information not extracted but obtained by or originating from, for example, a depth information measurement system such as, for example in the case of an autonomous vehicle, a LIDAR.This depth information can be added and / or substituted in at least part of the pixels of an image (source I, reconstructed M, reconstructed M during processing or restored C) in order to obtain a reconstructed image Mdt, Mgt. Still by way of non-limiting example, the reconstruction step, i.e. obtaining a reconstructed image Mdt, Mgt, can comprise or consist of a displacement of image blocks, for example from the restored image Cdt-1, Cgt-1 to the previous processing phase Pt-1, according to a velocity field and / or a depth field (preferably, but not necessarily, contained in the pixel table representing the image).

[0166] For this purpose, the method may also comprise an extraction of speed data and / or the initialization step may comprise an addition and / or a substitution of speed data from at least two source images It, It-1, It-2..., from at least two reconstructed images Mt, Mt-1, Mt-2..., from at least two reconstructed images Mt, Mt-1, Mt-2... currently being processed, from at least two restored images Ct, Ct-1, Ct-2..., from at least one source image It-1, It-2, It-3... and at least one restored image Ct, Ct-1, Ct-2..., from at least one reconstructed image Mt-1, Mt-2, Mt-3 and at least one restored image Ct, Ct-1, Ct-2... or from at least one source image It-1, It-2, It-3... and at least one reconstructed image Mt, Mt-1, Mt-2. In the same way, the method can also comprise a step of updating the speed information contained in an image from at least two source images It, It-1, It-2..., from at least two reconstructed images Mt, Mt-1, Mt-2..., of at least two reconstructed images Mt, Mt-1, Mt-2... currently being processed, of at least two restored images Mt, Mt-1, Mt-2..., of at least one source image It-1, It-2, It-3... and at least one resituated image Ct, Ct-1, Ct-2..., of at least one reconstructed image Mt-1, Mt-2, Mt-3 and at least one restored image Ct, Ct-1, Ct-2... or of at least one source image It-1, It-2, It-3... and at least one reconstructed image Mt, Mt-1, Mt-2 but also from reconstructed images Mt currently being processed at iterations k-1, k-2, k-3... preceding the iterative loop 3 of the current processing phase Pt. The extraction of velocity data or information is performed from two (or more) separate images, preferably, but not necessarily, acquired by the same imaging system 2, at two separate times so that the moving objects in the imaged scene are in two separate positions on the two (or more) images being compared.

[0167] According to a non-limiting embodiment, the extraction of the velocity field comprises the comparison of two or more images originating from the same imaging system 2 (d or g) and acquired at separate times. A step of identifying a pixel or a group or a block of pixels common to the two (or more) compared images is carried out. This step aims to identify the same object which is present in the two compared images. The comparison relates to any information or value or data contained in the pixels of the pixel matrix of the compared images. Preferably but not limitingly, the information may be color data and / or texture data, and / or brightness data, and / or hue data, and / or saturation data, and / or RGB data.

[0168] When the objects common to the two images have been identified, the speed information is extracted from the positions (and / or the difference in position) of the pixel or group or block of pixels identified on the two compared images and the time elapsed between the two images (time lapse between t and t-1 for example if the two compared images are It and It-1). It should be noted that an extraction from the restored images (for example Rt and Rt-1) is advantageous because of the better quality, the better resolution and the few defects that the images restored according to the method include in comparison with the source images I.

[0169] Knowledge of the relative movement speed of the imaging systems 2 with respect to the imaged scene, and possibly of the imaging systems 2 with respect to each other, can be taken into account for a better resolution of the extracted velocity field or can even be used to deduce the velocity field which can thus be added and / or substituted in the pixel table of the image. Therefore, the velocity field is not necessarily determined by extraction but can be obtained by or come from, for example, a depth information measuring system such as, for example in the case of an autonomous vehicle, a rev counter, a tachometer or a LIDAR. Thus, according to the invention, the images (reconstructed Mt, reconstructed Mt during processing and restored Ct) can also be enriched with a velocity field. Consequently, the velocity field contained in the images (reconstructed Mt-1, Mt-2, Mt-3..., reconstructed Mt-1, Mt-2, Mt-3... during processing and restored Ct-1, Ct-2, Ct-3...) from previous processing phases Pt-1, Pt-2, Pt-3 can be used for the implementation of the extraction step, the initialization step (including the movement of image blocks according to the velocity field) and the velocity data update step.

[0170] Preferably, for one, several or each of the processing phases P, the method 1 preferably comprises a reconstruction step implemented prior to the or each of the processing phases P. With reference to FIGURES 3 to 5, the reconstruction step implemented prior to the processing phase P is called the initialization step. Furthermore, alternatively, preferably in a complementary manner, the reconstruction step can be implemented during the processing phase. This is particularly advantageous when a large disparity between the source images Id and Ig and / or the reconstructed images Md and Mg and / or the restored images Cd and Cg is observed or when the differences D are significant. The reconstruction step implemented during the processing phase P is called the depth information updating step.However, in certain cases, for example when few or no notable disparities are observed between the source images Id and Ig and / or the reconstructed images Md and Mg and / or the restored images Cg and Cd or when the differences D are small or negligible, the method may not comprise, for one or more processing phases P, a reconstruction step. In this case, the restored image Ct-1 at the processing phase Pt-1 may be used as the reconstructed image Bt at the processing phase Pt, possibly with the implementation of block displacements of the image Ct-1 (as described in detail above and below) prior to the processing phase Pt.

[0171] For a processing phase Pt considered, the reconstruction step therefore makes it possible to obtain an image (reconstructed Mt, reconstructed Mt during processing or restored Ct) in which depth information has been added and / or substituted in at least part of the pixels.

[0172] Furthermore, as detailed above, the efficiency of the processing phase makes it possible to reduce the resources required for processing the source images Idt, Igt and therefore makes it possible to process the source images Idt, Igt in real time and to restore images Cdt, Cgt prior to the acquisition of the source images Idt+1, Igt+1. Thus, with reference to FIGURE 1, the image Ct, associated with the source image It acquired at time t, which is restored at the end of the iteration Kl, is generally and ideally obtained before the acquisition of the source image Ct+1. Thus, the restored image Ct can be used for the processing phase Pt+1, and where appropriate for the subsequent processing phases Pt+2, Pt+3....The reconstructed image Mdt or reconstructed Mdt being processed can be used for the reconstruction of a reconstructed image Mdt+1, Mdt+2 associated with another source image (preferably subsequent) Idt+1, Idt+2 or for the reconstruction of a reconstructed image Mgt, Mgt+1, Mgt+2 associated with a source image Igt, Igt+1, Igt+2 originating from another imaging system. The restored image Cdt can be used for the reconstruction of a reconstructed image Mdt+1, Mdt+2 associated with another source image (preferably subsequent) Idt+1, Idt+2 or for the reconstruction of a reconstructed image Mgt, Mgt+1, Mgt+2 associated with a source image Igt, Igt+1, Igt+2 originating from another imaging system. This is of particular interest because the processing phase makes it possible to obtain reconstructed images M during processing and restored images C whose quality is much higher than that of the source images I.Also, the reconstruction step is preferably implemented from reconstructed images Mt or reconstructed images Mt being processed or, particularly advantageously, from images restored Rt-1 at the previous processing phase, and / or at the previous processing phases. In this way, the depth information being extracted from images of better quality (containing few or no defects, or, at least, fewer defects than the source images), the depth information contained in the restored images C is more reliable, more precise and includes fewer errors.

[0173] The initialization step and / or the depth information updating step can be implemented by extracting the depth information from at least two images (source I, reconstructed B, reconstructed B being processed and / or restored R). With reference to FIGURE 6, a method for extracting the depth information known to those skilled in the art under the name of photogrammetric comparison is illustrated. This method is well known to those skilled in the art and will not be described in detail. This method comprises the comparison of two or more images originating from distinct imaging systems 2 (two or more) placed in two distinct positions in space. A step of identifying a pixel or a group or a block of pixels common to the two (or more) compared images is carried out. This step aims to identify the same object which is present in the two compared images.The comparison relates to any information or value or data contained in the pixels of the pixel matrix of the compared images. Preferably but not limited to, the information may be color data, and / or texture data, and / or brightness data, and / or hue data, and / or saturation data, and / or RGB data.

[0174] When the objects common to the two images have been identified, the disparity is measured or determined for each common object, including the offset (in coordinates or pixel positions) of the common object of one image relative to the other. In practice, the depth information is extracted from the disparities, in accordance with FIGURE 6 using principles of optics and geometry available to those skilled in the art. It should be noted that the embodiment presented is only a non-limiting example. Those skilled in the art will be able to adapt this method or choose another of the existing methods.

[0175] Unlike the extraction of the velocity field which is carried out, preferably from two images of the same scene acquired at two different t, t-1, the extraction of the depth field is carried out, preferably, from two different images, acquired at the same time t, coming from the same scene. However, it should be noted that in both cases, it is possible that objects are occluded by certain other objects of the scene so that the velocity and / or depth field associated with these objects cannot be extracted. Therefore, the displacement of image blocks, according to the velocity and / or depth field (as described in detail above and below) makes it possible to solve this problem.

[0176] In the case of an autonomous vehicle, the extraction of depth information makes it possible, among other things, not to equip the autonomous vehicle with a LIDAR. In this case, the extraction of depth information can be carried out by comparing at least two source images I. During the initialization step, the comparison can advantageously be carried out from the two source images Idt, Igt acquired at time t. In the case where no depth information is provided by a depth information measurement system, the initialization step by comparing the two source images IdtO, IgtO acquired at time tO is necessary so that the reconstructed images MdtO, MgtO include depth information prior to the implementation of the processing phase PO.The extracted depth information is entered in all or part of the pixels of the pixel matrix of the source images IoT, IgO used during the comparison to obtain the reconstructed images MdO, MgO. It should be noted that the extraction can be carried out by comparing as many source images I acquired at the same time t as long as they comprise or correspond to, at least in part, the same scene.

[0177] The extraction of depth information by comparison of at least two source images It acquired at time t, can be implemented, for any initialization step and for any step of updating the depth information (and not only for the initialization step nor only for the source images 10 acquired at time t = 0). In addition, for any source image It (whatever the acquisition time t), if the pitch of the reconstructed image Mt differs from the pitch of the associated source image It, for example due to geometric transformation calculations, it is then appropriate to interpolate the values ​​of the pixels in the frame of reference of the reconstructed image Mt, to obtain a representation matrix at positions compatible with the matrix for example of PSF so that the output coefficients of the convolution Reconstructed image B (*) PSF correspond to the positions of the photosites, without offset.If this precaution is not taken, the quality of the reconstructed image Mt, and therefore the restored image Ct, may contain defects such as shifts between the restored colors, causing, for example, the appearance of colored fringe residues around the elements of the restored image, or geometric micro-distortions linked to these fractional parts in the calculated position indices not taken into account.

[0178] With reference to FIGURES 3 to 5, and for the processing phases P where t>0, the extraction of the depth information, used for the implementation of the reconstruction step, can also be carried out from the comparison:

[0179] - of at least one source image, for example: Idt and Igt, or of at least two reconstructed images, for example: Mdt and Mdt-1 and / or Mgt and Mgt-1 and / or Mdt and Mgt, or of at least two reconstructed images M being processed, for example: Mdt and Mdt-1 and / or Mgt and Mgt-1 and / or Mdt and Mgt, or of at least two restored images, for example: Cdt-1 and Cdt-2 and / or Cgt-1 and Cgt-2 and / or Cdt-1 and Cgt-1, or of at least one source image (Idt or Igt) with at least one restored image M, preferably at a previous processing phase (for example Mdt-1 or Mgt-1) (or at previous processing phases Mdt-2, Mdt-3... or Mgt-2, Mgt-3...), or at least one source image (Idt or Igt) with at least one reconstructed image (Mdt or Mgt) and / or with at least one reconstructed image (Mdt or Mgt) being processed, or at least one source image (Idt or Igt) with at least one restored image C, preferably at a previous processing phase (e.g. Cdt-1 or Cgt-1) (or at previous processing phases Cdt-2, Cdt-3... or Cgt-2, Cgt-3...), or at least one reconstructed image (Mdt or Mgt) and / or at least one reconstructed image (Mdt or Mgt) being processed with at least one restored image C, preferably at a previous processing phase (e.g. Cdt-1 or Cgt-1) (or at previous processing phases Cdt-2, Cdt-3... or Cgt-2, Cgt-3...), or of at least one reconstructed image (Mdt or Mgt) being processed with at least one reconstructed image (Mdt or Mgt).

[0180] Preferably, when the extraction is carried out by comparing a source image I with a reconstructed image M, a reconstructed image M being processed or a restored image C, the source image I is convolved by an inverse function of the PSF, called iPSF, prior to its comparison with the reconstructed image M, the reconstructed image M being processed or the restored image C. This makes it possible to obtain a better correlation between the source image I and the image to which it is compared. The iPSF is chosen so that the composition of the PSF with the iPSF tends to give the identity function at least on the low frequencies of the image (which have overall a good signal-to-noise ratio). The inversion of the PSF is arbitrary because it is not possible to invert (in general amplify) all the frequency bands; only the bands having a good signal-to-noise ratio can be amplified to avoid degrading the image.This is the concept known in the literature, of Wiener filtering: typically, the spatial Fourier transform (in dimension 2) of the PSF is calculated. Preferably, an amplification of the low frequencies is applied up to a certain threshold (typically a frequency that can be previously determined so as not to degrade the image) so as to make a compound PSF gain with flat iPSF up to the threshold. More preferably, the amplitude of the higher frequencies is voluntarily limited to limit the appearance of high-frequency noise and artifacts in the image. Preferably, from the PSF, a certain inversion of the matrix representing it is calculated, at least on the lowest, significant spatial frequencies.From this matrix called iPSF, which can be preferentially specific according to each wavelength, and each position in the field of the image, and one or more pieces of information or data or values ​​contained in a pixel or in one or each group or block of pixels, in particular according to the depth of the object of each object, a convolution product is carried out with the values ​​of the neighboring photosites from the source image I to produce a first reconstructed image value M or of the reconstructed image M being processed to initialize the restitution table.

[0181] Prior to comparing images with each other (source(s) I, reconstructed(s) M, reconstructed(s) M being processed and / or restored(s) C), it may be advantageous to use an entity called geometric transformation (GT) providing a geometric transformation channel of each scene to make it spatially coincide between the different images. This TG is generally not linear to take into account differences in distortions between the imaging systems 2 and that the modeling of the geometric distortions is not perfect, or that the scene has moved relative to one or more imaging systems. For example, the TG of each source image, for example IoT, IgO acquired at t=0, can be calculated for example by seeking correlation optima by moving parts of an IoT source image relative to another IgO source image.The translation that has allowed the best correlation to be obtained is retained for subsequent image comparisons (at t>0). This operation is to be repeated on the set of several parts of each image, not necessarily exhaustively on all parts of the image. Preferably, the translation field obtained is interpolated by an interpolation method, for example linear, or preferably cubic, or any other method that allows such interpolation. It is the set of these interpolations that makes it possible to obtain complete coverage of the image field to obtain each TG operator.

[0182] As mentioned above, for the processing phases P>PO where t>0, and for a processing phase P considered, the reconstruction step is carried out, without extracting depth information from images, by moving image blocks, according to a velocity field and / or a depth field:

[0183] - of the restored image Ct-1 obtained in the processing phase Pt-1, in particular to obtain a reconstructed image Mt (initialization step), or - of the reconstructed image Mt in the processing phase Pt, in particular to obtain the reconstructed image Mt currently being processed (step of updating the depth information prior to one or more of the iterations k>0 of the processing phase Pt), or

[0184] - from the reconstructed image Mt-1 to the processing phase Pt-1, in particular to obtain the restored image Ct-1 (step of updating the depth information carried out during the last iteration k=K2 of the processing phase Pt), or

[0185] - of the source image It, in particular to obtain a reconstructed image Mt (initialization step).

[0186] The initialization step, by extracting depth information from at least two source images Idt, Igt, can be implemented once prior to the first processing phase PO. The subsequent reconstruction steps, implemented beforehand or during the subsequent processing phases (P>PO at t>0) can be implemented without depth extraction but, for example:

[0187] - by moving blocks of an image (source I, reconstructed M, reconstructed M during processing or restored C) according to a speed field and / or a depth field, or

[0188] - by adding and / or modifying depth information in at least part of the pixels of an image (source I, reconstructed M, reconstructed M during processing or restored C) obtained by or originating from a depth information measurement system.

[0189] Preferably, the method 1 comprises the implementation of at least one reconstruction step, preferably at least one initialization step, by processed source images It, that is to say by processing phases Pt.

[0190] The velocity field can also be taken into account during the reconstruction step. The velocity field is preferentially associated with objects identified within an image. Also, in order to extend this field in the parts of the objects where velocity data are missing, the image associated with time t-1 can be moved locally to produce the expected image at time t. For this, each object can be moved according to its own velocities by taking into account the time elapsed between t-1 and t. For example, the best correlation around the displacement of the elements of an image associated with time t-1 could be determined to produce the initialization of the reconstructed image Mt at time t.The correlation is established, for example, between the source image It convolved by the iPSF and the reconstructed image Mt-1, the reconstructed image Mt-1 being processed or, preferably, the restored image Ct-1 in which displacements of pixels or image blocks are carried out according to the velocity field. In other words, this amounts to considering the displacements of objects, according to the depth velocity field and for a time interval between t and t-1 elapsed in the reconstructed image Mt-1, the reconstructed image Mt-1 being processed or, preferably, the restored image Ct-1, varying the position of the objects obtained so as to obtain the best correlation. Thus, the knowledge of hidden objects, as well as their depth and / or their speed, in the source image It are known.

[0191] As specified above, the depth field (Z) can also be taken into account during the reconstruction step to calculate and identify hidden objects at time t. Indeed, according to the invention, the images (reconstructed Mt-1, Mt-2, Mt-3..., reconstructed Mt-1, Mt-2, Mt-3... during processing and restored Ct-1, Ct-2, Ct-3...) coming from previous processing phases Pt-1, Pt-2, Pt-3... are enriched with a depth field. Therefore, it is appropriate to move the objects, generally locally, according to the depth field, preferably by ordering them, according to the depth field, during reconstruction. In practice, only the pixels of the pixel matrix corresponding to the objects in the scene closest to the imaging systems, i.e. those whose position variations are the most marked, are moved.Thus, the reconstructed image Mt-1, the reconstructed image Mt-1 being processed or, preferably, the restored image Ct-1 can, at least in part, serve as a basis for the reconstruction step of the reconstructed image Ct. In addition, the depth information contained in the pixel matrix can be used to generate a stack of object planes at different depths. Each plane will include the objects present at the depth of the plane considered. In this case, the displacement of objects, according to the depth field, can be carried out in each plane of the stack of planes. The calculations can be simplified in that only the objects included in the planes of the scene closest to the imaging systems 2 are displaced. In general, it should be noted that the depth field contained in the images (reconstructed Mt-1, Mt-2, Mt-3..., reconstructed Mt-1, Mt-2, Mt-3... being processed and restored Ct-1, Ct-2, Ct-3...) from previous processing phases Pt-1, Pt-2, Pt-3... can be used for implementing the extraction step, the initialization step (including moving image blocks according to the depth field) and the update step.

[0192] The reconstruction of the reconstructed image M including the consideration of the speed and / or depth field is particularly advantageous because it makes it possible to know the position of hidden objects, as well as their depth in the imaged scene, in the source image It. This information, entered in the pixel matrix without necessarily being restored in the displayed restored image Mt, is crucial in the case of autonomous vehicles because the automatic driving system contains information, not present in and not deducible from the source image It, allowing it, for example, to anticipate the upcoming presence of an obstacle on the trajectory or the detection of a traffic sign or a traffic light which is not visible on the source image It.

[0193] Generally speaking, all the characteristics relating to the depth field (which concern the enriched images, the enrichment of the images, the reconstruction of the images, the processing phase P of the images, the extraction of data, the initialization step, the step of updating the depth data, etc.) are transposable, mutatis mutandis, to the velocity field and vice versa.

[0194] Generally speaking, according to the invention, it is advantageous, when extracting data (depth information, velocity field and depth field) and / or during the initialization step (including when moving objects according to the velocity field, or according to the velocity field and the depth field, when reconstructing the reconstructed image Mt at k>0), to use as the starting base(s) or image(s) (for the extraction and / or for the initialization or for the displacement of blocks), an image restored Ct-1 in the previous processing phase Pt-1 or an image restored Ct-2, Ct-3 ... in one of the previous processing phases Pt-2, Pt-3..., because this will already include all the different modifications made during the previous processing phase Pt-1. This can reduce the iterations to obtain the Ct rendered image at the Pt processing phase compared to the composition method with iPSF for example.Furthermore, the source image It is generally close to the rendered image Ct-1 at the processing phase Pt-1 which achieves the exit criteria of the iteration loop (processing phase), because few details change between it-1 and it, except for the displacement of objects. The convergence of the iteration loop can then be all the faster, that is to say include fewer iterations. Thus, we obtain an image closer to the end-of-iteration image k, than during initialization at it = 0 or during a change of plane. The translation of image parts (according to the velocity field and according to the velocity field and the depth field) is generally less resource-intensive than convolution calculations with PSF. Indeed, they correspond to pixel shifts, and / or interpolations, one-dimensional whereas the PSF is two-dimensional. Thus, this initialization method is particularly advantageous.It generally allows fewer k iterations to be made during the processing phase. It also allows calculations to be performed only on parts of images to be refined, without having to perform calculations on the entire image and for parts where the image already meets the comparison criteria (as is the case in state-of-the-art image processing methods where a succession of steps performed on the entire image are implemented).

[0195] In general, and as detailed above, it is advantageous to extract data and / or implement the initialization step from a restored image Ct-1 in the previous processing phase Pt-1 or from a restored image Ct-2, Ct-3 ... in one of the previous processing phases Pt-2, Pt-3.... The previous processing phase Pt-2, Pt-3... to which it is possible to go back to use the restored image Ct-2, Ct-3... depends, in particular, on the acquisition frequency of the source images I, the speed of movement of the objects in the scene and / or the imaging systems 2 relative to the scene. In addition, the time elapsed between the acquisition of the source image It-2, It-3... associated with the previous processing phase Pt-2, Pt-3... to which it is possible to go back to use the restored image Ct-2, Ct-3...and the acquisition of the source image It corresponding to the current processing phase Pt must not be too high so that the majority of the objects in the scene are still present in the scene. The fact that the objects have moved, even significantly, during this period of time is not critical in that it is possible to identify identical objects (having different positions) on different images. In addition, and in particular for depth and / or speed extraction, it may be interesting to perform multiple comparisons between reconstructed images Ct-1 with Ct-2 and Ct-1 with Ct-3 and Ct-2 with Ct-3... to obtain a trend in the variation of depth and / or speed of the objects in the scene and thus better predict and anticipate the position and / or depth of future objects. This also makes it possible to obtain more precise and reliable depth and / or speed information.

[0196] In general, everything detailed above regarding the extraction of data and / or the implementation of the initialization step from restored images Ct-1, Ct-2, Ct-3... at previous processing phases Pt-1, Pt-2, Pt-3... is directly transposable to the update step carried out from reconstructed images Mt-1, Mt-2, Mt-3... or from reconstructed images Mt-1, Mt-2, Mt-3 being processed at previous processing phases Pt-1, Pt-2, Pt-3... but also from reconstructed images Mt being processed at previous iterations k-1, k-2, k-3... of the iterative loop 3 of the current processing phase Pt.

[0197] However, there is nothing to prevent the initialization step from comprising the reconstruction of the reconstructed image Mt from the source image It, then convolving the thus reconstructed image Mt by the iPSF, then, possibly, moving pixels or blocks of the thus reconstructed convolved image Mt according to the velocity field or according to the velocity field and the depth field as described above. In particular, it is advantageous for the reconstruction step, and in particular the initialization step, to be carried out by extracting the information by comparing the source images Idt, Igt when significant position distortions in the PSF are present, or when focusing aberrations (blur) would be expected, in order to benefit from more precise data in Z prior to the processing phase Pt.Indeed, in this case, the convolution by the iPSF of the source image I, before reconstruction, can be preferable because it allows to take into account distortions of positions and precorrection of the aberrations of focus (correction of the 'blur') which provides more precise and more faithful images of the imaged scene. Consequently, this proves advantageous because the extraction is carried out from more precise images containing fewer errors and therefore allows a better evaluation of the disparity between the compared images and therefore of the depth.

[0198] Thus, in summary, and according to the variants of the invention and / or the embodiments of the invention and / or the combination of variants and embodiments, the invention has the following advantages: not producing a reduction in sharpness at a noise reduction step which must then be compensated for, which is difficult to maintain good homogeneity of the compensation across the entire image, and / or

[0199] - to avoid the appearance of colored fringes often visible around the contours of objects in the image, and / or - to avoid the appearance of color noise (a sort of red, green, blue snow) inside the texture of objects, or more generally in certain areas of the image, and / or

[0200] - to include in the restored image all corrections to take into account distortion, as well as loss of sharpness, in a homogeneous and precise manner across the entire image, and / or

[0201] - to avoid false color effects in parts of the objects' texture, and / or

[0202] - to improve sharpness across the entire depth of field of the image, and beyond the sharpness zone of the hyperfocal distance, provided that the Z of each object is taken into account in the PSF (X, Y, Z), and / or

[0203] - to provide enriched images with a depth field (Z) and / or a velocity field presenting precise depth and / or velocity information, and / or

[0204] - to know the position and / or the movement and / or the depth of hidden objects, that is to say of objects which cannot be observed because they are hidden by other objects in the scene, and / or

[0205] - to predict or forecast the future movement and / or position of objects in the imaged scene and / or the future depth and / or depth changes of said objects.

[0206] Of course, the invention is not limited to the examples which have just been described and numerous adjustments can be made to these examples without departing from the scope of the invention.

[0207] Thus, in variants of the previously described embodiments that can be combined with each other: the reconstructed image Mt is obtained by merging at least two reconstructed images, for example Mt-1 and Mt-2 or Mdt-1 and Mgt-1, and / or the processing phase P comprises a step of merging at least two restored images, for example Cdt and Cgt and / or Cdt-1 and Cdt, to provide at least one restored image Ct, and / or there is proposed according to the invention a device, a system or any apparatus comprising a processing unit implementing any one of the embodiments of the method according to the invention that has just been described for correcting / processing / restored images acquired by one or more imaging systems 2 of said device, said system or said apparatus or for correcting / processing / restored images stored in said device, said system or said apparatus, said device, said system or said apparatus being able to be,by way of non-limiting examples: a smart mobile phone (or smartphone), a computer, a camera, a vehicle, a drone, a medical device or a satellite, and / or it is proposed according to the invention a device, a system, a machine (preferably motorized) or any device comprising a processing unit implementing any of the embodiments of the method according to the invention just described for correcting / processing / restore images acquired by one or more imaging systems 2 of said device, said system, said machine or said device or for correcting / processing / restore images stored in said device, said system, said machine or said device, said device, said system, said machine or said device being able to be, by way of non-limiting examples: a smart mobile phone (or smartphone), a tablet, a camera, a vehicle, a drone, a medical device or a satellite,and / or it is proposed to use the restored images Ct, preferably a stream of restored images Ct, obtained by implementing the method according to the invention from images, preferably a stream of images, preferably originating from a smartphone, to feed, preferably with a continuous stream of restored images Ct, a virtual three-dimensional space (for example, the metaverse), and / or to use any of the embodiments of the device according to the invention just described and / or any of the embodiments of the method according to the invention just described within a device, a system, a machine or an apparatus just described or within a processing unit from images, preferably a stream of images, originating from a device, a system, a machine or an apparatus just described, and / or it is proposed according to the invention a computer program including instructions which,when the program is executed by a computer, lead the latter to implement the method according to any one of the embodiments described, and / or there is provided according to the invention a support readable, in particular, by computer or by any device comprising a processing unit comprising instructions which, when executed by said computer or said device, lead the latter to implement the method according to any one of the embodiments described.,

[0208] In addition, the various features, forms, variations and embodiments of the invention may be combined with each other in various combinations to the extent that they are not incompatible or mutually exclusive.

Claims

CLAIMS 1. Method for restoring (1) at least one image from one or more source images, each source image being an image of a scene obtained by an optical sensor of an imaging system (2), said method is implemented by a machine, a vehicle or a drone and comprises: - a processing phase P of a reconstructed image M, associated with a source image I, at least some of the pixels of which comprise depth information and / or at least speed information, at the end of the processing phase a restored image C is obtained for each reconstructed image processed, the processing phase comprises an iterative modification of the reconstructed image so as to minimize a function E comprising: • a term D, called difference term, determined by comparison of the reconstructed image convolved by a point spread function (PSF) with the source image, to which the reconstructed image is associated, the PSF describing the response of the imaging system from which the source image used for the comparison is obtained, and • a term A, called anomaly term, representative of defects or anomalies within the reconstructed image, determined from the reconstructed image.

2. Method (1) according to the preceding claim, comprising prior to and / or, respectively, during the processing phase, a step of reconstructing the reconstructed image M or respectively the reconstructed image currently being processed from at least one source image and / or at least one reconstructed image and / or at least one reconstructed image M currently being processed and / or at least one restored image C.

3. Method (1) according to the preceding claim, comprising an extraction of the depth information and / or the speed information of the imaged scene by comparison of at least two source images I or of at least two reconstructed images M or of at least two reconstructed images M currently being processed or of at least two restored images C or of at least one source image with at least one restored image or with a reconstructed image or with a reconstructed image currently being processed.

4. Method (1) according to claim 2 or 3, in which the reconstruction step comprises a convolution of the source image I, to which the reconstructed image is associated, by an inverse function of the PSF, called iPSF, describing the response of the imaging system from which the source image, convolved by the iPSF, is obtained.

5. Method (1) according to any one of claims 2 to 4, in which the reconstruction step comprises a movement of blocks: - according to a speed field, of a source image I, of a reconstructed image M, of a reconstructed image M being processed or of a restored image C, and / or - according to a depth field, of a reconstructed image M, of a reconstructed image M being processed or of a restored image C.

6. Method (1) according to any one of the preceding claims, in which the term D is determined by comparison of: - the reconstructed image M convolved with the PSF or the reconstructed image M being processed convolved with the PSF, with - the source image I, to which the reconstructed image is associated.

7. Method (1) according to the preceding claim, comprising, prior to the comparison of the reconstructed image M convolved with the PSF or of the reconstructed image M currently being processed convolved with the PSF with the source image I, with which the reconstructed image is associated, a spreading, at least locally, according to a velocity field and / or according to an acquisition duration of said source image I, of pixel(s) or group(s) or block(s) of pixels of said reconstructed image convolved with the PSF or of said reconstructed image currently being processed convolved with the PSF.

8. Method (1) according to any one of the preceding claims, in which the processing phase P comprises, prior to the convolution of the reconstructed image M or of the reconstructed image M being processed with the PSF, a selection of the PSF.

9. Method (1) according to any one of the preceding claims, in which the iterative modification (3) of the reconstructed image M ends when the function E, or a combination of partial derivatives of the function E with respect to the reconstructed image M being processed, is less than a minimization threshold, or when a certain number of iterations of the iterative modification of the reconstructed image is reached; the at least one reconstructed image thus modified is restored.

10. Method (1) according to any one of the preceding claims, in which the reconstructed image M is obtained by merging at least two reconstructed images.

11. Method (1) according to any one of the preceding claims, in which the processing phase P comprises a step of merging at least two restored images C to provide at least one restored image.

12. Method (1) according to any one of the preceding claims, in which, for an imaging system (2) considered, the PSF depends on: - an orientation of an optical axis of the imaging system considered relative to at least one other optical axis of another imaging system, and / or - a relative position of the imaging system considered with respect to at least one other imaging system, and / or - a distance between the sensor of the imaging system considered and an object imaged by the sensor of the imaging system considered, and / or - a distance between a part of the imaging system considered and the at least one sensor, and / or - a distance between a part of the imaging system considered and an object imaged by the sensor of the imaging system considered, and / or - a state of the imaging system, such as a zoom or focus or numerical aperture setting of the imaging system, and / or - of the pixel or of a group or of a block of pixels of a source image or of a reconstructed image or of a restored image and / or of a photosite of the sensor of the imaging system considered, and / or - one or more pieces of information contained in a pixel or a group or block of pixels of an image, and / or - one or more angles between the optical sensor of the imaging system considered and the imaging system.

13. Method (1) according to claim 2 or any one of claims 3 to 12 taken in combination with claim 2, in which the at least one source image I comprises a set of source images consisting of a temporal succession of source images (Itl, It2...) acquired at different times (tl, t2...), each processing phase (Pl, P2...), each reconstruction step, each reconstructed image (Ml, M2...) and each restored image (Cl, C2...) are associated with a source image obtained at a given time t.

14. Method (1) according to claim 13, in which the reconstructed image Mt, associated with the source image It, is obtained by comparison: - of an image obtained by displacements of blocks of the restored image Ct-1, or of the reconstructed image Mt-1 or of the reconstructed image Mt-1 being processed, at the processing phase Pt-1, with - the source image It, to which the reconstructed image Mt is associated, convolved by the inverse function of the PSF, called iPSF, describing the response of the imaging system (2) from which the source image It is obtained.

15. Method (1) according to claim 13 or 14, in which the reconstructed image Mt is obtained from the source image It with which the reconstructed image Mt is associated.

16. Method (1) according to any one of claims 13 to 15, in which the reconstructed image Mt, associated with the source image It, is obtained from the restored image Ct-1 obtained in the processing phase Pt-1.

17. Method (1) according to claim 2 or any one of claims 3 to 12 taken in combination with claim 2, in which the at least one source image I comprises a set of source images (Id, Ig...) acquired by different imaging systems (d, g...), each processing phase (Pd, Pg...), each reconstructed image (Md, Mg...) and each restored image (Cd, Cg...) are associated with a source image I acquired by a different imaging system (2); the imaging systems being arranged so as to acquire, at least in part, the same scene.

18. Method (1) according to the preceding claim, in which the step of reconstructing the reconstructed image Mt is carried out from: - of a source image Id or a rendered image Cd, and - from a source image Ig or a rendered image Cg.

19. Method (1) according to claim 17 or 18 taken in combination with any one of claims 13 to 16, in which each of the source images of the set of source images (Idtl, Idt2, Igtl, Igt2...) is acquired by a separate imaging system (2) (d, g...) at a given time (tl, t2...), each processing phase (Pdtl, Pdt2, Pgtl, Pgt2...), each reconstructed image (Mdtl, Mdt2, Mgtl, Mgt2...) and each restored image (Cdtl, Cdt2, Cgtl, Cgt2...) are associated with a source image (Idtl, Idt2, Igtl, Igt2...) obtained by a given imaging system and at a given time t.

20. Data processing device comprising means arranged and / or programmed and / or configured to implement the method (1) according to any one of claims 1 to 19.

21. Computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method (1) according to any one of claims 1 to 19.

22. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement the method (1) according to any one of claims 1 to 19.

Images