Method for obtaining a stack of images, and associated computer program product, processing device and electronic apparatus
Patent Information
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- FOGALE OPTIQUE
- Filing Date
- 2023-07-08
- Publication Date
- 2026-05-13
AI Technical Summary
Conventional image acquisition methods require a large number of images at varying focusing distances to achieve satisfactory sharpness, leading to increased storage needs and potential errors due to scene movement between shots.
A method that determines optical transfer functions between the scene and the camera sensor for each focusing distance, allowing for deconvolution of input images to produce a stack of corrected images with reduced image requirements, improved sharpness, and lower storage needs.
This approach reduces the number of images needed for sharpness, minimizes delays due to movement, and enhances image quality while reducing memory footprint.
Smart Images
Figure FR2023051060_16012025_PF_FP_ABST
Abstract
Description
Title: Method for obtaining a stack of images, computer program product, processing device and associated electronic apparatus DESCRIPTION Technical field
[0001] The present invention relates to a method of obtaining an image stack.
[0002] The invention also relates to a computer program, a processing device implementing such a method, and an electronic apparatus comprising such a processing device.
[0003] The invention applies to the field of image processing. State of the art
[0004] When acquiring a representative image of a scene using a camera module, it is known to adjust the camera module so that its focus distance corresponds to a distance at which an object of interest in the scene is located. A consequence of such an adjustment is that elements of the scene located at a distance from the camera module equal to the focus distance appear sharper than other elements of the scene.
[0005] In other words, for each image, elements of the scene that are located, relative to the camera module, at a distance different from the focusing distance generally have optical aberrations and / or defocus blur.
[0006] To clearly visualize each area of the scene, it is known to vary the focus distance and acquire an image for each focus distance: we then obtain a stack of images with different sharp areas from one image to another, called a stack of focus bracketing images.
[0007] However, such a process is not satisfactory.
[0008] Indeed, such a method requires the acquisition of a large number of images, at several focusing distances corresponding to an exhaustive coverage of the observed scene, according to a fairly dense discretization step, to obtain satisfactory sharpness of the entire image stack.
[0009] This results in a large stock of images to memorize. In addition, the image capture process is lengthened proportionally to the number of images acquired. This results in a risk that, in this pile of images, the scene contains movements that make the restitution of an image by assembly subject to errors in connection of objects that have moved between shots.
[0010] An aim of the present invention is to remedy at least one of the drawbacks of the state of the art.
[0011] Another object of the invention is to propose a method for obtaining a stack of images which requires fewer images than the known methods for obtaining satisfactory sharpness of the entire stack of images, without requiring significant storage capacities. Statement of the invention
[0012] To this end, the invention relates to a method for obtaining an image stack from a plurality of input images, each input image depending on a respective raw image, the raw images being representative of the same scene, each raw image being acquired by a sensor of a camera module through an optical system of the camera module, for a respective focusing distance of the optical system, the method being implemented by computer and comprising, for each input image, the steps: - determination of at least one optical transfer function between the scene and the sensor, for the focusing distance corresponding to the respective raw image; and - for each determined optical transfer function, calculation of a corrected image by deconvolution of the input image from said optical transfer function, so as to obtain a set of corrected images, the stack of images obtained depending on the set of corrected images.
[0013] Indeed, a consequence of the deconvolution step is that the number of images required to obtain satisfactory sharpness of the entire image stack is greatly reduced. As a result, any offsets between the images taken, when they include moving elements (or when the camera module itself moves) are also reduced. Thus, the discretization step between the focus planes can be chosen to be larger than in conventionally known methods. Consequently, the memory footprint occupied by the image stack is reduced.
[0014] The process also allows fewer images to be stored in memory, while improving the overall quality of the rendering.
[0015] Advantageously, the method according to the invention has one or more of the following characteristics, taken in isolation or in any technically possible combination:
[0016] for each input image, the at least one optical transfer function is the optical transfer function between, on the one hand, a plane located, relative to the camera module, at a distance equal to the corresponding focusing distance, and, on the other hand, the sensor of the camera module;
[0017] for each input image, each optical transfer function is the optical transfer function between: - on the one hand, a set of corresponding points of the scene located at the same distance, within a predetermined distance margin, from the camera module; and - on the other hand, the camera module sensor;
[0018] for each input image, each optical transfer function is the optical transfer function between: - on the one hand, a set of corresponding points of the scene presenting, with respect to an optical axis of the optical system of the camera module, the same angle of observation from the camera module, to within a predetermined angular margin; and - on the other hand, the camera module sensor;
[0019] the method further comprises establishing, for each input image, a transfer function map, the transfer function map associating, with each point of the scene represented on said input image, the corresponding optical transfer function implemented during the step of calculating the corrected image;
[0020] for each input image, and for each associated optical transfer function, the calculation of the corresponding corrected image includes: - initial calculation of an estimated image, by deconvolution of the optical transfer function of the input image; - iterative implementation of a processing loop comprising: • calculation of a current simulated image from the current estimated image and the optical transfer function; • determination of a cost function from at least one of the current simulated image and the current estimated image; • if a predetermined stopping criterion is reached, exit from the processing loop, otherwise, update of the estimated image from the determined cost function; - providing, as output, the corrected image as being the current estimated image;
[0021] the method further comprises, for each corrected image, a calculation of a sharpness index in at least one part of the corrected image;
[0022] for each pixel of at least one part of the corrected image, the sharpness index is a value taken, at said pixel, by a predetermined variance estimator, a predetermined entropy estimator or a predetermined energy estimator;
[0023] the method further comprises establishing a sharpness map, the sharpness map associating, with each point of the scene, the corrected image, from among the set of corrected images, in which the value of the sharpness index of the pixel representing said point of the scene is maximum;
[0024] the method further comprises generating, from all the corrected images, a fused image, the fused image being such that, for each point of the acquired scene, the pixel representing said point of the scene in the fused image has a value equal to the value of the pixel representing said point of the scene in the corrected image associated with said point of the scene via the sharpness map;
[0025] the method further comprises, in response to an input instruction representative of an area of interest in the scene, a calculation of a restored image, the restored image being such that, for each point of the area of interest, the value of the pixel representative of said point in the restored image is equal to the value of the pixel representative of said point in the corrected image associated with said point of the area of interest via the sharpness map;
[0026] the restored image is such that, for each point outside the area of interest: - the value of the pixel representing said point in the restored image is equal to the value of the pixel representing said point in a corrected image of the set of corrected images; and - the sharpness index of the pixel representing said point in the restored image is lower than a minimum sharpness index among the pixels of the restored image representing points in the area of interest.
[0027] According to another aspect of the invention, there is provided a computer program comprising executable instructions which, when executed by computer, implement the steps of the method as defined above.
[0028] The computer program can be in any computer language, such as machine language, C, C++, JAVA, Python, etc.
[0029] According to another aspect of the invention, there is provided an image processing device for obtaining a stack of images from a plurality of input images, each input image depending on a respective raw image, the raw images being representative of the same scene, each raw image being acquired by a sensor of a camera module through an optical system of the camera module, for a respective focusing distance of the optical system, the image processing device being configured to: - determine at least one optical transfer function between the scene and the sensor, for the focusing distance corresponding to the respective raw image; and - for each determined optical transfer function, calculating a corrected image by deconvolution of the input image from said optical transfer function, so as to obtain a set of corrected images, the stack of images obtained depending on the set of corrected images.
[0030] The device according to the invention can be any type of device such as a server, a computer, a tablet, a calculator, a processor, a computer chip, programmed to implement the method according to the invention, for example by executing the computer program according to the invention.
[0031] The device according to the invention can be any type of device such as a server, a computer, a tablet, a calculator, a processor, a computer chip, programmed to implement the method according to the invention, for example by executing the computer program according to the invention.
[0032] According to another aspect of the invention, there is provided an electronic apparatus comprising a processing device as defined above and a camera module configured to acquire a plurality of raw images, the processing device being connected to the camera module to receive, as input, all or part of the raw images acquired by the camera module.
[0033] Preferably, the electronic device is a smart mobile phone (or "smartphone" in English), also called a ordiphone or multifunction mobile phone).
[0034] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the smart mobile telephone, for obtaining a stack of images from a plurality of raw images acquired by the camera module of the smart mobile telephone.
[0035] Alternatively, the electronic device is a touchscreen tablet.
[0036] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the touchscreen tablet, to obtain a stack of images from a plurality of raw images acquired by the camera module of the touchscreen tablet.
[0037] Alternatively, the electronic device is a computer.
[0038] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the computer, for obtaining a stack of images from a plurality of raw images acquired by the camera module of the computer.
[0039] Alternatively, the electronic device is a television.
[0040] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the television, to obtain a stack of images from a plurality of raw images acquired by the camera module of the television.
[0041] Alternatively, the electronic device is a virtual reality headset.
[0042] In this case, the invention also relates to a use of a treatment device according to the invention and / or a method according to the invention, within the virtual reality headset, to obtain a stack of images from a plurality of raw images acquired by the camera module of the virtual reality headset.
[0043] Alternatively, the electronic device is a medical imaging device, for example an endoscope.
[0044] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the medical imaging device, to obtain a stack of images from a plurality of raw images acquired by the camera module of the medical imaging device.
[0045] Alternatively, the electronic device is a vehicle, autonomous or not, such as a land vehicle (e.g., a car), an aircraft (e.g., a drone, an airplane, a helicopter, etc.) or a maritime vehicle (e.g., a boat, a submarine, etc.).
[0046] In this case, the invention also relates to a use of a processing device according to the invention and / or a method according to the invention, within the vehicle, to obtain a stack of images from a plurality of raw images acquired by the camera module of the vehicle. Brief description of the figures
[0047] The invention will be better understood on reading the description which follows, given solely as a non-limiting example and made with reference to the appended drawings in which:
[0048] Figure 1 is a schematic representation of an electronic apparatus comprising an image processing device according to the invention;
[0049] Figure 2 is a flowchart of a method for obtaining a stack of images according to the invention, implemented by the image processing device of Figure 1;
[0050] Figure 3 is a schematic representation of a first cutting of an image during the implementation of the method of Figure 2; and
[0051] Figure 4 is a schematic representation of a second cutting of an image during the implementation of the method of Figure 2.
[0052] It is understood that the embodiments which will be described below are in no way limiting. In particular, it is possible to imagine variants of the invention comprising only a selection of characteristics described below isolated from the other characteristics described, if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection includes at least one preferably functional characteristic without structural details, or with only part of the structural details if it is this part which is only sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.
[0053] In particular, all the variants and embodiments described can be combined with each other if there is no technical obstacle to this combination.
[0054] In the figures and in the rest of the description, the elements common to several figures retain the same reference. Detailed description
[0055] An image processing device 2 (hereinafter, “processing device”) according to the invention is illustrated in FIG. 1.
[0056] The processing device 2 is configured to provide a stack of output images, from a plurality of raw images IB representative of a scene 3 and acquired by a camera module 4. In particular, each raw image IB has been acquired for a corresponding focusing distance of the camera module 4.
[0057] The images of the image stack provided by the processing device 2 advantageously have better sharpness and / or better resolution and / or less noise than the raw images IB acquired by the camera module 4.
[0058] The processing device 2 and the camera module 4 are, for example, both integrated into the same electronic device 6. In this case, the processing device 2 is connected to the camera module 4 to receive, as input, each of the plurality of raw images IB acquired by the camera module 4.
[0059] The electronic device is, for example, a smartphone, a tablet, or a digital camera. Alternatively, the electronic device 6 is a video surveillance device, a vehicle, a drone, a satellite, a microscopy device, or a medical imaging device (such as a tomography imaging device).
[0060] Preferably, the electronic device 6 also comprises a human / machine interface 8 for the input, by a user, of an input instruction representative of an area of interest of the scene 3. Alternatively, or additionally, the human / machine interface 8 is capable of displaying a restored image. Such a restored image will be described more precisely below.
[0061] Optionally, the electronic device 6 also comprises a distance measurement module 10, configured to deliver distance information representative of distances between all or part of the objects of the scene 3 and a sensor and / or an optical system (described later) of the camera module 4. The measurement module 10 is, in particular, a rangefinder, for example an optical rangefinder. Camera module
[0062] The camera module 4 comprises at least one assembly comprising an optical system 12 associated with a sensor 14.
[0063] As mentioned previously, the camera module 4 is adapted to acquire a plurality of raw images IB representative of the same scene 3, at different focusing distance conditions (for example by moving the optical system 12 relative to the sensor 14, or by modifying a shape of a lens of the optical system 12 such as a liquid lens), and preferably from the same position.
[0064] The camera module 4 is associated with at least one optical transfer function between the scene 3 and the sensor 14. Such an optical transfer function establishes a relationship between the characteristics (spatial and spectral, in particular) of the light coming from the scene 3 and the characteristics of the light reaching the sensor 14, in particular due to the influence of the optical system 12.
[0065] Each optical transfer function of the camera module 4 is, for example, described by means of a point spread function of said camera module 4.
[0066] Each optical transfer function of the camera module 4 depends, in particular, on a focusing distance of the camera module 4. Each optical transfer function is also likely to depend on at least one of: - a distance between the sensor 14 and an object of the scene 3 imaged by the sensor 14; - a distance between the optical system 12 (in particular the optical center of the optical system 12) and the sensor 14; - a distance between the optical system 12 (in particular the optical center of the optical system 12) and an object of the scene 3 imaged by the sensor 14; - a state of the optical system 12, such as a zoom, focal length and / or numerical aperture setting; - a photosite considered in a matrix of photosites of the sensor; and - one or more angle(s) between the sensor 14 and the optical system 12.
[0067] The distance between the sensor 14 and each imaged object and / or the distance between the optical system 12 and each object of the scene 3 is, for example, measured by means of the distance measurement module 10. of treatment
[0068] The processing device 2 may be in a hardware form, such as a computer, a server, a processor, an electronic chip, etc. Alternatively, or additionally, the processing device 2 may be in a software form such as a computer program, or an application, for example an application for a user device such as a tablet or smartphone.
[0069] The processing device 2 is configured to implement a processing method 20 (FIG. 2) in order to provide, as output, a stack of images, from the plurality of raw images IB.
[0070] As illustrated in FIG. 2, the processing method 20 comprises a reception step 22, an optional preprocessing step 24, a step 26 of determining an optical transfer function (hereinafter, “determination step”) and a step 28 of calculating corrected images (hereinafter, “calculation step”).
[0071] Preferably, the processing method 20 also comprises a restitution step 30. Reception step 22
[0072] The processing device 2 is configured to receive, during the reception step 22, the plurality of raw images IB from the camera module 4.
[0073] As previously stated, each received raw IB image is representative of scene 3, and was acquired for a corresponding focus distance of camera module 4.
[0074] Alternatively, the plurality of raw images IB is already stored in a memory of the processing device 2.
[0075] According to another alternative, the plurality of raw images IB was acquired by another electronic device, then transmitted to the processing device 2 for processing.
[0076] Optionally, the processing device 2 is configured to apply, during the preprocessing step 24, at least one predetermined preprocessing to each raw image IB.
[0077] Such preprocessing is, for example, demosaicing, possibly followed by pixel gain correction.
[0078] In this case, for each raw image IB, the corresponding image resulting from the implementation of the preprocessing step forms a respective input image IE.
[0079] Alternatively, the processing device 2 is not configured to implement a preprocessing step. In this case, each raw image IB of the plurality of raw images IB received during the reception step 22 forms an input image IE. Determination step 26
[0080] The processing device 2 is also configured to determine, during the determination step 26, and for each input image IE, at least one corresponding optical transfer function between the scene 3 and the sensor 14.
[0081] More specifically, for each input image IE, the processing device 2 is configured to determine at least one optical transfer function between the scene 3 and the sensor 14, for the focusing distance corresponding to the respective raw image IB. • Only one optical transfer function per input image
[0082] According to a first variant, for each input image IE, the processing device 2 is configured to determine a single corresponding optical transfer function.
[0083] Advantageously, for any given input image IE, the optical transfer function determined by the processing device 2 is the optical transfer function between: - on the one hand, a plane located, relative to the camera module 4, at a distance equal to the focusing distance corresponding to the respective raw image IB; and - on the other hand, sensor 14 of the camera module.
[0084] In this case, by "distance from the camera module" is meant a distance from the (optical) center of the optical system 12. However, since the distance between the optical system 12 and the sensor 14 is generally of the order of only a few millimeters, a distance measured from the optical center of the optical system 12 is considered equivalent to a distance measured from the sensor 14.
[0085] Such a characteristic is advantageous, insofar as, in this variant, for the input image IE considered, the optical transfer function described above is the most representative of the aberrations introduced by the camera module 4 during the acquisition of the corresponding raw image. It is therefore, for this variant, the optimal optical transfer function (with regard to the elements located, relative to the camera module 4, at a distance close to the focusing distance) for the improvement of the input image IE by deconvolution.
[0086] For example, for each input image IE, the processing device 2 is configured to determine the corresponding optical transfer function using calibration data from a calibration previously carried out on the camera module 4. • Case of a plurality of optical transfer functions per input image
[0087] Alternatively, for each input image IE, the processing device 2 is advantageously configured to determine a plurality of corresponding optical transfer functions.
[0088] Such an approach is advantageous, since the optical transfer function is not necessarily spatially uniform.
[0089] In this case, for each input image IE, the processing device 2 is preferably configured to determine each corresponding optical transfer function as being the optical transfer function between: - on the one hand, a set of corresponding points of scene 3 located at the same distance from the camera module, within a predetermined distance margin; and - on the other hand, the sensor 14 of the camera module 4.
[0090] This corresponds to the example of figure 3, in which each sector 32 (here rectangular sectors) corresponds to a part of the input image IE representing a set of points located substantially at the same distance from the camera module 4.
[0091] For example, information indicative of the different sets of points located substantially at the same distance from the camera module 4 is received from the distance measurement module 10.
[0092] Naturally, the parts of the input image IE representing a set of points located at the same distance from the camera module 4 do not necessarily define rectangles, but are likely to have various shapes, depending on the objects present in the scene 3.
[0093] As before, by “distance from the camera module” is meant a distance from the (optical) center of the optical system 12, or a distance from the sensor 14.
[0094] Such a characteristic is advantageous, insofar as, in each sector 32, the optical transfer function can be assimilated to a uniform function on said sector. This makes it possible to carry out a deconvolution in the frequency domain requiring relatively short calculation times.
[0095] Alternatively, for each input image IE, the processing device 2 is preferably configured to determine each optical transfer function as the optical transfer function between: - on the one hand, a set of corresponding points of the scene 3 having, with respect to an optical axis A of the optical system 12 of the camera module 4, the same observation angle 0 from the camera module 4, to within a predetermined angular margin; and - on the other hand, the camera module sensor.
[0096] This corresponds to the situation of figure 4, in which each of the circular sector 34, the annular sectors 36 and the complementary sector 38, corresponds to a part of the input image IE representing a set of points situated substantially on the same cone whose vertex is the camera module 4 (in particular the optical center of the optical system 12, or possibly the sensor 14).
[0097] By "observation angle relative to the camera module" is meant an angle between the optical axis A and any straight line passing through the (optical) center of the optical system 12, or through the sensor 14.
[0098] Such a characteristic is advantageous, insofar as the physical properties of the optical system 12 generally have a symmetry of revolution around the optical axis, so that the optical transfer function can be likened to a uniform function on each sector 34, 36 and / or 38 (in particular if all the points of the scene of one of the angular sectors are located at the same distance from the camera module 4).
[0099] Advantageously, the processing device 2 is also configured to establish a transfer function map for each input image IE. More precisely, for each input image IE, the corresponding transfer function map associates, with each point of the scene represented on said input image IE, the corresponding determined optical transfer function.
[0100] Advantageously, the processing device 2 is also configured to establish a distance map for each input image IE, representative of the distance at which each point of the scene is located relative to the camera module 4.
[0101] The processing device 2 is also configured to calculate, during the calculation step 28, a corrected image le for each input image IE and for each corresponding optical transfer function determined during the determination step 26. More precisely, for each input image IE, and for each associated optical transfer function, the processing device 2 is configured to calculate the corresponding corrected image le by deconvolution of the input image IE from said optical transfer function.
[0102] The result is a set of corrected images.
[0103] Preferably, for each input image IE, and for each associated optical transfer function, the processing device 2 is configured to calculate the corresponding corrected image by implementing an algorithm comprising: - an initial calculation of an estimated image, by deconvolution of the optical transfer function of the input image IE; - an iterative implementation of a processing loop comprising: • calculation of a current simulated image from the current estimated image and the optical transfer function; • determination of a cost function from at least one of the current simulated image and the current estimated image; • if a predetermined stopping criterion is reached, exit from the processing loop, otherwise, update of the estimated image from the determined cost function; - providing, as output, the corrected image as being the current estimated image.
[0104] Alternatively, for each input image IE, and for each associated optical transfer function, the processing device 2 is configured to calculate the corresponding corrected image by deconvolution by composition by a correction function resulting from the optical transfer function (for example, Wiener filter technique).
[0105] For example, in the case where, for each input image IE, the processing device 2 has determined a plurality of corresponding optical transfer functions, the processing device 2 is preferably configured to carry out, for each zone of the input image IE, a deconvolution by the corresponding optical transfer function.
[0106] Advantageously, the processing device 2 is also configured to calculate, for each corrected image le, a sharpness index in at least one part of the corrected image le, and preferably in each pixel of the corrected image le.
[0107] For example, for at least one pixel of the corrected image, the processing device 2 is configured to calculate the sharpness index as being the value taken, at said pixel, by a predetermined variance estimator, a predetermined entropy estimator or a predetermined energy estimator.
[0108] Furthermore, the processing device 2 is advantageously configured to establish a sharpness map.
[0109] More precisely, such a sharpness map associates, with each point of scene 3, the corrected image le, among the set of corrected images le calculated, in which the value of the sharpness index of the pixel representing said point of scene 3 is maximum. In other words, the sharpness map associates with each point of scene 3 a respective corrected image le, which is the corrected image le (among the set of corrected images le calculated) in which the value of the sharpness index of the pixel representing said point of scene 3 is maximum.
[0110] Alternatively, or additionally, the sharpness map associates, with each point of scene 3, the pixel representative of said point of scene 3 for which, among the set of corrected images, the value of the sharpness index of the pixel is maximum. Subsequently, such a pixel is called “optimal pixel”.
[0111] Alternatively, the sharpness map is represented by lists of shapes of boundaries of the same sharpness zones. In this case, each element of the list corresponds to the same corrected image index. This variant has the advantage of reducing the memory footprint. Such a variant is generally advantageous if the number of different sharpness zones is reduced, in particular because it requires a longer calculation time than the method described previously. Restitution step 30
[0112] Advantageously, the processing device 2 is also configured to generate, during the optional restitution step 30, a merged image IF from all of the corrected images 1c. In particular, the processing device 2 is configured to generate the merged image IF such that, for each point of the scene 3, the pixel representing said point of the scene 3 in the merged image IF has a value equal to the value of the pixel representing said point of the scene 3 for which the value of the sharpness index is maximum among all of the corrected images.
[0113] In other words, the fused image IF is such that, for each point of scene 3, the pixel representing said point of scene 3 in the fused image IF has a value equal to the value of the pixel representing said point of scene 3 in the corrected image associated with said point of scene 3 via the sharpness map.
[0114] Therefore, for each point of scene 3, the pixel representative of said point of scene 3 in the fused image IF is the corresponding optimal pixel.
[0115] Preferably, the processing device 2 is also configured to transmit the fused image IF to the human / machine interface 8 for display.
[0116] More preferably, the processing device 2 is configured to receive, from the human / machine interface 8, an input instruction representative of an area of interest in the scene 3. For example, such an instruction is representative of the selection by a user, by the through the human / machine interface 8, of an area of a raw image IB (OR of the fused image IF) which is of interest to him.
[0117] In this case, the processing device 2 is configured to calculate an IR restored image such that, for each point of the area of interest, the value of the pixel representing said point in the IR restored image is equal to the value of the pixel representing said point in the corrected image associated with said point of the area of interest via the sharpness map.
[0118] In other words, in the rendered image I , each point of the area of interest is represented by the corresponding optimal pixel.
[0119] This is advantageous, as the area of interest in the rendered IR image has improved sharpness compared to the raw IB images.
[0120] Preferably, the processing device 2 is also configured to transmit the restored IR image to the human / machine interface 8 for display.
[0121] Advantageously, the processing device 2 is also configured so that, in the restored IR image, for each point outside the area of interest, the sharpness index of the pixel representing said point in the restored image is less than a minimum sharpness index among the pixels of the restored image representative of points in the area of interest.
[0122] Such a feature is advantageous, as it gives elements outside the area of interest, in the rendered IR image, less sharpness compared to those present in the area of interest. This allows the user to focus on the elements in the area of interest.
[0123] Furthermore, in the rendered IR image, for each point outside the area of interest, the value of the pixel representative of said point is preferably equal to the value of the pixel representative of said point in a corrected image of the set of corrected images.
[0124] Alternatively, in the restored IR image, for each point outside the area of interest, the value of the pixel representing said point is equal to the value of the pixel representing said point in a corrected image, a raw image IB OR an input image IE, to which a low-pass spatial filter has been applied.
[0125] This is advantageous, as processing an image with a low-pass filter reduces the sharpness of such an image. Functioning
[0126] The operation of the processing device 2 will now be described with reference to Figure 2.
[0127] During the reception step 22, the processing device 2 receives each raw image IB previously acquired by the camera module 4.
[0128] Then, during the optional preprocessing step 24, the processing device 2 applies at least one predetermined preprocessing to each raw image IB, so as to obtain a plurality of input images IE.
[0129] Then, during the determination step 26, the processing device 2 determines, for each input image IE, at least one corresponding optical transfer function between the scene 3 and the sensor 14.
[0130] Then, during the calculation step 28, the processing device 2 calculates, for each input image IE and for each corresponding optical transfer function, a corresponding corrected image le. This results in a set of corrected images le.
[0131] Preferably, the processing device 2 also calculates, for each corrected image le, a sharpness index in at least one part of the corrected image le.
[0132] Advantageously, the processing device 2 also establishes a sharpness map for all of the corrected images.
[0133] Then, during the optional restitution step 30, the processing device 2 generates a merged image IF from all of the corrected images.
[0134] Preferably, the processing device 2 also calculates a restored IR image as a function of an input instruction representative of an area of interest in the scene 3.
[0135] Of course, the invention is not limited to the examples which have just been described.
Claims
CLAIMS 1. Method for obtaining an image stack from a plurality of input images (IE), each input image (IE) depending on a respective raw image (IB), the raw images (IB) being representative of the same scene (3), each raw image (IB) being acquired by a sensor (14) of a camera module (4) through an optical system (12) of the camera module (4), for a respective focusing distance of the optical system (12), the method being implemented by computer and comprising, for each input image (IE), the steps: - determination (26) of at least one optical transfer function between the scene (3) and the sensor (14), for the focusing distance corresponding to the respective raw image (IB); and - for each determined optical transfer function, calculation (28) of a corrected image (le) by deconvolution of the input image (IE) from said optical transfer function, so as to obtain a set of corrected images (le), the stack of images obtained depending on the set of corrected images (le).
2. Method according to claim 1, wherein, for each input image (IE), the at least one optical transfer function is the optical transfer function between, on the one hand, a plane located, relative to the camera module (4), at a distance equal to the corresponding focusing distance, and, on the other hand, the sensor of the camera module (4).
3. Method according to claim 1, in which, for each input image (IE), each optical transfer function is the optical transfer function between: - on the one hand, a set of corresponding points of the scene (3) located at the same distance, within a predetermined distance margin, from the camera module (4); and - on the other hand, the camera module sensor (4).
4. Method according to claim 1 or 2, wherein, for each input image (IE), each optical transfer function is the optical transfer function between: - on the one hand, a set of corresponding points of the scene (3) having, with respect to an optical axis of the optical system of the camera module (4), the same observation angle from the camera module (4), to within a predetermined angular margin; and - on the other hand, the camera module sensor (4).
5. Method according to any one of claims 1 to 4, further comprising establishing, for each input image (IE), a transfer function map, the transfer function map associating, with each point of the scene (3) represented on said input image (IE), the corresponding optical transfer function implemented during the step of calculating the corrected image (le).
6. Method according to any one of claims 1 to 5, in which, for each input image (IE), and for each associated optical transfer function, the calculation of the corresponding corrected image (le) comprises: - initial calculation of an estimated image, by deconvolution of the optical transfer function of the input image (IE); - iterative implementation of a processing loop comprising: • calculation of a current simulated image from the current estimated image and the optical transfer function; • determination of a cost function from at least one of the current simulated image and the current estimated image; • if a predetermined stopping criterion is reached, exit from the processing loop, otherwise, update of the estimated image from the determined cost function; - providing, as output, the corrected image (the) as being the current estimated image.
7. Method according to any one of claims 1 to 6, further comprising, for each corrected image (le), a calculation of a sharpness index in at least one part of the corrected image (le).
8. Method according to claim 7, in which, for each pixel of the at least one part of the corrected image (le), the sharpness index is a value taken, at said pixel, by a predetermined variance estimator, a predetermined entropy estimator or a predetermined energy estimator.
9. Method according to claim 7 or 8, further comprising establishing a sharpness map, the sharpness map associating, with each point of the scene (3), the corrected image (le), from among the set of corrected images (le), in which the value of the sharpness index of the pixel representative of said point of the scene (3) is maximum.
10. Method according to claim 9, further comprising a generation, from the set of corrected images (le), of a fused image (IF), the fused image (IF) being such that, for each point of the scene (3) acquired, the pixel representative of said point of the scene (3) in the fused image (IF) has a value equal to the value of the pixel representative of said point of the scene (3) in the corrected image (le) associated with said point of the scene (3) via the sharpness map.
11. Method according to claim 9 or 10, further comprising, in response to an input instruction representative of an area of interest in the scene (3), a calculation of a restored image (IR), the restored image (I) being such that, for each point of the area of interest, the value of the pixel representative of said point in the restored image (IR) is equal to the value of the pixel representative of said point in the corrected image (Ic) associated with said point of the area of interest via the sharpness map.
12. Method according to claim 11, in which the restored image (IR) is such that, for each point outside the area of interest: - the value of the pixel representing said point in the restored image (I) is equal to the value of the pixel representing said point in a corrected image (le) of the set of corrected images (le); and - the sharpness index of the pixel representing said point in the restored image (IR) is lower than a minimum sharpness index among the pixels of the restored image (IR) representing points in the area of interest.
13. A computer program comprising executable instructions which, when executed by a computer, implement the steps of the method according to any one of claims 1 to 12.
14. Image processing device for obtaining a stack of images from a plurality of input images (IE), each input image (IE) depending on a respective raw image (IB), the raw images being representative of the same scene (3), each raw image being acquired by a sensor of a camera module (4) through an optical system of the camera module (4), for a respective focusing distance of the optical system, the image processing device being configured to: - determining at least one optical transfer function between the scene (3) and the sensor, for the focusing distance corresponding to the respective raw image; and - for each determined optical transfer function, calculating a corrected image (le) by deconvolution of the input image (IE) from said optical transfer function, so as to obtain a set of corrected images, the stack of images obtained depending on the set of corrected images.
15. Electronic apparatus (6) comprising a processing device (2) according to claim 14 and a camera module (4) configured to acquire a plurality of raw images, the processing device (2) being connected to the camera module (4) to receive, as input, all or part of the raw images acquired by the camera module (4).