Method for obtaining an image stack and related computer program product, processing device and electronic device

Abstract

The invention relates to a method for obtaining an image stack from a plurality of input images (IE), each input image depending on a respective original image (IB) representing the same scene (3) and each original image being acquired by a sensor (14) of a camera module (4) through an optical system (12) for a respective focus distance, the method being implemented by a computer, and for each input image, the method comprises the steps of: determining (26) at least one optical transfer function between the scene and the sensor for a focus distance corresponding to the respective original image; and calculating (28), for each determined optical transfer function, a corrected image (IC) by deconvolution of the input image using the optical transfer function to obtain a set of corrected images, the obtained image stack depending on the set of corrected images.

Description

Technical Field

[0001] This invention relates to a method for obtaining an image stack.

[0002] The present invention also relates to a computer program for implementing the method, a processing device, and an electronic device including the processing device.

[0003] This invention relates to the field of image processing. Background Technology

[0004] When acquiring an image representing a scene using a camera module, it is well known that the camera module is set so that its focus distance corresponds to the distance of the object of interest in the scene. The result of setting the camera module in this way is that elements in the scene that are at a distance equal to the focus distance from the camera module appear sharper than other elements in the scene.

[0005] In other words, for each image, elements in the scene that are at a distance from the camera module that differs from the focus distance typically include optical aberrations and / or out-of-focus blur.

[0006] As is well known, in order to clearly present each area in a scene, it is known to change the focus distance and acquire an image for each focus distance: this produces an image stack called "focus bracketing" image stack, in which the sharp area differs between images.

[0007] However, this method is not satisfactory.

[0008] In practice, this method requires acquiring a large number of images at multiple focus distances corresponding to the broad coverage of the observed scene, based on fairly dense discretization steps, in order to obtain a satisfactory sharpness of the image stack.

[0009] This necessitates storing a large image stack. Furthermore, the image acquisition process lengthens proportionally with the number of images acquired. Therefore, there is a risk that the scene in the image stack includes movement, which could cause errors in rendering the stitched images when connecting objects moving between shots.

[0010] One object of the present invention is to overcome at least one of the disadvantages of the prior art.

[0011] Another objective of this invention is to propose a method for obtaining an image stack that requires fewer images than known methods to achieve satisfactory sharpness of the entire image stack without requiring a large amount of storage space. Summary of the Invention

[0012] Therefore, the present invention relates to a method for obtaining an image stack from multiple input images.

[0013] Each input image depends on a corresponding original image, which represents the same scene.

[0014] Each raw image is acquired by the camera module's sensor through the camera module's optical system at the corresponding focusing distance of the optical system.

[0015] This method is implemented by a computer, and for each input image, the method includes the following steps:

[0016] - For a focus distance corresponding to the corresponding original image, determine at least one optical transfer function between the scene and the sensor; and

[0017] - For each determined optical transfer function, a corrected image is computed by deconvolving the input image using the optical transfer function to obtain a set of corrected images, the resulting image stack depending on the set of corrected images.

[0018] Indeed, the result of the deconvolution step is a significant reduction in the number of images required to achieve satisfactory sharpness in the image stack. Therefore, any time delay between acquired images is reduced when moving elements are included in the acquired images (or even when the camera module itself is moving). Consequently, the discretization step between focal planes can be chosen to be larger than conventionally known methods. Therefore, the memory space occupied by the image stack is reduced.

[0019] This method also allows for storing fewer images in memory while improving overall rendering quality.

[0020] Advantageously, the method according to the invention has one or more of the following features, which are employed individually or in any technically feasible combination:

[0021] For each input image, at least one optical transfer function is the optical transfer function between the camera module and the sensor of the camera module relative to a plane located at a distance equal to the corresponding focus distance.

[0022] For each input image, each optical transfer function is an optical transfer function between the following two:

[0023] - On the one hand, the set of corresponding points in the scene that are located at the same distance from the camera module within a predetermined distance margin; and

[0024] - On the other hand, the camera module's sensor;

[0025] For each input image, each optical transfer function is an optical transfer function between the following two:

[0026] - On the one hand, a set of corresponding points in the scene that have the same viewpoint relative to the camera module within a predetermined angular margin, relative to the optical axis of the camera module's optical system; and

[0027] - On the other hand, the camera module's sensor;

[0028] The method also includes establishing a transfer function mapping for each input image, which associates the corresponding optical transfer function realized during the step of calculating the corrected image with each point in the scene represented on the input image;

[0029] For each input image and for each associated optical transfer function, the corresponding corrected image is calculated as follows:

[0030] - The estimated image is initially calculated by deconvolving the optical transfer function with the input image;

[0031] - Implement the processing loop iteratively, which includes:

[0032] • Calculate the current simulated image based on the current estimated image and optical transfer function;

[0033] • Determine the cost function based on at least one of the current simulated image and the current estimated image;

[0034] • If the predetermined stopping criterion is met, exit the processing loop; otherwise, update the estimated image based on the determined cost function.

[0035] - Output the corrected image as the current estimated image;

[0036] The method further includes: for each corrected image, calculating a sharpness index in at least a portion of the corrected image;

[0037] For each pixel of at least a portion of the corrected image, the sharpness index is a value obtained for that pixel by a predetermined variance estimator, a predetermined entropy estimator, or a predetermined energy estimator;

[0038] The method also includes establishing a sharpness map that associates each point in the scene with the corrected image that has the largest sharpness index value among all corrected images representing the point in the scene;

[0039] The method further includes generating a merged image based on a set of corrected images, the merged image being such that: for each point in the acquired scene, the value of the pixel representing the point in the scene in the merged image is equal to the value of the pixel representing the point in the scene in the corrected image associated with the point in the scene via a sharpness mapping;

[0040] The method further includes: in response to an input setpoint representing a region of interest in a scene, calculating a rendered image such that: for each point in the region of interest, the value of the pixel representing the point in the rendered image is equal to the value of the pixel representing the point in a corrected image associated with the point in the region of interest via a sharpness mapping;

[0041] This rendered image makes it possible for each point outside the region of interest:

[0042] - The value of the pixel representing the point in the rendered image is equal to the value of the pixel representing the point in the set of corrected images; and

[0043] - The sharpness index of the pixel representing the point in the rendered image is less than the minimum sharpness index among the pixels representing the point in the region of interest in the rendered image.

[0044] According to another aspect of the present invention, a computer program is provided, the program comprising executable instructions that, when executed by a computer, implement the steps of the method as defined above.

[0045] The computer program can be in any computer language, such as machine language, C, C++, JAVA, Python, etc.

[0046] According to another aspect of the invention, an image processing apparatus for obtaining an image stack from multiple input images is proposed.

[0047] Each input image depends on a corresponding original image, which represents the same scene.

[0048] Each raw image is acquired by the camera module's sensor through the camera module's optical system at the corresponding focusing distance of the optical system.

[0049] The image processing device is configured as follows:

[0050] - For a focus distance corresponding to the corresponding original image, determine at least one optical transfer function between the scene and the sensor; and

[0051] - For each determined optical transfer function, a corrected image is computed by deconvolving the input image based on the optical transfer function to obtain a set of corrected images, the resulting image stack depending on the set of corrected images.

[0052] The device according to the invention can be any type of apparatus programmed, for example, to implement the method according to the invention by running a computer program according to the invention, such as a server, computer, tablet computer, calculator, processor, or computer chip.

[0053] According to another aspect of the invention, an electronic device is provided, comprising a processing device as defined above and a camera module configured to acquire a plurality of raw images, the processing device being connected to the camera module to receive all or part of the raw images acquired by the camera module as input.

[0054] Preferably, the electronic device is a smart mobile phone (also known as a smartphone or multi-functional mobile phone).

[0055] In this context, the invention also relates to using a processing device and / or a method according to the invention within a smartphone to obtain an image stack from multiple raw images acquired by the smartphone's camera module.

[0056] As a variant, the electronic device is a touch-sensitive tablet computer.

[0057] In this context, the invention also relates to using a processing device and / or a method according to the invention within a touch-sensitive tablet computer to obtain an image stack from a plurality of raw images acquired by the camera module of the touch-sensitive tablet computer.

[0058] As a variant, this electronic device is a computer.

[0059] In this context, the invention also relates to using a processing device and / or a method according to the invention within a computer to obtain an image stack from multiple raw images acquired by a camera module of the computer.

[0060] As a variant, this electronic device is a television set.

[0061] In this context, the invention also relates to using the processing apparatus and / or method according to the invention within a television set to obtain an image stack from a plurality of raw images acquired by the camera module of the television set.

[0062] As a variant, the electronic device is a virtual reality headset.

[0063] In this context, the invention also relates to using a processing device and / or a method according to the invention within a virtual reality headset to obtain an image stack from a plurality of raw images acquired by a camera module of the virtual reality headset.

[0064] As a variation, the electronic device is a medical imaging device, such as an endoscope.

[0065] In this context, the invention also relates to using a processing device according to the invention and / or a method according to the invention within a medical imaging apparatus to obtain an image stack from multiple raw images acquired by a camera module of the medical imaging apparatus.

[0066] As a variant, the electronic device is an autonomous or non-autonomous vehicle, such as a land vehicle (e.g., a car), an aircraft (e.g., a drone, an airplane, a helicopter, etc.), or even a marine vehicle (e.g., a ship, a submarine, etc.).

[0067] In this context, the invention also relates to using a processing device and / or a method according to the invention within a vehicle to obtain an image stack from multiple raw images acquired by a camera module of the vehicle. Attached Figure Description

[0068] The invention will be better understood by reading the following description, which is given by way of non-limiting example only and with reference to the accompanying drawings: wherein:

[0069] Figure 1 This is a schematic diagram of an electronic device including an image processing apparatus according to the present invention;

[0070] Figure 2 This is a flowchart of a method for obtaining an image stack according to the present invention, the method being... Figure 1 Image processing devices are used to achieve this;

[0071] Figure 3 It is to achieve Figure 2 The method is a schematic diagram of the first partition of the image; and

[0072] Figure 4 It is to achieve Figure 2 The method is a schematic diagram of the second partition of the image.

[0073] It is clearly understood that the embodiments described below are by no means limiting. Specifically, variations of the invention may be conceived to include only selections of the features disclosed below, independent of the other disclosed features, if including only the selection of features disclosed below is sufficient to provide a technical advantage or to differentiate the invention from the prior art. Such selections include at least one preferred functional feature that has no structural details, or if it has only partial structural details (if such partial structural details alone are sufficient to provide a technical advantage or to differentiate the invention from the prior art).

[0074] Specifically, all the described variations and embodiments can be combined with each other, as long as there are no technical obstacles preventing such combination.

[0075] In the remainder of the drawings and description, the same reference numerals are used for common features in many of the drawings. Detailed Implementation

[0076] exist Figure 1 The image processing apparatus 2 (or hereinafter referred to as "processing apparatus") according to the present invention is shown.

[0077] Processing device 2 is configured to process multiple raw images I representing scene 3 and acquired by camera module 4. B To output the image stack. Specifically, for each original image I... B All of these data are obtained based on the corresponding focus distance of camera module 4.

[0078] The images in the image stack provided by processing device 2 have an advantage over the original images I acquired by camera module 4. B Better clarity and / or better resolution and / or less noise.

[0079] For example, both the processing device 2 and the camera module 4 are integrated into a single electronic device 6. In this case, the processing device 2 is connected to the camera module 4 to receive multiple raw images I acquired by the camera module 4. B Each of the elements is used as input.

[0080] Electronic devices include, for example, smartphones, touch-sensitive tablets, or even digital cameras. Alternatively, electronic device 6 may be video surveillance equipment, vehicles, drones, satellites, microscopes, or even medical imaging equipment (e.g., computed tomography imaging equipment).

[0081] Preferably, the electronic device 6 further includes a human-machine interface 8 to allow the user to input input setpoints representing regions of interest in scene 3. Alternatively or additionally, the human-machine interface 8 can display a rendered image. Such a rendered image will be described in more detail below.

[0082] Optionally, the electronic device 6 also includes a distance measurement module 10, which is configured to transmit distance information representing the distance between all or some objects in scene 3 and the sensors and / or optical systems (described below) in camera module 4. The distance measurement module 10 is, in particular, a rangefinder, such as an optical rangefinder.

[0083] Camera module

[0084] The camera module 4 includes at least one component, which includes an optical system 12 associated with the sensor 14.

[0085] As previously stated, camera module 4 is adapted to acquire multiple original images I representing the same scene 3 under different focusing distances and preferably from the same location. B (For example, by moving the optical system 12 relative to the sensor 14, or even by modifying the shape of the lens (e.g., a liquid lens) of the optical system 12).

[0086] Camera module 4 is associated with at least one optical transfer function between scene 3 and sensor 14. This optical transfer function establishes a relationship between the characteristics of light from scene 3 (particularly spatial and spectral characteristics) and the characteristics of light arriving at sensor 14, primarily due to the influence of optical system 12.

[0087] For example, each optical transfer function of the camera module 4 can be described by the point spread function of the camera module 4.

[0088] Each optical transfer function of camera module 4 depends specifically on the focusing distance of camera module 4. Each optical transfer function may also depend on at least one of the following:

[0089] - The distance between sensor 14 and objects in scene 3 imaged by sensor 14;

[0090] - The distance between the optical system 12 (especially the optical center of the optical system 12) and the sensor 14;

[0091] - The distance between the optical system 12 (especially the optical center of the optical system 12) and the object in scene 3 imaged by the sensor 14;

[0092] - The state of the optical system 12, such as zoom, focal length and / or numerical aperture settings;

[0093] - Pixels considered in the photosite array of sensor 14; and

[0094] - One or more angles between sensor 14 and optical system 12.

[0095] For example, the distance measurement module 10 is used to measure the distance between the sensor 14 and each imaged object and / or to measure the distance between the optical system 12 and each object in scene 3.

[0096] Processing equipment

[0097] The processing device 2 can be in the form of hardware, such as a computer, server, processor, electronic chip, etc. Alternatively or additionally, the processing device 2 can be in the form of software, such as a computer program or application, for example, an application for user devices such as tablet computers or smartphones.

[0098] Processing device 2 is configured to implement processing method 20. Figure 2 ), based on multiple original images I B Output image stack.

[0099] like Figure 2 As shown, the processing method 20 includes a receiving step 22, an optional preprocessing step 24, an optical transfer function determination step 26 (hereinafter referred to as the "determination step"), and a corrected image calculation step 28 (hereinafter referred to as the "calculation step").

[0100] Preferably, the processing method 20 further includes a rendering step 30.

[0101] Receiving step 22

[0102] Processing device 2 is configured to receive multiple raw images I from camera module 4 during receiving step 22. B .

[0103] As mentioned earlier, each received raw image I B This represents scenario 3, and is obtained for the corresponding focus distance of camera module 4.

[0104] As a variant, multiple original images I B It has already been stored in the memory of processing device 2.

[0105] According to another variant, multiple original images I B It has been acquired by another electronic device and then transmitted to processing device 2 for processing.

[0106] Preprocessing step 24

[0107] Optionally, the processing device 2 is configured to process each raw image I during the preprocessing step 24. B Apply at least one pre-defined preprocessing step.

[0108] For example, this preprocessing is dematrixing (also known as demosaicing), and pixel gain correction may also be performed afterward.

[0109] In this case, for each original image I B The corresponding input image I is formed by the preprocessing steps that produce the corresponding images. E .

[0110] Alternatively, processing device 2 is not configured to perform the preprocessing step. In this case, multiple raw images I received during receiving step 22... B Each original image I B Forming input image IE .

[0111] Determine step 26

[0112] Processing device 2 is also configured to: during determination step 26, for each input image I E Determine at least one corresponding optical transfer function between scene 3 and sensor 14.

[0113] More specifically, for each input image I E Processing device 2 is configured to: for the corresponding original image I B The corresponding focusing distance determines at least one optical transfer function between scene 3 and sensor 14.

[0114] • Single optical transfer function for each input image

[0115] According to the first variant, for each input image I E The processing device 2 is configured to determine a single corresponding optical transfer function.

[0116] Advantageously, for any given input image I E The optical transfer function determined by processing device 2 is an optical transfer function between the following two:

[0117] - On the one hand, relative to camera module 4, it is located at a position equal to the corresponding original image I. B The plane at the corresponding focal distance; and

[0118] - On the other hand, the sensor of camera module 4.

[0119] In this context, "distance relative to the camera module" is understood to refer to the distance relative to the (optical) center of the optical system 12. However, since the distance between the optical system 12 and the sensor 14 is typically only on the order of a few millimeters, the distance measured from the optical center of the optical system 12 is considered equivalent to the distance measured from the sensor 14.

[0120] Such a feature is advantageous because, in this variant, for the input image I under consideration... E The aforementioned optical transfer function best represents any aberrations introduced by camera module 4 when acquiring the corresponding original image. Therefore, this variant represents the method for enhancing the input image I by deconvolution. E The optimal optical transfer function (with regard to the element located at a distance close to the focusing distance relative to camera module 4).

[0121] For example, for each input image I EThe processing device 2 is configured to determine the corresponding optical transfer function using calibration data derived from a previous calibration performed on the camera module 4.

[0122] • Case with multiple optical transfer functions for each input image

[0123] Or, for each input image I E The processing device 2 is advantageously configured to determine a plurality of corresponding optical transfer functions.

[0124] This method is advantageous because the optical transfer function is not necessarily spatially uniform.

[0125] In this case, for each input image I E The processing device 2 is preferably configured to determine each corresponding optical transfer function as an optical transfer function between the following two:

[0126] - On the one hand, the set of corresponding points in scene 3 that are at the same distance from camera module 4 within a predetermined distance margin; and

[0127] - On the other hand, the sensor 14 of the camera module 4.

[0128] This corresponds to Figure 3 The example shown has each region 32 (here, a rectangular region) corresponding to the input image I. E Part of the input image I E This part represents the set of points located at approximately the same distance from camera module 4.

[0129] For example, information indicating different sets of points located at substantially the same distance from camera module 4 is received from distance measurement module 10.

[0130] Naturally, the input image I represents the set of points located at the same distance from camera module 4. E The part is not necessarily defined as a rectangle, but may take on various shapes depending on the objects present in Scenario 3.

[0131] As mentioned earlier, "distance relative to the camera module" is understood to mean the distance relative to the (optical) center of the optical system 12, or even the distance relative to the sensor 14.

[0132] This feature is advantageous because within each region 32, the optical transfer function is equivalent to a uniform function over the region. This enables frequency-domain deconvolution, which requires relatively little computation time.

[0133] As a variation, for each input image I EThe processing device 2 is preferably configured to determine each optical transfer function as an optical transfer function between the following two:

[0134] - On the one hand, in scene 3, the optical axis Δ of the optical system 12 relative to camera module 4 has a set of corresponding points with the same viewing angle θ relative to camera module 4 within a predetermined angular margin; and

[0135] - On the other hand, the sensor of camera module 4.

[0136] This corresponds to Figure 4 As shown, each of the circular region 34, the annular region 36, and the complementary region 38 corresponds to the input image I. E The representation is a portion of a set of points that are substantially located on the same conical surface, the vertex of which is the camera module 4 (in particular the optical center of the optical system 12, or alternatively the sensor 14).

[0137] "View relative to the camera module" is understood to refer to the angle between the optical axis Δ and any straight line passing through the (optical) center of the optical system 12 or even through the sensor 14.

[0138] This feature is advantageous because the physical properties of the optical system 12 are generally rotationally symmetric about the optical axis, making the optical transfer function comparable to a uniform function on each of regions 34, 36 and / or 38 (especially in the case that all points in the scene of one of the angular regions are located at the same distance from the camera module 4).

[0139] Advantageously, the processing device 2 is also configured to process each input image I E Establish a transfer function mapping. More specifically, for each input image I... E The corresponding transfer function mapping maps the determined optical transfer function to the input image I. E Each point in the scene shown above is related.

[0140] Advantageously, the processing device 2 is also configured to process each input image I E Establish a distance map that represents the distance of each point in the scene relative to camera module 4.

[0141] Calculation step 28

[0142] Processing device 2 is also configured to: during calculation step 28, for each input image I E And for each corresponding optical transfer function determined during step 26, calculate the corrected image I. C More specifically, for each input image I EAnd for each associated optical transfer function, the processing device 2 is configured to process the input image I using the optical transfer function. E Perform deconvolution to calculate the corresponding corrected image I C .

[0143] This produced the corrected image I C A set of.

[0144] Preferably, for each input image I E And for each associated optical transfer function, processing device 2 is configured to calculate the corresponding corrected image I by implementing an algorithm. C The algorithm includes:

[0145] - By comparing the optical transfer function with the input image I E Deconvolution is used to initially calculate the estimated image;

[0146] - Implement a processing loop iteratively, which includes:

[0147] • Calculate the current simulated image based on the current estimated image and optical transfer function;

[0148] • Determine the cost function based on at least one of the current simulated image and the current estimated image;

[0149] • If the predetermined stopping criterion is met, exit the processing loop; otherwise, update the estimated image based on the determined cost function.

[0150] - Correct image I C The output image is the current estimated image.

[0151] As a variation, for each input image I E And for each associated optical transfer function, processing device 2 is configured to calculate the corresponding corrected image I by deconvolution using a combination of correction functions derived from the optical transfer function (e.g., Wiener filtering). C .

[0152] For example, for each input image I E When processing device 2 has determined multiple corresponding optical transfer functions, processing device 2 is preferably configured to process input image I. E For each region, deconvolution is performed using the corresponding optical transfer function.

[0153] Advantageously, the processing device 2 is also configured to: for each corrected image I C Calculate the corrected image I C The sharpness index of at least a portion of the image, and preferably the corrected image I is calculated. CThe sharpness index in each pixel.

[0154] For example, for the corrected image I C For at least one pixel, the processing device 2 is configured to calculate the sharpness index as a value obtained for said pixel by a predetermined variance estimator, a predetermined entropy estimator, or even a predetermined energy estimator.

[0155] Furthermore, the processing device 2 is advantageously configured to establish a resolution map.

[0156] More specifically, this sharpness mapping maps each point in scene 3 to all the calculated corrected images I. C Among them, the corrected image I represents the pixel with the largest sharpness index value of the point in scene 3. C Related. In other words, the sharpness mapping will correlate with the corresponding corrected image I. C Associated with each point in Scene 3, this image is the corrected image I in which the pixel representing that point in Scene 3 has the largest sharpness index value. C (From the calculated corrected image I) C (a set).

[0157] Alternatively or otherwise, the sharpness mapping associates each point in scene 3 with a pixel representing that point in scene 3, wherein in all corrected images I C Among them, this pixel has the highest sharpness index value. In the following text, this type of pixel is referred to as the "optimal pixel".

[0158] Alternatively, the sharpness map can be represented by a list of shapes that define regions of equal sharpness. In this case, each item in the list corresponds to the same corrected image I. C Indexing. This variation has the advantage of reducing memory usage. This variation is generally advantageous if the number of regions with different resolutions is reduced, especially since it requires more computation time than the methods described above.

[0159] Rendering step 30

[0160] Advantageously, the processing device 2 is also configured to: during the optional rendering step 30, based on the corrected image I C The set generates a merged image I F Specifically, processing device 2 is configured to generate merged image I. F This allows for merging image I for each point in scene 3. F The value of the pixel representing the point in scene 3 is equal to the pixel representing the point in scene 3 with the largest sharpness index value in the set of corrected images.

[0161] In other words, merge image IF This allows for merging image I for each point in scene 3. F The value of the pixel representing the point in scene 3 is equal to the corrected image I associated with the point in scene 3 through a sharpness mapping. C The value of the pixel at the point in scene 3 is represented in the image.

[0162] Therefore, for each point in scene 3, merge image I F The pixel in the image represents the corresponding optimal pixel for the point in scenario 3.

[0163] Preferably, the processing device 2 is further configured to merge the images I F Transmitted to the human-machine interface 8 so that it can be displayed.

[0164] More preferably, the processing device 2 is configured to receive input setpoints representing regions of interest in scene 3 from the human-machine interface 8. For example, such setpoints represent the user selecting the original image I they are interested in via the human-machine interface 8. B (or merge image I) F The area in ).

[0165] In this case, processing device 2 is configured to calculate and render image I. R This allows for rendering image I for each point in the region of interest. R The value of the pixel representing the point is equal to the corrected image I associated with the point in the region of interest via a sharpness map. C The value of the pixel at the point is represented in the image.

[0166] In other words, in rendering image I R In this context, each point in the region of interest is represented by its corresponding optimal pixel.

[0167] This is advantageous because rendering image I R Region of interest in the original image I B It offers improved clarity compared to other products.

[0168] Preferably, the processing device 2 is further configured to render image I R The data is transmitted to the human-machine interface 8 so that it can be displayed.

[0169] Advantageously, the processing device 2 is also configured to enable the rendering of image I R In the rendering image, for each point outside the region of interest, the sharpness index of the pixel representing that point is less than the minimum sharpness index of the pixels representing points within the region of interest.

[0170] This feature is advantageous because it makes rendering image I... RElements outside the region of interest have lower resolution compared to elements within the region of interest. This allows users to focus on elements within the region of interest.

[0171] Furthermore, in rendering image I R In the image, for each point outside the region of interest, the pixel value representing that point is preferably equal to the corrected image I. C Corrected image I in the set C The value of the pixel at the point is represented in the image.

[0172] As a variation, in rendering image I R In the image, for each point outside the region of interest, the pixel value representing that point is equal to the corrected image I to which a low-pass spatial filter has been applied. C Original Image I B Or input image I E The value of the pixel at the point is represented in the image.

[0173] This is advantageous because processing an image with a low-pass filter would reduce the sharpness of that image.

[0174] operate

[0175] Now refer to Figure 2 Describe the operation of processing device 2.

[0176] During receiving step 22, processing device 2 receives each raw image I previously acquired by camera module 4. B .

[0177] Then, during the optional preprocessing step 24, the processing device 2 processes each raw image I B Apply at least one predetermined preprocessing step to obtain multiple input images I E .

[0178] Then, during step 26, for each input image I E The processing device 2 determines at least one corresponding optical transfer function between the scene 3 and the sensor 14.

[0179] Then, during calculation step 28, for each input image I E And for each corresponding optical transfer function, processing device 2 calculates the corresponding corrected image I. C This produces the corrected image I. C A set of.

[0180] Preferably, for each corrected image I C Processing device 2 also calculates and corrects image I C At least a portion of the clarity index.

[0181] Advantageously, the processing device 2 also processes the corrected image I C The set establishes a clarity mapping.

[0182] Then, during optional rendering step 30, processing device 2, based on the corrected image I... C The set generates a merged image I F .

[0183] Preferably, the processing device 2 further calculates the rendered image I based on the input setpoints representing the region of interest in scene 3. R .

[0184] Of course, the present invention is not limited to the examples disclosed above.

Claims

1. A method for extracting data from multiple input images (I E Methods for obtaining image stacks, Each input image (I E ) depends on the corresponding original image (I B ), original image (I B ) represents the same scene (3), Each original image (I B The sensor (14) of the camera module (4) acquires the corresponding focus distance of the optical system (12) of the camera module (4) through the optical system (12). The method is implemented by a computer, and for each input image (I E The method includes the following steps: - For the corresponding original image (I) B (26) Determine at least one optical transfer function between the scene (3) and the sensor (14) corresponding to the focusing distance; as well as - For each determined optical transfer function, the input image (I) is processed based on the optical transfer function. E (28) Perform deconvolution to calculate the corrected image (I) C ), to obtain the corrected image (I C The resulting image stack depends on the set of corrected images (I). C A set of ).

2. The method according to claim 1, wherein, For each input image (I) E The at least one optical transfer function is the optical transfer function between the sensor of the camera (4) and the plane located at a distance equal to the corresponding focus distance relative to the camera module (4).

3. The method according to claim 1, wherein, For each input image (I) E Each optical transfer function is an optical transfer function between the following two: - On the one hand, the set of corresponding points in the scene (3) located at the same distance from the camera module (4) within a predetermined distance margin; and - On the other hand, the sensor of the camera module (4).

4. The method according to claim 1 or 2, wherein, For each input image (I) E Each optical transfer function is an optical transfer function between the following two: - On the one hand, in the scene (3), there is a set of corresponding points in the optical system relative to the camera module (4) that have the same viewing angle relative to the camera module within a predetermined angular margin; and - On the other hand, the sensor of the camera module (4).

5. The method according to any one of claims 1 to 4, the method further comprising providing each input image (I E Establish a transfer function mapping, which will be used in the calculation of the corrected image (I). C The corresponding optical transfer function implemented during the steps of ) and the input image (I) E Each point in the scenario (3) represented above is associated with it.

6. The method according to any one of claims 1 to 5, wherein, For each input image (I) E And for each corresponding optical transfer function, calculate the corresponding corrected image (I). C )include: - By comparing the optical transfer function with the input image (I) E Deconvolution is used to initially calculate the estimated image; - Implement a processing loop iteratively, the processing loop comprising: • Calculate the current simulated image based on the current estimated image and the optical transfer function; • Determine the cost function based on at least one of the current simulated image and the current estimated image; • If the predetermined stopping criterion is met, the processing loop is exited; otherwise, the estimated image is updated based on the determined cost function. - The corrected image (I) C The current estimated image is output as the current image.

7. The method according to any one of claims 1 to 6, further comprising: For each corrected image (I) C ), calculate the sharpness index in at least a portion of the corrected image (IC), and calculate the sharpness index.

8. The method according to claim 7, wherein, For the corrected image (I) C For each pixel of at least a portion of the resolution index, the resolution index is a value obtained at the pixel by a predetermined variance estimator, a predetermined entropy estimator, or a predetermined energy estimator.

9. The method according to claim 7 or 8, further comprising establishing a sharpness map, the sharpness map linking each point in the scene (3) with all corrected images (I C The image associated with the pixel with the highest sharpness index value in the scene (3) is represented by the corrected image.

10. The method of claim 9, further comprising basing the correction image (I) on the correction image (I) C The set of images generates a merged image (I). F The merged image (I) F ) such that: for each point in the acquired scene (3), the merged image (I F The value of the pixel in the scene (3) is equal to the corrected image (I) associated with the point in the scene (3) through the sharpness mapping. C The value of the pixel of the point in the scene (3) is represented in ).

11. The method according to claim 9 or 10, further comprising: In response to an input instruction representing a region of interest in the scene (3), a rendered image (I) is calculated. R ), The rendered image (I) R ) such that: for each point in the region of interest, the rendered image (I R The pixel value representing the point in the image is equal to the corrected image (I) associated with the point in the region of interest through the sharpness map. C The value of the pixel at the point is represented in ().

12. The method according to claim 11, wherein, The rendered image (I) R This ensures that for each point outside the region of interest: - The rendered image (I R The value of the pixel representing the point in the image is equal to the value of the corrected image (I). C The corrected images in the set of ) C The value of the pixel at the point is represented in (). as well as - The rendered image (I R The sharpness index of the pixel representing the point is less than that of the rendered image (I). R The minimum sharpness index is represented by ) among the pixels of the points in the region of interest.

13. A computer program comprising executable instructions that, when executed by a computer, implement the steps of the method according to any one of claims 1 to 12.

14. A method for processing multiple input images (I E Image processing devices that obtain image stacks Each input image (I E ) depends on the corresponding original image (I B The original image represents the same scene (3). Each original image is acquired by the sensor of the camera module (4) through the optical system of the camera module (4) at the corresponding focus distance of the optical system. The image processing device is configured to: - For the focus distance corresponding to the corresponding original image, determine at least one optical transfer function between the scene (3) and the sensor; and - For each determined optical transfer function, the input image (I) is processed based on the optical transfer function. E Deconvolution is performed to calculate the corrected image (I). C ( ), to obtain a set of corrected images, the resulting image stack depending on the set of corrected images.

15. An electronic device (6) comprising a processing device (2) according to claim 14 and a camera module (4) configured to acquire a plurality of raw images, the processing device (2) being connected to the camera module (4) to receive all or part of the raw images acquired by the camera module (4) as input.

Images