Method and system for correcting fixed-pattern noise in an image
Patent Information
- Application Number
- EP2023817304
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-12-13
- Filing Date
- 2023-11-16
- Publication Date
- 2025-09-10
Smart Images
Figure 1.1
Abstract
Description
Description Title of the invention: METHOD AND SYSTEM FOR CORRECTING FIXED NOISE IN AN IMAGE Technical field
[0001] The present invention relates to a method for correcting fixed noise in at least one image. It also relates to a system for correcting fixed noise in at least one image. State of the prior art
[0002] Although factory calibrated, infrared cameras are known to experience slight deviations in sensor responses. These deviations change slowly over time, from one image shot to the next. But this change is relatively slow compared to the camera's refresh rate. On the image, these deviations cause a spatially non-uniform response for the same light signal, which visually appears as fixed noise from one image to the next, hence the name "Fixed Pattern Noise" (FPN). This noise is typically modeled as having a fixed component and a multiplicative component with respect to the input signal (offsets and gains). State-of-the-art documents on methods for correcting image non-uniformity are known, such as document US8503821.This document proposes to determine the movement of the scene relative to the sensor between two shots of the same scene and to use the different pixel responses for the same location in the scene to determine said correction.
[0003] US9900526 proposes using a visual shutter to calibrate a scene. This visual shutter corresponds to an object that appears almost uniform and covers a large part of the camera.
[0004] US9208542 proposes a method for reducing pixel-level noise in thermal images. This method counts the number of times a pixel is smaller or larger than its neighboring pixels and, accordingly, corrects the pixel in question.
[0005] The methods in question are therefore based either on the comparison of different views of the same object with different batches of pixels, or on the presence of a physical or virtual shutter, or on assumptions of regularity of real images. Methods based on the regularity of real images have the problem that the assumption of regularity is not well verified everywhere in the image. Some pixels cannot therefore be corrected correctly from a single image. Methods based on the physical shutter have the disadvantage of not exactly correcting the noise observed in practice. Methods based on the virtual shutter involve a some complex logic to implement. Finally, methods comparing two views of the same object or scene after a movement, or using several sensors, are quite sensitive to the quality of the matching of the two scenes and the similarity between the images. In addition, the matching algorithm can be expensive in terms of computational resources.
[0006] The aim of the present invention is to solve at least one of the aforementioned drawbacks. Statement of the invention
[0007] This goal is achieved with a fixed noise correction method in at least one image. The method comprises the following steps:
[0008] - acquisition of at least N substantially distinct and regular images by at least one rotating image sensor so that the combination of the N images forms a panorama of a surrounding scene, the fixed noise included in each of the N acquired images being substantially identical,
[0009] - determination of at least one fixed noise correction parameter by a processing module from the at least N acquired images, F at least one correction parameter minimizing a functional,
[0010] - correction of the fixed noise included in each of the N images acquired by the processing module from at least one determined fixed noise correction parameter.
[0011] Many environmental parameters can affect sensor non-uniformity, such as small temperature variations. This non-uniformity therefore varies over time. However, it can be considered to remain almost fixed over several consecutive images taken at the sensor's maximum frame rate. It is therefore advisable to update the correction frequently.
[0012] The present invention relies on several acquired images of different scenes taken with the same image sensor in a short time. This is achieved with a rotating image sensor that can reconstruct a panorama using images taken during the rotation of the image sensor: each image corresponds to a different piece or sector of panoramic scene. An acquisition frequency E of the image sensor is higher than a fixed noise change frequency between the acquisition of two images. The acquisition frequency E is sufficient so that any changes in the fixed noise parameters are very small. The invention then makes it possible to avoid the defects of traditional methods based on regularity, because each fixed noise correction is estimated not from one, but from several acquired images.
[0013] It is also considered that the acquired images are substantially regular over a majority of the image, and that in each pixel one can find one or more very regular images locally around this pixel among the acquired images. We understand by "Regular" means that the intensity gradient in the image under consideration does not undergo abrupt changes over a majority of the image. An image is also said to be "locally regular" if the pixel values in a small neighborhood can be described with a simple mathematical formula, for example the equation of a plane, with few errors. Natural images are generally considered to be, with high probability, regular in a majority of pixels. This corresponds, for example, to flattened areas of the image, or with slight gradients. Areas of high contrast, such as the edges of objects, or heavily textured areas, for example a pile of branches, are not considered regular.
[0014] The non-uniformity correction parameters can be re-estimated frequently enough to compensate for variations in sensor non-uniformity. By "non-uniformity variations" we mean variations in fixed noise. Given the high image acquisition frequency, it is assumed here that the value of the fixed noise in each image is substantially identical. By "substantially identical" we mean that the fixed noise has a negligible change between the first and last of the N acquired images.
[0015] The at least one fixed noise correction parameter determined by the processing module according to the invention minimizes a functional of the form
[0017] In this functional, . corresponds to image i, and corresponds to image i 1 i corrected, that is to say
[0019] where G , O and f are image-sized matrices, and * is the Hadamard product. G and O correspond to a gain and offset matrix, respectively.
[0020] The values of matrices O and G are the parameters that are estimated.
[0021] A variable number of parameters can be determined from the N acquired images. In the case where only O is estimated, the invention gives more robust and faster results. In the case where several parameters are estimated, the signal is better represented and the correction is more efficient in practice. The values of the matrices O and G are the parameters that are estimated in the present case.
[0022] The step of determining at least one fixed noise correction parameter by the processing module may comprise the following step:
[0023] - determination of a set of at least two correction parameters for all N images acquired by the processing module, the set of correction parameters comprising a gain value and an offset value.
[0024] In order to minimize the functional, a set of two parameters is directly determined from the N acquired images. All N images are taken into account to determine the parameter set. In this implementation the functional can be defined by:
[0028] The step of determining at least one fixed noise correction parameter by the processing module can be carried out via a direct implementation.
[0029] Direct implementation may include the following steps:
[0030] - first filtering of the N images,
[0031] - second filtering along the N images.
[0032] Advantageously, the first filtering of the N images can include the use of a high-pass filter.
[0033] A judicious choice of G and h allows a direct implementation, where a succession of filterings makes it possible to directly obtain the minimum of the functional. At least one correction parameter can be determined in this way. In one implementation, the judicious choice of G and h corresponds to
[0037] where ï is the image size matrix containing 1 in each box,
[0038] where F is a high-pass filter on the acquired image, such as subtracting the median of neighboring pixels from the value of each pixel,
[0039] And
[0040] h(G, G) = 0 if G = I otherwise / z(G, O) = + «
[0041] h G, O) = + oo allows us to force the gain to be one in each pixel without penalizing the offset value. We then obtain that the functional is minimized by:
[0043] where the median operator is understood to be applied independently at each position in the matrix.
[0044] A direct implementation of the functional minimization then consists first of filtering the N images according to the high-pass filter F. The high-pass filter corresponds to a spatial filter. Then the results of these filterings are filtered by the median operator along the N images.
[0045] Other similar functionals have direct implementations. By direct implementation, we mean that the result of minimizing the functional is obtained directly without using a functional minimization algorithm.
[0046] To use has the effect of replacing in the direct implementation the median by the average operator. A weighted average is obtained by using Q- operators integrating a different weighting for each image. In the case of the direct implementation using the median operator, we can see the interest of using the norm 1 in Q . Indeed, the application of the median on the result of the high-pass filters has the effect of not taking into account the scenes where, for the pixel considered, and there is strong contrast of the scene, and therefore an aberrant response of the high-pass filter. The majority of scenes will have a non-erroneous value in this pixel, and this is therefore what the median returns.
[0047] The step of determining at least one fixed noise correction parameter by a processing module may also comprise the following step:
[0048] - temporal smoothing of F at least one correction parameter common to each of the N images acquired.
[0049] This temporal smoothing step allows the results of the determined correction parameter(s) to be refined.
[0050] According to yet another aspect of the invention, there is provided a system configured for fixed noise correction of at least one image, the system comprising:
[0051] - at least one rotating image sensor arranged to acquire at least N substantially distinct and regular images, so that the combination of the distinct images forms a panorama of a surrounding scene, the fixed noise included in each of the N acquired images being substantially identical,
[0052] - a processing module arranged and / or programmed to determine at least one fixed noise correction parameter from the N images acquired by the image sensor, F at least one correction parameter minimizing a functional, and to correct the fixed noise in each of the N images acquired from said estimation of F at least one determined correction parameter.
[0053] According to yet another aspect of the invention, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to the invention. Description of figures and embodiments
[0054] Other advantages and particularities of the invention will appear on reading the detailed description of implementations and embodiments which are in no way limiting, and the following appended drawings:
[0055] [Fig.l] is a schematic profile section view of a system according to the invention according to one embodiment.
[0056] [Fig.2] is a flowchart of the method of the invention according to one embodiment.
[0057] Since these embodiments are in no way limiting, it is possible in particular to consider variants of the invention comprising only a selection of characteristics described or illustrated subsequently isolated from the other characteristics described or illustrated (even if this selection is isolated within a sentence comprising these other characteristics), if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection comprises at least one preferably functional characteristic without structural details, and / or with only a part of the structural details if this part alone is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.
[0058] We will first describe, with reference to [Fig.l], a system configured for the correction of fixed noise of at least one image. The system comprises a rotating image sensor 1 arranged to acquire N substantially distinct and regular images, so that the combination of the distinct images forms a panorama of a surrounding scene, the fixed noise included in each of the N acquired images being substantially identical. In another embodiment, the system may comprise several image sensors whether they are rotating or not. The image sensor 1 is rotated in order to acquire several images of the environment by following a panorama. The acquired images are distinct from one another, that is to say that each acquired image images a panorama sector. In other embodiments, there may also be among the acquired images images of the same scene or with a strong overlap with other images.
[0059] An image is defined as a set of pixel values measured during a capture by an image sensor 1. A scene corresponds to a physical reality in front of the image sensor 1 and which is captured by said image sensor 1 which produces an image of it. When the image sensor 1 rotates, it observes another scene. The rotating image sensor 1 corresponds, in a preferred embodiment, to an infrared rotating image sensor 1. The image sensor is composed of a set of sensors each corresponding to a pixel. Indeed, depending on the model, the infrared rotating image sensor 1 acquires multiple images to reconstruct a panorama. A panorama therefore corresponds to a set of scenes, or images, placed end to end to produce a 360-degree observation, preferably of at least 90 degrees. The rotating image sensor 1 rotates at a rotation frequency typically of 2 Hz to 0.5 Hz.The image sensor 1 can rotate clockwise and vice versa. Preferably, the rotating image sensor 1 acquires images according to a minimum acquisition frequency F. F can take values such as 7.5Hz, 45Hz or 64Hz. The invention can be applied to others. systems moving or rotating the system at a frequency high enough that multiple images are acquired according to the embodiment presented.
[0060] The system also comprises a processing module (not shown in the figure) arranged and / or programmed to determine at least one fixed noise correction parameter from the N images acquired by the image sensor, F at least one correction parameter minimizing a functional, and to correct the fixed noise in each of the N images acquired from said estimation of F at least one determined correction parameter.
[0061] A flowchart of the method of the invention according to one embodiment will be described with reference to [Fig.2].
[0062] The method for correcting fixed noise in at least one image comprises three steps (E1 to E3). Step E1 corresponds to an acquisition by the image sensor 1 of N substantially distinct and regular images, such that the combination of the distinct images forms a panorama of a surrounding scene, the fixed noise included in each of the N acquired images being substantially identical. The acquisition of the images is processed in the same way as that presented [Fig.l].
[0063] Step E2 corresponds to the determination of at least one fixed noise correction parameter by a processing module 2 from the N acquired images, F at least one correction parameter minimizing a functional.
[0064] According to a first embodiment, the processing module 2 is configured to directly determine at least one fixed noise correction parameter by minimizing a functional. The processing module 2 is then configured to estimate a set of parameters for all of the N acquired images, the set of parameters comprising a gain value and an offset value.
[0065] The functional returns a number, often defined as positive, which is related to the amount of fixed noise present in the group of N acquired images considered. The objective is to reduce this value to eliminate the fixed noise. The functional varies according to the defined correction parameters (offset, gain, etc.). The value of the functional corresponds to the value of a function related to the amount of noise present in the image in which these corrections are applied. The set of correction parameters determined by the processing module 2 minimizes this functional.
[0066] In practice, the functional is often composed of a data attachment term and a regularization term. The choice of the two terms reflects regularity assumptions made about the scenes that we expect to observe in practice. Here, a “total variation” term is used as the regularization term, and a quadratic penalty, on the norm of the estimated gain (minus one) and offset corrections, as the data attachment. The regularization term, based on the total variation, reflects that the image f after correction 'f = Q*f _|_ ()■ must give an image whose variation ii total will be low. The data attachment term reflects that the intensities of the images f. and 'e must be close, and therefore that the gain must be close to 1 in each pixel J i and the offset close to 0. Thus, correction parameters are applied to the image, here an offset parameter (of order 0) and a gain parameter (of order 1). The parameters that minimize this quantity are sought.
[0067] Here, several images are available (the image index i ranging from 1 to N), a single set of gain and offset parameters is determined for all the images (the fixed noise is assumed to be fixed on this set of acquired images). The total variations are summed for the different images because all the images must have their discontinuities reduced by the correct fixed noise correction. Other aggregation methods could also be used such as the minimum or the median.
[0068] We then obtain the functional:
[0070] Since the fixed noise varies slightly over time, the correction parameters are therefore updated for a new group of images. Each time, the correction parameters are estimated from several images. A "new group" is defined as a batch of N images acquired during a rotation by the image sensor. Once the correction has been estimated for N images, it is assumed that, for the following N images, the fixed noise will have changed very little. We can therefore distribute the determined correction parameters and adapt them.
[0071] In this or other embodiments, in order to minimize the functional and obtain the correction parameters, several techniques may be used, for example:
[0072] - an iterative optimization method,
[0073] - a method with machine learning, where a function is learned to directly produce a good approximation of the solution of the minimization of the functional. This function will take as input the N images and either return the correction parameters, or directly the N corrected images (the estimation of the parameters being then considered implicit).
[0074] According to a second embodiment, the processing module 2 is configured to determine at least one fixed noise correction parameter from the N images acquired via a direct implementation.
[0075] This embodiment consists of taking each of the acquired images individually and applying a processing equivalent to a high-pass filter on each of the N acquired images. More precisely, this high-pass filter consists of taking, for each pixel of the acquired image, the median value of its eight direct neighboring pixels and subtracting it from the value of the pixel in question. The result of the high-pass filter on Each image contains several components, such as the natural difference of the scene content at each pixel relative to the neighbors and the noise composed of a time-dependent component (fixed noise) and a time-independent component (photon and electronic noise). In other embodiments, a number of other alternative filters are possible. For example, for the first determination step, one can use the subtraction of the mean of the neighbors or the subtraction of any other low-pass filter (which is equivalent to applying a high-pass filter). Machine learning can also be used to independently predict on each image the correction at each pixel.
[0076] According to the embodiment described, step E2 also comprises a step of filtering the N processed images to extract at least one correction parameter common to each of the N acquired images. The filtering makes it possible to greatly reduce the contribution of the observed scenes, which implies that the scenes must correspond to varied scenes. This is the case here since, according to the invention, a rotating image sensor is used, each acquired image covering a different viewing angle and with little overlap with the previous image. The result of the processing step with a high-pass filter on each acquired image is therefore filtered to produce a single image, i.e. a single estimate at each pixel. More precisely, a median at each pixel of all the images is produced, recovered and serves as a correction parameter.This second filtering has the effect of strongly reducing in each pixel the contribution of the temporally independent noise and the observed scenes, while the fixed noise is preserved. The common correction parameter therefore corresponds to an estimate of the fixed noise present in the acquired images. In other embodiments, other filters than that presented in this embodiment are conceivable, such as averaging after removal of the elements beyond the extreme quantiles or filtering the value appearing the most.
[0077] This second filtering step is equivalent to seeking consensus between the independent estimates of the correction parameters of each image.
[0078] If the number of images used is small and the variation frequency of the non-uniformity is sufficiently slow, a temporal smoothing of F at least one common correction parameter is used in order to refine the results of F at least one common correction parameter.
[0079] Step E3 of the method corresponds to the correction of the fixed noise included in each of the N images acquired by the processing module from at least one determined fixed noise correction parameter.
[0080] Regardless of the implementation method, the correction is applied in the same way. This involves applying the determined correction parameter(s) to the N images, i.e. multiplying each image by the gain and adding the determined offset. correction can be applied following a factory-predetermined image sensor calibration, for example.
[0081] Typically, at least one of the means of the device according to the invention previously described, preferably each of the means of the device according to the invention previously described, is a technical means.
[0082] Typically, each of the means of the device according to the invention previously described may comprise at least one computer, a central or calculation unit, an analog electronic circuit (preferably dedicated), a digital electronic circuit (preferably dedicated), and / or a microprocessor (preferably dedicated), and / or software means.
[0083] Of course, the invention is not limited to the examples which have just been described and numerous adjustments can be made to these examples without departing from the scope of the invention.
[0084] Of course, the various embodiments, features, forms and variations of the invention may be combined with each other in various combinations, provided that they are not incompatible or mutually exclusive.
Claims
Claims
1. Method for correcting fixed noise in at least one image, characterized in that the method comprises the following steps: - acquisition (El) of at least N substantially distinct and regular images by at least one rotating image sensor (1) so that the combination of the N images forms a panorama of a surrounding scene, the fixed noise included in each of the N acquired images being substantially identical, - determination (E2) of at least one fixed noise correction parameter by a processing module (2) from the at least N acquired images, F at least one correction parameter minimizing a functional, - correction (E3) of the fixed noise included in each of the N images acquired by the processing module (2) from at least one determined fixed noise correction parameter.
2. Method according to claim 1, characterized in that the step of determining at least one fixed noise correction parameter by the processing module (2) comprises the following step: - determination of a set of at least two correction parameters for all N images acquired by the processing module (2), the set of correction parameters comprising a gain value and an offset value.
3. Method according to claim 1, characterized in that the step of determining at least one fixed noise correction parameter by the processing module (2) is carried out via a direct implementation.
4. Method according to claim 3, characterized in that the direct implementation comprises the following steps: - first filtering of the N images, - second filtering along the N images.
5. Method according to claim 4, characterized in that the first filtering of the N images comprises the use of a high-pass filter.
6. Method according to any one of claims 1 to 5, characterized in that the step of determining at least one fixed noise correction parameter by a processing module (2) also comprises the following step: - temporal smoothing of F at least one correction parameter common to each of the N acquired images.
7. A system configured for fixed noise correction of at least one image, the system comprising: - at least one rotating image sensor (1) arranged to acquire at least N substantially distinct and regular images so that the combination of the distinct images forms a panorama of a surrounding scene, the fixed noise included in each of the N acquired images being substantially identical, - a processing module (2) arranged and / or programmed to determine at least one fixed noise correction parameter from the N images acquired by the image sensor, the at least one correction parameter minimizing a functional, and to correct the fixed noise in each of the N images acquired from said estimation of the at least one determined correction parameter.
8. A computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to any one of claims 1 to 6.