DETECTION AND CORRECTION OF DEFECTIVE PIXELS
Patent Information
- Application Number
- FR2022010277
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-10-07
- Publication Date
- 2025-02-21
- Estimated Expiration
- 2042-10-07
Smart Images
Figure 00000013_0000 
Figure 00000014_0000 
Figure 00000015_0000
Abstract
Description
Title of the invention: DETECTION AND CORRECTION OF DEFECTIVE PIXELS Technical field
[0001] The present invention relates to imaging systems using a matrix of photo-detectors, in particular in the infrared spectral range, for example mid-infrared or far-infrared.
[0002] These imaging systems integrate infrared cameras to detect thermal radiation from an object or scene and convert it into images representative of the thermal variations of the object or scene. These may be cooled or uncooled infrared cameras. State of the prior art
[0003] Such an imaging system delivers an image in which each pixel corresponds to a photodetector of the photodetector matrix. It is known that a photodetector matrix comprises a certain number of photodetectors whose response does not follow a predictable law as a function of the received light intensity. The pixels that they deliver do not directly restore the value of the received light intensity. These pixels are called defective.
[0004] In order to obtain a consistent processed image for all photodetectors of a photodetector array, defective pixels must be detected and corrected by image processing algorithms.
[0005] The challenge of managing defective pixels is to predict for each pixel its value based on the values of its neighborhood in order to detect and correct these defective pixels.
[0006] It is possible to perform factory detection and correction of defective pixels. Factory detection consists, for example, of detecting pixels that do not follow an affine distribution as a function of temperature by imaging a black body at several temperatures. The black body ensures an equal input flux on all pixels. Pixels whose response deviates sufficiently from the affine model are defined as defective and are recorded in a table of defective pixels. These pixels are then replaced by a combination of their healthy neighborhood.
[0007] However, this technique only allows correction of pixels detected during factory calibration. It is therefore not suitable for cases where pixels can randomly switch from a healthy state to a defective state and vice versa, as happens with certain cooled infrared sensors, for example based on a technology including semiconductor materials made of mercury-cadmium telluride (MCT).
[0008] Detection and correction of defective pixels can also be performed on stage: detection of a defective pixel is based on the difference between the value of the pixel in question and a prediction of the value of the pixel based on its neighborhood.
[0009] Existing methods (for example in WO201068372 and US20150172576) predict the value of the pixel in question by averaging neighboring pixels in space.
[0010] On-scene detection methods are sensitive to false detections because current methods operate on isotropic filters and do not take into account pixel temporality.
[0011] Highly contrasting elements of the scene are likely to be detected as defective pixels and removed from the processed image.
[0012] Other more complex methods are based on dense motion estimations (for example in FR3028376) or use iterative neighborhood search methods that are expensive in computational resources and therefore poorly suited to embedded systems. In addition, dense motion estimations are sensitive to noise, which can generate numerous estimation errors. Existing methods do not take into account local textures, which generates blurring effects on contrasting objects. The method proposed in FR3038194 is based on the iterative construction of the neighborhood, which is not suitable for implementation by FPGA circuits that process the pixels in the stream. Statement of the invention
[0013] The invention aims to solve the problems of the prior art by providing a method for detecting and correcting defective pixels on scene in real time in an image stream, which uses both the global movement of the textures on the images of the image stream and the local orientation of the textures of the images considered individually, to improve the performance of the detection and provide a light solution in FPGA resources.
[0014] The global movement corresponds to the movement of the line of sight. The global movement induces a temporal variation of the information perceived by all the photo-detectors of the photo-detector matrix, each corresponding to a pixel. The global movement makes it possible to differentiate the points of high contrast of the scene, whose positions vary temporally on the pixels, and the defective pixels which remain static.
[0015] The local orientation of textures in an image is a representation of the orientations of textures at each pixel in the image. The orientation of the texture at a given pixel in the image depends on the neighboring pixels.
[0016] Thus, the invention provides a method for detecting and correcting defective pixels in images delivered by a matrix of photo-detectors, comprising
[0017] - taking into account a stream of images representing a scene, acquired by a device integrating the pixel matrix,
[0018] - detecting movement in the scene, and when movement is detected,
[0019] characterized in that it comprises, for a current pixel of each image of a set of images of the image stream,
[0020] - the prediction of the value of the current pixel, as a function of pixels neighboring the pixel current, selected according to a pattern orientation around the current pixel, to determine a predicted pixel value,
[0021] - the detection of a healthy or defective state of the current pixel, comprising
[0022] o incrementing the value of a counter if the absolute value of the difference between the value of the current pixel and the value of the predicted pixel is greater than a difference threshold, and decrement of the counter value otherwise,
[0023] o comparing the value of the counter to a first and a second hysteresis threshold, the first threshold being greater than the second threshold,
[0024] o the assignment to the current pixel of the defective state if the value of the counter is greater than the first threshold, or of the healthy state if the value of the counter is less than the second threshold, or the conservation of a previous state of the current pixel if the value of the counter is between the first and second hysteresis thresholds,
[0025] - correction of the value of the current pixel if the current pixel is in a defective state, by replacing the value of the current pixel with the value of the predicted pixel.
[0026] Thanks to the invention, the correction of defective pixels is carried out in a manner locally adaptive to the textures of the scene. For a given defective pixel of a given image, the correction depends on its neighborhood in the image.
[0027] Furthermore, the detection of defective pixels is based on the integration over time of the potential successive detections carried out in the previous images of the image stream. The invention thus combines spatial processing of each image and temporal processing of all the images.
[0028] The robustness of the detection of defective pixels and the visual quality of the corrected image are improved compared to known techniques.
[0029] The implementation can be carried out by FPGA. The overall architecture allows the functions to operate together by pooling their FPGA resources and their power consumption.
[0030] According to a preferred characteristic, the prediction of the value of the current pixel comprises the application of high-frequency spatial filters oriented on an area including the current pixel to determine the pattern orientation around the current pixel.
[0031] According to a preferred characteristic, the high-frequency spatial filters are chosen from a vertical filter, a horizontal filter and two diagonal filters.
[0032] According to a preferred characteristic, the prediction of the value of the current pixel involves calculating an average of pixel values neighboring the current pixel selected in the image orientation determined by the pattern orientation around the current pixel.
[0033] The invention also relates to a device integrating a matrix of photo-detectors, comprising means of:
[0034] - taking into account a stream of images representing a scene, acquired by a device integrating the pixel matrix,
[0035] - detecting movement in the scene, and when movement is detected,
[0036] characterized in that it comprises, for a current pixel of each image of a set of images of the image stream, means of:
[0037] - prediction of the value of the current pixel, as a function of pixels neighboring the pixel current, selected according to a pattern orientation around the current pixel, to determine a predicted pixel value,
[0038] - detection of a healthy or defective state of the current pixel, the detection means being suitable for
[0039] o increment the value of a counter if the absolute value of the difference between the value of the current pixel and the value of the predicted pixel is greater than a difference threshold, and decrement the value of the counter otherwise,
[0040] o compare the value of the counter to a first and a second hysteresis threshold, the first threshold being greater than the second threshold,
[0041] o assign the defective state to the current pixel if the value of the counter is greater than the first threshold, or the healthy state if the value of the counter is less than the second threshold, or keep a previous state of the current pixel if the value of the counter is between the first and second hysteresis thresholds,
[0042] - correction of the value of the current pixel if the current pixel is in the defective state, in replacing the value of the current pixel with the value of the predicted pixel.
[0043] The device has advantages similar to those previously presented.
[0044] In a particular embodiment, the steps of the method according to the invention are implemented by computer program instructions.
[0045] Consequently, the invention also relates to a computer program on an information medium, this program being capable of being implemented in a computer, this program comprising instructions adapted to the implementation of the steps of a method as described above.
[0046] The invention also relates to an information medium readable by a computer, and comprising computer program instructions adapted to the implementation of the steps of a method as described above.
[0047] The information carrier may be any entity or device capable of storing the program. The information carrier may be an integrated circuit in in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method according to the invention. Brief description of the drawings
[0048] Other characteristics and advantages will appear on reading the following description of a preferred embodiment, given by way of non-limiting example, described with reference to the figures in which:
[0049] [Fig.l] illustrates a method for detecting and correcting defective pixels according to an embodiment of the invention,
[0050] [Fig.2] illustrates a method of predicting the value of a current pixel, implemented work in the process of [Fig.l],
[0051] [Fig.3] illustrates a method for detecting a healthy or defective state of a pixel current, implemented in the method of [Fig.l], and
[0052] [Fig.4] illustrates a device for detecting and correcting defective pixels according to an embodiment of the invention.
[0053] The different possibilities (variants and embodiments) must be understood as not being mutually exclusive and can be combined with each other.
[0054] Detailed description of particular embodiments
[0055] According to a preferred embodiment, shown in [Fig.l], the method for detecting and correcting defective pixels in images delivered by a matrix of photo-detectors is applied to a stream of images representing a scene, acquired by a device integrating the matrix of photo-detectors. Each photo-detector of the matrix of photo-detectors corresponds to a pixel. The device is for example an infrared matrix imager, or a color camera.
[0056] A pixel is defective when its value does not represent the value of the light intensity received by the photodetector associated with it in the photodetector matrix.
[0057] The method comprises steps E1 to E4, implemented by a computer or in hardware form, such as an FPGA.
[0058] Step E1 is the detection of a movement in the scene. Indeed, the detection of defective pixels is carried out when a movement is detected in the scene in order not to detect as defective the pixels positioned on hot spots of the scene, in the subsequent step E3 described below. By "hot spot", respectively "cold spot", is meant an area of a few pixels in the image which have a value representative of a received light intensity much higher, respectively lower, than the value of the neighboring pixels of the area considered.
[0059] Motion detection can be carried out from an image processing device which, for example, performs an estimation of translations by correlations or a change estimation based on average variations in the gray levels of the image. Motion detection can also be performed by an external device that applies, for example, a movement instruction when a line of sight is controlled by a motorized system, or by an inertial module that measures movement.
[0060] In the following, we assume that a movement has been detected in the image stream. We therefore assume that the hot spots and cold spots of the scene do not remain on the same pixels over time. We consider a set of images of the image stream, and the following steps are performed for each of the pixels of each of the images of the set of images of the image stream.
[0061] Step E2 is a prediction of the value of the current pixel of the current image. One embodiment of pixel value prediction is detailed in [Fig.2] and comprises steps E21 to E22.
[0062] Step E21 is the application of high-frequency spatial filters oriented on a zone z including the current pixel. For example, the zone is centered on the current pixel. It may in particular be a 3x3 pixel pattern whose center is the current pixel.
[0063] For example, a horizontal filter Fign and a vertical filter Fcoi are applied to the considered area z. These can be gradient, Laplacian or other types of filters. The difference in the results of applying the oriented high-frequency spatial filters Fign(z) and Fcoi(z) is then compared to a predetermined threshold Si:
[0064] - If Fcoi(z) - Fign(z) > Si, then the considered zone z including the current pixel has a vertically inclined texture,
[0065] - If Fign(z) - Fcoi(z) > If then the considered zone z including the current pixel has a horizontal texture,
[0066] - Otherwise, the considered zone z including the current pixel has a homogeneous texture.
[0067] According to another example, diagonal filters Fdiagi and Fdiag2 are also applied to the considered zone z. The following differences are compared to the threshold Si:
[0068] - If Fcoi(z) - (Fign(z) + Fdiagl(z) + Fdiag2(z)) / 3 > Sb then the zone considered z including the current pixel has a vertically inclined texture,
[0069] - If Fign(z) - (Fcoi(z) + Fdiagl(z) + Fdiag2(z)) / 3 > Sb then the zone considered z including the current pixel has a horizontally inclined texture,
[0070] - If Fdiagi(z) - (Fign(z) + Fcoi(z) + Fdiag2(z)) / 3 > Sb then the zone considered z including the current pixel has a diagonally oriented texture 1,
[0071] - If Fdiag2(z) - (Fign(z) + Fdiagi(z) + Fcoi(z)) / 3 > If, then the zone considered z including the current pixel has a diagonally 2-trend texture,
[0072] - Otherwise, the considered zone z including the current pixel has a texture with a tendency homogeneous.
[0073] The result of step 21 is a local pattern orientation in the current image. The The result can be a vertical, horizontal, diagonal orientation if diagonal filters are used, or even no orientation.
[0074] The orientation makes it possible to determine the type of prediction (vertical, horizontal or diagonal average) to use for each pixel. In the absence of orientation, the prediction is made by an isotropic average given by the average of the different averages.
[0075] The following step E22 is the determination of the value of the predicted pixel corresponding to the current pixel, as a function of the local orientation of the zone z including the current pixel. The determination of the value of the predicted pixel comprises the calculation of an average of values of pixels neighboring the current pixel selected in the orientation of the image.
[0076] If the local orientation of the z-zone including the current pixel is vertical, then the value of the predicted pixel is determined by averaging the values of the two closest healthy pixels in the vertical orientation of the image.
[0077] Similarly, if the local orientation of the z-zone including the current pixel is horizontal, then the value of the predicted pixel is determined by averaging the values of the two closest healthy pixels in the horizontal orientation of the image.
[0078] If the local orientation of the z-zone including the current pixel is diagonal, then the value of the predicted pixel is determined by averaging the values of the two closest healthy pixels in the same diagonal orientation of the image.
[0079] In the absence of orientation, the value of the predicted pixel is determined by averaging the values of the eight closest healthy pixels around the current pixel.
[0080] Of course, in all the previous cases, the number of healthy pixels taken into account to determine the value of the predicted pixel may be different.
[0081] Alternatively, a number of neighboring pixels is considered, whether these pixels are healthy or corrected. In all cases, the number of pixels taken into account to determine the value of the predicted pixel is not linked to the size of the area considered in step E21.
[0082] Step E2 of predicting the value of the current pixel of the current image is followed by step E3 which is the detection of a healthy or defective state of the current pixel. An embodiment of step E3 is detailed in [Fig.3] and comprises steps E31 to E38.
[0083] A confidence counter is defined for each pixel and its value is incremented or decremented as the images in the set of images considered are processed. The confidence counter provides temporal information on the healthy or defective state of each pixel.
[0084] It should be noted that applying the described processing if motion has been detected in the image stream makes the use of the confidence counter, and therefore the detection of defective pixels, robust to hot or cold spots in the scene.
[0085] Step E31 is the calculation of the absolute value of the difference between the value of the current pixel and the value of the predicted pixel which corresponds to it, determined in step E22.
[0086] The absolute value of the difference between the value of the current pixel and the value of the corresponding predicted pixel is then compared with a predetermined difference threshold. The value of the difference threshold is defined by a preliminary analysis.
[0087] If the absolute value of the difference between the value of the current pixel and the value of the predicted pixel that corresponds to it is greater than the predetermined difference threshold, then step E31 is followed by step E32 in which the value of the counter corresponding to the current pixel is incremented by one. Otherwise, step E31 is followed by step E33 in which the value of the counter corresponding to the current pixel is decremented by one.
[0088] We thus have an image of confidence counters for each image of the image stream. The detection of defective pixels is then carried out using hysteresis thresholding, of which a high threshold and a low threshold are chosen to ensure a compromise between reaction speed of the detection of defective pixels and stability in the image.
[0089] Step E32 and step E33 are followed by step E34 in which the value of the counter is compared to the high hysteresis threshold.
[0090] If the value of the counter is greater than the high hysteresis threshold, then step E34 is followed by step E36 in which a defective state is assigned to the current pixel.
[0091] If the value of the counter is lower than the high hysteresis threshold, then step E34 is followed by step E35 in which the value of the counter is compared to the low hysteresis threshold.
[0092] If the value of the counter is lower than the low hysteresis threshold, then step E35 is followed by step E37 in which a healthy state is assigned to the current pixel. Otherwise, the value of the counter is between the low threshold and the high threshold, and the state of the current pixel remains unchanged (step E38). The previous state of the current pixel, i.e. the state of the current pixel in the previous image of the set of images, is retained.
[0093] Steps E31 to E38 make it possible to change the state of the current pixel during the processing of the images in the set of images considered.
[0094] Step E3 of detecting a state of the current pixel is followed by step E4 in which an output value of the current pixel is determined, depending on the healthy or defective state of the current pixel.
[0095] If the pixel is healthy, then its value is unchanged. In other words, its output value is identical to its input value.
[0096] If the pixel is defective, then its value is corrected by replacing it with the predicted pixel value.
[0097] According to one embodiment, the functions of detection, correction of the pixels de Defective and motion estimation functions are integrated into a single function. Recurring operations such as spatial filters and global statistics calculations are therefore only performed once for all functions. In the case of an FPGA implementation, this pooling makes it possible to drastically reduce the FPGA resources used as well as the power consumption in use, particularly for high-frequency filters which require memory access.
[0098] This solution allows for on-stage management of defective pixels suitable for embedded applications such as portable binoculars.
[0099] The method for detecting and correcting defective pixels according to the invention can be implemented by one or more dedicated integrated circuits or by programmable processors, or even in the form of computer programs stored in the memory of a computer.
[0100] Thus, [Fig.4] represents a particular embodiment of implementation of pixel detection and correction according to the invention.
[0101] The pixel detection and correction device has the general structure of a computer. It comprises in particular a processor 100 executing a computer program implementing the method according to the invention, a memory 101, an input interface 102 and an output interface 103.
[0102] These different elements are conventionally connected by a bus 105.
[0103] The input interface 102 is intended to receive the stream of images to be processed.
[0104] The processor 100 executes the previously described processing operations. These processing operations are carried out in the form of computer program code instructions which are stored by the memory 101 before being executed by the processor 100.
[0105] The memory 101 can furthermore store the results of the treatments carried out.
[0106] The output interface 103 provides the corrected images which can then be stored and / or transmitted in a conventional manner.
Claims
Claims
1. Method for detecting and correcting defective pixels in images delivered by a matrix of photo-detectors, comprising - taking into account a flow of images representing a scene, acquired by a device integrating the pixel matrix, - the detection (El) of a movement in the scene, and when a movement is detected, characterized in that it comprises, for a current pixel of each image of a set of images of the image stream, - the prediction (E2) of the value of the current pixel, as a function of pixels neighboring the current pixel, selected according to a pattern orientation around the current pixel, to determine a predicted pixel value, the detection (E3) of a healthy or defective state of the current pixel, comprising • incrementing (E32) the value of a counter if the absolute value of the difference between the value of the current pixel and the value of the predicted pixel is greater than a difference threshold, and decrementing (E33) the value of the counter otherwise, • the comparison (E34, E35) of the value of the counter with a first and a second hysteresis threshold, the first threshold being greater than the second threshold, • the allocation (E36) to the current pixel of the defective state if the value of the counter is greater than the first threshold, or (E37) of the healthy state if the value of the counter is less than the second threshold, or the conservation (E38) of a previous state of the current pixel if the value of the counter is between the first and second hysteresis thresholds, the correction (E4) of the value of the current pixel if the current pixel is in the defective state, by replacing the value of the current pixel with the value of the predicted pixel.
2. A method for detecting and correcting defective pixels according to claim 1, wherein the prediction of the value of the current pixel comprises the application (E21) of high-frequency spatial filters oriented on an area including the current pixel to determine the pattern orientation around the current pixel.
3. A method for detecting and correcting defective pixels according to claim 2, wherein the high-frequency spatial filters are selected from a vertical filter, a horizontal filter and two diagonal filters.
4. A method for detecting and correcting defective pixels according to claim 2 or 3, wherein the prediction of the value of the current pixel comprises calculating (E22) an average of values of pixels neighboring the current pixel selected in the orientation of the image determined by the pattern orientation around the current pixel.
5. Device integrating a matrix of photo-detectors, comprising means for: - taking into account a stream of images representing a scene, acquired by a device integrating the matrix of pixels, - detecting a movement in the scene, and when a movement is detected, characterized in that it comprises, for a current pixel of each image of a set of images of the stream of images, means for: - predicting the value of the current pixel, as a function of pixels neighboring the current pixel, selected according to a pattern orientation around the current pixel, to determine a predicted pixel value, - detecting a healthy or defective state of the current pixel, the detection means being adapted to • increment the value of a counter if the absolute value of the difference between the value of the current pixel and the value of the predicted pixel is greater than a difference threshold, and decrementing the value of the counter otherwise,• compare the value of the counter to a first and a second hysteresis threshold, the first threshold being higher than the second threshold, • assign the current pixel the defective state if the value of the counter is higher than the first threshold, or the healthy state if the value of the counter is lower than the second threshold, or, keep a previous state of the current pixel if the counter value is between the first and second hysteresis thresholds, - correction of the value of the current pixel if the current pixel is in a defective state, by replacing the value of the current pixel with the value of the predicted pixel.
6. A computer program comprising instructions for carrying out the steps of the method according to any one of claims 1 to 4 when said program is executed by a computer.
7. A computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to any one of claims 1 to 4.