An image defect element suppression method, device and computer readable storage medium

Abstract

The application discloses an image defect element suppression method, device and computer readable storage medium, and the image defect element suppression method comprises the following steps: collecting an infrared image; performing spatial local extremum detection on a current infrared image to obtain a spatial potential defect element in the current infrared image; continuously monitoring a pixel corresponding to the spatial potential defect element in the current infrared image in subsequent multiple infrared images to determine whether the pixel is a defect element; and performing defect element suppression and filling on the defect element. According to the technical scheme, the potential defect element is obtained by performing local extremum detection on the collected infrared image, and the corresponding potential defect element is monitored in the subsequent multiple infrared images, so that whether the potential defect element is a defect element is determined, and the defect element is suppressed and filled, thereby reducing the quality defect of infrared imaging and improving the accuracy of target detection through the infrared image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for suppressing image defect elements, and a computer-readable storage medium. Background Art

[0002] Due to factors such as material defects, manufacturing processes, photoelectric signal readout amplifier circuits, and line noise, domestic detectors inevitably contain defective pixels when outputting infrared images. The defect rate for medium-wave detectors generally ranges from 1‰ to 6‰, while for long-wave detectors, it can reach 8‰ to 20‰. Without proper processing, the output infrared image can contain fixed bright spots, dark spots, and flickering pixels. Sometimes, these defects, such as blind pixel clusters and flickering blind pixel clusters, can severely impact detector imaging quality. This is particularly true in applications requiring small and dim target detection and tracking, as they can easily lead to missed detections or false alarms. Therefore, effective processing of these defects is crucial. Summary of the Invention

[0003] The present invention provides an image defect element suppression method, device and computer-readable storage medium, which are used to solve the problem in the prior art that infrared images output by detectors are prone to defect elements, resulting in poor imaging quality.

[0004] A method for suppressing image defect elements according to an embodiment of the first aspect of the present invention includes:

[0005] Collect infrared images;

[0006] Performing spatial local extremum detection on the current infrared image to obtain spatial potential defect elements in the current infrared image;

[0007] Continuously monitoring the pixel corresponding to the spatial potential defect element in the current infrared image in subsequent multiple frames of infrared images, and determining whether the pixel is a defect element through dynamic time domain statistics;

[0008] Defective element suppression and filling are performed on the defective element.

[0009] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image includes:

[0010] Based on the factory blind pixel table, the current infrared image is processed by pixel removal;

[0011] Perform spatial local extreme value detection on the current infrared image after pixel elimination.

[0012] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image to obtain spatial potential defect elements in the current infrared image includes:

[0013] Performing at least one pooling operation on the current infrared image to obtain at least one infrared image of a different size;

[0014] Performing spatial local extrema detection on the current infrared image and the at least one infrared image of different sizes to obtain spatial potential defect elements in the current infrared image.

[0015] According to some embodiments of the present invention, performing at least one pooling operation on the current infrared image includes:

[0016] Perform 2-6 pyramid pooling operations on the current infrared image.

[0017] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image includes:

[0018] A 4-neighborhood filter or an 8-neighborhood filter is used to detect local extreme values ​​in the spatial domain of the current infrared image.

[0019] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image includes:

[0020] Based on morphological filtering, mean difference, gradient difference, or saliency features, the current infrared image is subjected to spatial local extreme value detection.

[0021] According to some embodiments of the present invention, continuously monitoring the pixel corresponding to the spatial potential defect element in the current infrared image in subsequent multiple frames of infrared images, and determining whether the pixel is a defect element through dynamic time domain statistics, includes:

[0022] Counting the proportion of pixels corresponding to the spatial potential defect element in the current infrared image that are determined to be spatial potential defect elements in subsequent multiple frames of infrared images;

[0023] When the ratio exceeds a specific ratio threshold, the pixel is determined to be a defective pixel.

[0024] According to some embodiments of the present invention, the defective elements include isolated defective elements and defective element clusters.

[0025] According to an embodiment of the second aspect of the present invention, an image defect element suppression device is proposed, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps of the image defect element suppression method as described in any one of the embodiments of the first aspect are implemented.

[0026] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the image defect element suppression method as described in any one of the first aspect embodiments are implemented.

[0027] By adopting the embodiment of the present invention, local extreme value detection is performed on the collected infrared image to obtain potential defect elements, and the corresponding potential defect elements are monitored in subsequent multiple frames of infrared images to determine whether they are defect elements and suppress and fill them, thereby reducing the quality defects of infrared imaging and improving the accuracy of target detection through infrared images.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the embodiments below. The accompanying drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0030] Figure 1 1 is a flow chart of a method for suppressing image defect elements in an embodiment of the present invention;

[0031] Figure 2 1 is a flow chart of a method for suppressing image defect elements in an embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of image pooling and extreme value detection in an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram of directly performing DDE image enhancement on collected data in the prior art;

[0034] Figure 5 Schematic diagram of performing DDE image enhancement after defective elements are eliminated in an embodiment of the present invention;

[0035] Figure 6 Schematic diagram of the structure of the image defect element suppression device in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0037] The first embodiment of the present invention proposes a method for suppressing image defect elements, referring to Figure 1 ,include:

[0038] S1: Collect infrared images.

[0039] For example, the infrared image may be an infrared image output by devices such as an area array detector and a linear array detector.

[0040] S2: Performing spatial local extremum detection on the current infrared image to obtain spatial potential defect elements in the current infrared image.

[0041] The current infrared image here can be understood as the infrared image collected at the current moment.

[0042] S3: Continuously monitor the pixel corresponding to the spatial potential defect element in the current infrared image in subsequent multiple frames of infrared images, and determine whether the pixel is a defect element through dynamic time domain statistics.

[0043] It can be understood that if certain pixels in the current infrared image are judged to be potential spatial defect elements, the pixels will be monitored in real time in the subsequent multiple frames of infrared images. If the pixel meets the defect element judgment conditions in the subsequent multiple frames of infrared images, then this pixel is a defect element, and it is necessary to suppress and fill the defect element at the position corresponding to the pixel in the infrared image.

[0044] For example, the grayscale value of the potential defective element in the continuous time domain is monitored, and the grayscale value is compared with the threshold to determine whether it is a defective element. The defective element can be a blind element, a flashing element, etc.

[0045] S4: Suppressing and filling the defective elements.

[0046] For example, the position of the defective element is read and located, and a multi-scale median filter is used to suppress or fill the defective element in the image.

[0047] By adopting the embodiment of the present invention, local extreme value detection is performed on the collected infrared image to obtain potential defective elements, and the potential defective elements are continuously monitored to determine whether they are defective elements and suppress and fill them, thereby reducing the quality defects of the detector's infrared imaging and improving the accuracy of the detector's target detection.

[0048] Figure 4 The following shows the direct DDE image enhancement of the collected data. Figure 5 The image is enhanced by DDE after defect removal. Defect removal facilitates DDE image enhancement, allowing the detector to clearly display the target during target detection. The number of removals can be set to one or more times based on the desired defect removal effect.

[0049] Based on the above embodiment, various modified embodiments are further proposed. It should be noted that, in order to simplify the description, only the differences from the above embodiment are described in each modified embodiment.

[0050] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image includes:

[0051] Defective pixels are removed from the current infrared image based on the factory blind pixel table. This table lists hardware issues with the chip itself, such as defects in the chip array element itself, which cause problems in the corresponding pixels. This table can be used to initially filter out defective pixels in the infrared image.

[0052] Perform spatial local extreme value detection on the current infrared image after pixel elimination.

[0053] Performing spatial local extrema detection on the current infrared image can obtain isolated defect pixels, such as blind pixels and flicker pixels. Blind pixels also include overheated pixels and dead pixels.

[0054] By eliminating pixels, some defective pixels can be eliminated, thereby avoiding repeated identification and detection of these defective pixels in the subsequent detection process, thereby reducing the workload of subsequent detection and determination of defective pixels in the image.

[0055] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image to obtain spatial potential defect elements in the current infrared image includes:

[0056] At least one pooling operation is performed on the current infrared image to obtain at least one infrared image of different sizes. Thus, a frame of infrared image can be expanded into a series of infrared images of different sizes.

[0057] For example, average pooling or maximum pooling can be used to perform pooling operations on infrared images.

[0058] Performing spatial local extrema detection on the current infrared image and at least one infrared image of different sizes to obtain spatial potential defect elements in the current infrared image.

[0059] When performing differential comparisons between defective pixels and surrounding pixels to identify spatial potential defects in infrared images, if the surrounding pixels of a particular pixel are also defective, this method will not be able to detect all of them. However, if the above method is further applied to the pooled, scaled-down infrared image for spatial potential defects, all of these defect clusters can be detected. Multiple pooling and scaling can be performed, as this may not be possible in a single pooling run. This avoids the problem of incorrect defective pixel determination caused by indifferent comparisons when the surrounding pixels of the defective pixel are also defective, and improves the accuracy of defective extreme value detection.

[0060] According to some embodiments of the present invention, performing at least one pooling operation on the current infrared image includes:

[0061] Perform 2-6 pyramid pooling operations on the current infrared image.

[0062] For example, reference Figure 3 , perform pyramid pooling operations three times on the current infrared image. The original infrared image size is 640×512. After one pooling operation, the infrared image size obtained is 320×256. The infrared image obtained after the first pooling operation is pooled again, and the size of the infrared image obtained is 160×128.

[0063] Pooling infrared images at different pyramid levels can yield infrared images of corresponding sizes. Detecting extreme values ​​based on infrared images of different scales can improve the accuracy of extreme value detection.

[0064] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image includes:

[0065] A 4-neighborhood filter or an 8-neighborhood filter is used to detect local extrema in the spatial domain of the current infrared image. The process of using a 4-neighborhood filter to detect local extrema in the spatial domain of the current infrared image is to compare the grayscale of each pixel with the grayscale of the four surrounding pixels. The four surrounding pixels are the upper neighboring pixels, the lower neighboring pixels, the left neighboring pixels, and the right neighboring pixels of the current center pixel.

[0066] The process of using the 8-neighborhood filter to detect local extreme values ​​in the spatial domain of the current infrared image is to compare the grayscale of each pixel with the grayscale of all the eight adjacent pixels around it.

[0067] A picture element corresponds to a pixel in an infrared image. A picture element in an infrared image can be understood as a pixel in an infrared image.

[0068] For example, first, the size of the current pixel and the surrounding pixels in the infrared image are calculated respectively. ε is the error threshold related to the detector performance. Considering that the difference between the weak target and the actual blind flash element and the background is extremely small, ε is 0 in this embodiment. The potential defect element judgment value T is calculated according to the following formula:

[0069]

[0070]

[0071]

[0072]

[0073] T=|Q1+Q2+Q5+Q4|;

[0074] Among them, Q1, Q2, Q3, and Q4 respectively represent the comparison values ​​of the potential defect pixel and the pixels in the four directions above, below, left, and right. P represents the grayscale value of the pixel, and i and j represent the coordinate values ​​of the pixel.

[0075] If the T value is 4, it is determined to be a potential defective element. That is, if the grayscale value of the potential defective pixel is greater than or less than the grayscale value of the pixels in the four directions of up, down, left, and right, it is determined to be a defective element.

[0076] According to some embodiments of the present invention, performing spatial local extrema detection on the current infrared image includes:

[0077] Based on morphological filtering, mean difference, gradient difference, or saliency features, the current infrared image is subjected to spatial local extreme value detection.

[0078] According to some embodiments of the present invention, continuously monitoring the pixel corresponding to the spatial potential defect element in the current infrared image in subsequent multiple frames of infrared images, and determining whether the pixel is a defect element through dynamic time domain statistics, includes:

[0079] The proportion of pixels corresponding to the spatial potential defect elements in the current infrared image that are determined to be spatial potential defect elements in subsequent multiple frames of infrared images is counted.

[0080] For example, statistics are performed based on the time domain online dynamic statistical algorithm, as shown in Table 1, and the potential defect pixel position matrix S is obtained after local extreme value detection. C(i, j) is the number of frames for which statistics are performed for each corresponding pixel. R(i, j) is the number of frames in which the defective pixel corresponding to each pixel appears. Cth is the threshold for the number of statistical frames, and the value is 30 in combination with the real-time requirements of the detection system. V is the online update threshold for the number of frames, and it is 3000 with reference to the time domain characteristics of the flash element. Pth is the threshold for the ratio of the number of frames in which potential defects appear to the total number of statistical frames. The total number of frames here represents the frame value counted from the image frame in which the potential defect element first appears. The ratio threshold can be adjusted according to parameters such as the complexity of the scene, working time, and working temperature. In this embodiment, the value is 0.5.

[0081] Table 1 Principle of time domain statistical algorithm

[0082]

[0083] When the ratio exceeds a specific threshold, the pixel is determined to be defective. The threshold can be adjusted based on actual conditions. It is related to the system's tolerance for defective pixels. The algorithm in this embodiment enables dynamic statistics without specifying a specific frame count, offering strong robustness and enabling rapid and flexible identification and screening of isolated bright pixels, dark pixels, and flashing pixels based on their characteristics.

[0084] According to some embodiments of the present invention, infrared images at different pooling scales are filtered in the spatial and temporal domains to screen defect element clusters at different scales, thereby further improving the accuracy of defect element determination.

[0085] According to some embodiments of the present invention, defective elements include isolated defective elements and defective element clusters. Isolated defective elements include, but are not limited to, blind elements and flashing elements. Blind elements include, but are not limited to, dead elements and overheated elements. Defective element clusters include, but are not limited to, blind element clusters, flashing element clusters, and mixed clusters of blind and flashing elements.

[0086] According to some embodiments of the present invention, a defective pixel update mechanism is formulated and an online update threshold is set. When the value is below the threshold, the original defective pixel is judged as a normal pixel, so that the defective pixel restored to a normal pixel does not participate in filling, suppression or replacement.

[0087] The following describes a method for suppressing image defects in detail using a specific embodiment. It should be understood that the following description is merely illustrative and does not limit the present invention. Any similar structures and variations of the present invention should be included within the scope of protection of the present invention.

[0088] In this embodiment, reference Figure 2First, the infrared images are continuously collected and stored for a short time. It should be noted that not all images are stored for a long time. Only the position coordinate information of the candidate defective pixels needs to be stored after the collection operation to save storage space. Defective pixels are removed from the infrared image based on the factory blind pixel table. Three-layer pyramid pooling is performed on the removed infrared image to obtain infrared images of three scales. A 4-neighborhood filter is used to perform spatial local extreme value detection on infrared images of different scales to extract spatial potential defect pixels, as follows:

[0089] Calculate the size of the current pixel and surrounding pixels in the infrared image. ε is the error threshold related to detector performance. Considering that the difference between small targets and possible blind flash elements and the background is extremely small, ε is 0 in this embodiment. Calculate the potential defect element judgment value T according to the following formula:

[0090]

[0091]

[0092]

[0093]

[0094] T=|Q1+Q2+Q5+Q4|;

[0095] Among them, Q1, Q2, Q3, and Q4 respectively represent the comparison values ​​of the potential defect pixel and the pixels in the four directions above, below, left, and right. P represents the grayscale value of the pixel, and i and j represent the coordinate values ​​of the pixel.

[0096] If the T value is equal to 4, it is judged as a potential defect element.

[0097] Perform dynamic statistical filtering of time domain characteristics, and online count the time domain grayscale of consecutive frames of potential defective pixels. If the ratio of the number of frames in which the defective pixel appears to the total number of statistical frames reaches the ratio threshold, it is determined to be a defective pixel. The specific operation is shown in Table 1:

[0098] Table 1 Principle of time domain statistical algorithm

[0099]

[0100]

[0101] After local extreme value detection, the potential defect pixel position matrix S is obtained. C(i, j) is the number of frames for which statistics are performed for each corresponding pixel. R(i, j) is the number of frames in which the defective pixel corresponding to each pixel appears. Cth is the threshold for the number of statistical frames. Combined with the real-time requirements of the detection system, the value is 30. V is the threshold for online frame updates. Referring to the time domain characteristics of the flash element, the value is 3000. Pth is the ratio of the number of frames in which potential defects appear to the total number of statistical frames. If it is greater than this ratio, it is determined to be a defective pixel. In this embodiment, the value is 0.5 (50%). This value is related to the defective element tolerance of the system. The threshold of this ratio can also be adaptively controlled according to parameters such as the complexity of the scene, working time, and working temperature. At the same time, this probability is not 50% of the total number of imaging times after the pixel element is turned on, but 50% of the total number of statistics starting from the first appearance of the potential defect element.

[0102] Establish a defective pixel update mechanism and set an online update threshold. When the value is lower than the threshold, the original defective pixel will be judged as a normal pixel, so that the defective pixel that has been restored to a normal pixel will not be involved in filling, suppression or replacement.

[0103] Infrared images at other pooling scales are subjected to spatial local extremum detection and temporal dynamic statistical filtering to screen for defect clusters of different scales. The resulting defect cluster locations are multiplied by the number of pooling cycles to calculate their positions within the original infrared image. Multi-scale median filtering is then used to read the locations of the defect clusters, suppressing and filling them.

[0104] Finally, the processed image data is output to the display terminal through a high-speed data interface for display.

[0105] Using the technical solution of this embodiment, defect elements are identified, marked, suppressed, and filled as the detector acquires infrared images, without affecting ongoing detector operations. Pyramid-based pooling allows for detection of defects at varying scales, further improving detection accuracy.

[0106] It should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations, and that the various embodiments may be freely combined. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

[0107] According to the second aspect of the present invention, a proposed image defect element suppression device is implemented, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the image defect element suppression method as described in any one of the embodiments of the first aspect are implemented.

[0108] For example, reference Figure 6 The equipment includes an optical lens, a detector assembly, a focal plane interface board, a drive and signal processing circuit module and a computer terminal. The optical lens is installed on the detector assembly. The focal plane interface board is electrically connected to the detector assembly and the drive and signal processing circuit module. The drive and signal processing circuit includes an FPGA signal processing chip, a DDR3 cache unit and a power supply chip. The drive and signal processing circuit mainly completes the signal transfer of the detector assembly and filters the bias voltage, including: providing bias voltage for the detector assembly, generating the timing signal required by the detector assembly, conditioning the analog channel signal output by the detector, AD conversion and splicing processing (if it is a digital infrared detector, AD conversion is not required), digital image processing (including online blind pixel, flash pixel, defect pixel cluster suppression algorithm) and sending the imaging data to the terminal display through the camerlink interface. The drive and signal processing circuit is electrically connected to the computer terminal. The optical lens is used for detector imaging focusing. The imaging software runs on the computer, including functions such as interface data reception and image display verification.

[0109] The optical lens and detector assembly generate an infrared image, which is sent to the driver and signal processing circuit module via the focal plane interface board. Defective elements are first removed from the infrared image according to the factory blind element table. The removed infrared image is then pooled. Finally, the infrared image undergoes spatial local extreme value detection and dynamic statistical filtering of temporal characteristics to complete the determination of defective elements. A multi-scale median filter is used to suppress and fill in defective elements.

[0110] The image defect element suppression device in this embodiment has a low amount of computational effort on the image based on the image defect element suppression method, is easy to implement with hardware such as FPGA, and is a common module in this field, making the device highly adaptable.

[0111] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the image defect element suppression method as described in any one of the first aspect embodiments are implemented.

[0112] It should be noted that the computer-readable storage medium in this embodiment includes but is not limited to: ROM, RAM, magnetic disk or optical disk, etc. The program processor may be a mobile phone, computer, server, air conditioner, or network equipment, etc.

[0113] It should be noted that, in the description of this specification, well-known methods, structures and technologies are not shown in detail so as not to obscure the understanding of this specification.

Claims

1. A method for suppressing image defect elements, characterized in that: include: Collect infrared images; Performing spatial local extremum detection on the current infrared image to obtain spatial potential defect elements in the current infrared image; Continuously monitoring the pixel corresponding to the spatial potential defect element in the current infrared image in subsequent multiple frames of infrared images, and determining whether the pixel is a defect element through dynamic time domain statistics; Suppressing and filling the defective elements; The performing spatial local extremum detection on the current infrared image to obtain spatial potential defect elements in the current infrared image includes: Performing at least one pooling operation on the current infrared image to obtain at least one infrared image of a different size; Performing spatial local extrema detection on the current infrared image and the at least one infrared image of different sizes to obtain spatial potential defect elements in the current infrared image.

2. The method according to claim 1, wherein The performing of spatial local extrema detection on the current infrared image includes: Based on the factory blind pixel table, the current infrared image is processed by pixel removal; Perform spatial local extreme value detection on the current infrared image after pixel elimination.

3. The method according to claim 1, wherein The performing at least one pooling operation on the current infrared image includes: Perform 2-6 pyramid pooling operations on the current infrared image.

4. The method according to any one of claims 1 to 3, wherein The performing of spatial local extrema detection on the current infrared image includes: A 4-neighborhood filter or an 8-neighborhood filter is used to detect local extreme values ​​in the spatial domain of the current infrared image.

5. The method according to any one of claims 1 to 3, wherein The performing of spatial local extrema detection on the current infrared image includes: Based on morphological filtering, mean difference, gradient difference, or saliency features, the current infrared image is subjected to spatial local extreme value detection.

6. The method according to claim 1, wherein The continuously monitoring the pixel corresponding to the spatial potential defect element in the current infrared image in the subsequent multiple frames of infrared images, and determining whether the pixel is a defect element through dynamic time domain statistics, includes: Counting the proportion of pixels corresponding to the spatial potential defect element in the current infrared image that are determined to be spatial potential defect elements in subsequent multiple frames of infrared images; When the ratio exceeds a specific ratio threshold, the pixel is determined to be a defective pixel.

7. The method according to claim 1, wherein The defective elements include isolated defective elements and defective element clusters.

8. An image defect element suppression device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the image defect element suppression method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an information transmission implementation program, and when the program is executed by a processor, the steps of the image defect element suppression method according to any one of claims 1 to 7 are implemented.

Images