Method and apparatus for dead pixel extraction for infrared detector using deep learning model

The deep learning model-based method for infrared seekers enhances dead pixel extraction by combining traditional methods with YOLO, addressing clustering and edge issues, ensuring accurate and real-time detection.

KR103046213B1Active Publication Date: 2026-11-10LIG DEFENSE & AEROSPACE CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
KR1020230051411
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-04-19
Publication Date
2026-11-10
Estimated Expiration
2043-04-19

AI Technical Summary

Technical Problem

Existing methods for extracting dead pixels in infrared seekers struggle with clustered dead pixels and those located at image edges, particularly due to vignetting effects.

Method used

A method and apparatus using a deep learning model, such as YOLO, to predict dead pixels from infrared data, combined with traditional window or global threshold-based extraction, and a deduplication process to enhance accuracy and coverage.

Benefits of technology

Effectively and accurately extracts dead pixels regardless of clustering or edge location, enabling real-time processing and improved accuracy without requiring window or threshold settings.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112023044161296-PAT00001_ABST
    Figure 112023044161296-PAT00001_ABST
Patent Text Reader

Abstract

A method for extracting dead pixels in an infrared detector according to the present invention comprises: a step of acquiring infrared data from an infrared detector of an infrared detector; a step of extracting dead pixels from the infrared data based on a window or a global threshold; a step of extracting dead pixels or bad pixels from the infrared data through a deep learning model trained to predict dead pixels or bad pixels; and a step of combining the result of extracting dead pixels based on the window or global threshold with the result of extracting dead pixels or bad pixels through the deep learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a method and apparatus for extracting dead pixels of an infrared seeker. Background Technology

[0002] An infrared seeker is a device that identifies and tracks targets by detecting infrared radiation within a specific band, and it is mounted on the front of guided weapons such as guided missiles. Due to the nature of guided weapons, the background and the target must be clearly distinguished in the infrared image, and ambient noise must be minimized when tracking long-range targets. Therefore, to improve the quality of the infrared image from the seeker, a process is required to extract dead pixels that exhibit abnormal output.

[0003] Existing dead pixel extraction methods include window-based approaches that set up an N×N window around a target pixel and compare its value with that of pixels within the window. However, window-based methods have a problem in that they are difficult to extract when dead pixels are clustered.

[0004] In addition, there is an existing dead pixel extraction method based on a global threshold, which sets a global threshold by adding a specific value to the average of all pixel values ​​in an infrared image and extracts pixels exceeding this threshold as dead pixels. However, due to the nature of infrared seekers equipped with optical elements, there are difficulties in extracting dead pixels located at the outer edges of the image caused by vignetting, where the image becomes darker from the center to the edges. The problem to be solved

[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for extracting dead pixels of an infrared seeker that can effectively and accurately extract dead pixels regardless of whether the dead pixels are clustered or located at the outer edge of an image.

[0006] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0007] A method for extracting dead pixels in an infrared detector according to the present invention for solving the above technical problem comprises: a step of acquiring infrared data from an infrared detector of an infrared detector; a step of extracting dead pixels from the infrared data based on a window or a global threshold; a step of extracting dead pixels or bad pixels from the infrared data through a deep learning model trained to predict dead pixels or bad pixels; and a step of combining the result of extracting dead pixels based on the window or global threshold with the result of extracting dead pixels or bad pixels through the deep learning model.

[0008] The deep learning model described above is a deep learning model that takes an RGB 3-channel image converted from infrared data as input data and takes dead pixels or bad pixels as prediction data, and the step of extracting dead pixels or bad pixels through the deep learning model may include: a step of converting the infrared data into RGB 3-channel image data; and a step of inputting the RGB 3-channel image data into the deep learning model to extract dead pixels or bad pixels through the deep learning model.

[0009] The above deep learning model may be a YOLO model.

[0010] The deep learning model above may be trained using training data in which multiple RGB 3-channel images converted from multiple infrared data are used as input data, and dead pixels or bad pixels extracted by a user for each RGB 3-channel image are used as ground-truth data.

[0011] The dead pixel extraction method of the infrared seeker described above may further include the step of performing deduplication on the combined extraction result.

[0012] The above infrared data can be acquired for each frame that makes up the infrared video.

[0013] A dead pixel extraction device for an infrared seeker according to the present invention for solving the above technical problem comprises: an infrared data acquisition unit that acquires infrared data from an infrared detector of an infrared seeker; a window or global threshold-based dead pixel extraction unit that extracts dead pixels from the infrared data based on a window or global threshold; a deep learning model-based dead pixel extraction unit that extracts dead pixels or bad pixels from the infrared data through a deep learning model trained to predict dead pixels or bad pixels; and an extraction result combining unit that combines the dead pixel extraction result based on the window or global threshold and the dead pixel or bad pixel extraction result through the deep learning model.

[0014] The deep learning model described above is a deep learning model that takes an RGB 3-channel image converted from infrared data as input data and takes dead pixels or bad pixels as prediction data, and the deep learning model-based dead pixel extraction unit converts the infrared data into RGB 3-channel image data and inputs the RGB 3-channel image data into the deep learning model to extract dead pixels or bad pixels through the deep learning model.

[0015] The above deep learning model may be a YOLO model.

[0016] The dead pixel extraction device of the infrared seeker may further include a deep learning model training unit that trains the deep learning model using training data in which a plurality of RGB 3-channel images converted from a plurality of infrared data are used as input data, and dead pixels or bad pixels extracted by a user for each RGB 3-channel image are used as ground-truth data.

[0017] The above extraction result combining unit can perform duplicate removal on the combined extraction result.

[0018] The above infrared data can be acquired for each frame that makes up the infrared video. Effects of the invention

[0019] According to the dead pixel extraction method and apparatus of the infrared seeker of the present invention, dead pixels can be effectively and accurately extracted regardless of whether the dead pixels are clustered or located at the outer edge of the image.

[0020] In addition, it is possible to extract dead pixels in real time from infrared data input in real time.

[0021] In addition, dead pixel extraction through a deep learning model does not require window settings or threshold settings, and can be applied to various infrared explorers through learning various infrared data.

[0022] In addition, dead pixels and bad pixels can be extracted more accurately by combining the results of dead pixel extraction based on a window or global threshold with the results of dead pixel or bad pixel extraction through deep learning.

[0023] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0024] FIG. 1 shows a block diagram of a dead pixel extraction device of an infrared seeker according to one embodiment of the present invention. FIG. 2 shows the structure of a YOLOv5 model as an example of a deep learning model that can be used in an embodiment of the present invention. Figure 3 shows a flowchart of the process of training a deep learning model according to an embodiment of the present invention. FIG. 4 shows a flowchart of a dead pixel extraction method of an infrared seeker according to one embodiment of the present invention. FIG. 5 shows an example in which the results of dead pixel extraction based on a window or global threshold and the results of dead pixel or bad pixel extraction through a deep learning model are combined according to an embodiment of the present invention. Figure 6 shows an enlarged view of a portion of Figure 5. Specific details for implementing the invention

[0025] Preferred embodiments of the present invention will be described in detail below with reference to the drawings. In the following description and the attached drawings, substantially identical components are denoted by the same reference numerals to avoid redundant description. Furthermore, in describing the present invention, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the invention, such detailed description will be omitted.

[0026] FIG. 1 shows a block diagram of a dead pixel extraction device of an infrared seeker according to one embodiment of the present invention.

[0027] A dead pixel extraction device of an infrared seeker according to an embodiment of the present invention comprises an infrared data acquisition unit (110), a window or global threshold-based dead pixel extraction unit (120), a deep learning model-based dead pixel extraction unit (130), a deep learning model (140), a deep learning model training unit (150), and an extraction result combining unit (160).

[0028] The infrared data acquisition unit (110) acquires infrared data from the infrared detector of the infrared seeker.

[0029] Infrared data is data in which an infrared light signal is converted into an electrical signal by an infrared detector in an infrared seeker. Infrared data is two-dimensional data in which each pixel has a pixel value; for example, if the range of pixel values ​​is 0 to 16383, it can be represented by 14 bits, however, the size of the data may vary depending on the performance or specifications of the infrared seeker. Infrared data can be acquired for each frame that makes up an infrared video.

[0030] The window or global threshold-based dead pixel extraction unit (120) extracts dead pixels from infrared data based on a window or a global threshold. Window-based dead pixel extraction is a technique that sets an N×N window around a target pixel and extracts dead pixels by comparing the pixel values ​​of the pixels within the window. Global threshold-based dead pixel extraction is a technique that sets a global threshold by adding a specific value to the average of all pixel values ​​in the infrared image and extracts pixels that fall outside the global threshold as dead pixels.

[0031] The deep learning model (140) is a deep learning model that takes an RGB 3-channel image converted from infrared data as input data and takes dead pixels or bad pixels as prediction data. Here, a bad pixel is a pixel that is not completely dead pixel to the user (or, it may be a dead pixel but is ambiguous), and is a pixel that is likely to become a dead pixel in the future in the infrared detector.

[0032] The deep learning model (140) can be constructed using the YOLO model. The YOLO (You Only Look Once) model is a model designed as a Convolutional Neural Network (CNN) for object detection, and performs feature extraction, bounding box calculation, and class classification. The YOLO model performs training by dividing the input image into three RGB channels. Therefore, even a 1-channel grayscale image must be converted into a 3-channel image for training and detection to be possible. Therefore, grayscale infrared data must be converted into an RGB 3-channel image and input into the deep learning model (140). The deep learning model (140) can receive an RGB 3-channel image and output the location and size of dead pixels or bad pixels.

[0033] FIG. 2 shows the structure of the YOLOv5 model as an example of a deep learning model (140). The Focus layer is an evolution of the YOLOv3 structure, replacing the first three layers of YOLOv3 and creating a single layer. Conv represents a convolution layer, and C3 consists of three convolution layers and a module cascaded by various bottlenecks. SPP (Spatial Pyramid Pooling) is a pooling layer used to remove the network's size limit from being fixed. Upsample is used to upsample the fusion of the previous layer at the nearest node. Concat is a slicing layer and is used to slice the previous layer. The last three Conv2d are sensing modules used at the head of the network.

[0034] The deep learning model training unit (150) converts multiple infrared data acquired by the infrared data acquisition unit (110) into RGB 3-channel image data, and trains the deep learning model (140) using training data in which multiple RGB 3-channel images are used as input data and dead pixels or bad pixels extracted by the user for each RGB 3-channel image are used as ground-truth data.

[0035] FIG. 3 shows a flowchart of the process of training a deep learning model (140) according to an embodiment of the present invention.

[0036] In step 310, a number of infrared data are acquired by the infrared data acquisition unit (110).

[0037] In step 320, the deep learning model training unit (150) converts multiple infrared data into RGB 3-channel images.

[0038] In step 330, for each RGB 3-channel image, the user extracts dead pixels or bad pixels.

[0039] In step 340, multiple RGB 3-channel images are used as input data, and for each RGB 3-channel image, the extracted dead pixels or bad pixels are used as ground-truth data to construct the training data.

[0040] In step 350, a deep learning model (140) is trained using training data. Here, a neural network model is trained to minimize the loss function, and a back propagation algorithm may be used for training. As a result of training, the deep learning model (140) becomes a unique detection model capable of detecting dead pixels or bad pixels intended by the user from an input RGB 3-channel image.

[0041] Referring again to FIG. 1, the deep learning model-based dead pixel extraction unit (130) extracts dead pixels or bad pixels from infrared data through a deep learning model (140) trained to predict dead pixels or bad pixels. The deep learning model-based dead pixel extraction unit (130) can convert infrared data into RGB 3-channel image data and input the RGB 3-channel image data into the deep learning model (140) to extract dead pixels or bad pixels through the deep learning model (140).

[0042] When a GPU (Graphics Processing Unit) is utilized, when infrared data is acquired for each frame of an infrared video, dead pixels or bad pixels can be extracted in real time through a deep learning model (140) at a speed of, for example, 30 fps (frames per second) or more.

[0043] Referring again to FIG. 1, the extraction result combining unit (160) combines the window or global threshold-based dead pixel extraction result by the window or global threshold-based dead pixel extraction unit (120) with the deep learning model-based dead pixel extraction result through the deep learning model by the deep learning model-based dead pixel extraction unit (130).

[0044] The dead pixel extraction results based on a window or global threshold and the dead pixel or bad pixel extraction results through a deep learning model may contain the same dead pixel. Therefore, the extraction result combining unit (160) can perform deduplication on the combined extraction results.

[0045] FIG. 4 shows a flowchart of a dead pixel extraction method of an infrared seeker according to one embodiment of the present invention.

[0046] In step 410, infrared data is acquired from the infrared detector of the infrared seeker. Infrared data is data in which an infrared light signal is converted into an electrical signal by the infrared detector in the infrared seeker. Infrared data is two-dimensional data in which each pixel has a pixel value; for example, if the range of pixel values ​​is 0 to 16383, it can be represented by 14 bits, however, the size of the data may vary depending on the performance or specifications of the infrared seeker. Infrared data can be acquired for each frame that makes up the infrared video.

[0047] In step 420, an N×N window or global threshold is set for the infrared data.

[0048] In step 430, dead pixels are extracted from infrared data based on a window or a global threshold. That is, an N×N window is set around the target pixel and dead pixels are extracted by comparing the pixel values ​​of the pixels within the window, or a global threshold is set by adding a specific value to the average of the pixel values ​​of the entire infrared image and pixels that fall outside the global threshold are extracted as dead pixels.

[0049] In step 440, the infrared data is converted into RGB 3-channel image data.

[0050] In step 450, RGB 3-channel image data is input into a deep learning model (140) to extract dead pixels or bad pixels through the deep learning model (140). The deep learning model (140) is a deep learning model that takes an RGB 3-channel image converted from infrared data as input data and takes dead pixels or bad pixels as prediction data, and is trained through the learning process of FIG. 3. The deep learning model (140) can be constructed using a YOLO model.

[0051] In step 450, when a GPU (Graphics Processing Unit) is utilized, when infrared data is acquired for each frame of the infrared video, dead pixels or bad pixels can be extracted in real time through a deep learning model (140) at a speed of, for example, 30 fps (frames per second) or more.

[0052] In step 460, the window or global threshold-based dead pixel extraction results from step 430 are combined with the dead pixel or bad pixel extraction results from the deep learning model in step 450.

[0053] The extraction results of step 430 and step 450 may contain the same dead pixels. Therefore, in step 470, deduplication is performed on the combined extraction results.

[0054] FIG. 5 shows an example in which the results of dead pixel extraction based on a window or global threshold and the results of dead pixel or bad pixel extraction through a deep learning model are combined according to an embodiment of the present invention, and FIG. 6 shows an enlarged view of a part of FIG. 5.

[0055] Referring to Figures 5 and 6, it can be seen that dead pixels or bad pixels that are not extracted based on the window or global threshold are additionally extracted through the deep learning model.

[0056] According to an embodiment of the present invention, dead pixels or bad pixels can be effectively and accurately extracted regardless of whether the dead pixels are clustered or located at the outer edge of the image. Furthermore, it is possible to extract dead pixels in real time from infrared data input in real time in the form of video from an infrared detector. In addition, dead pixel extraction using a deep learning model does not require window settings or threshold settings, and can be applied to various infrared detectors through learning from various infrared data. Moreover, dead pixels and bad pixels can be extracted more accurately by combining the dead pixel extraction results based on windows or global thresholds with the dead pixel or bad pixel extraction results obtained through deep learning.

[0057] A device according to embodiments of the present invention may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with an external device, and user interface devices such as a touch panel, a key, a button, etc. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable code or program instructions executable on the processor. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROM, DVD (Digital Versatile Disc)). Computer-readable recording media may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium may be readable by a computer, stored in memory, and executed by a processor.

[0058] Embodiments of the present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the embodiment may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., which can execute various functions by the control of one or more microprocessors or other control devices. Similar to how components of the present invention may be implemented as software programming or software elements, the embodiment may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, etc., including various algorithms implemented as combinations of data structures, processes, routines, or other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors. Additionally, the embodiment may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations. The above terms may include the meaning of a series of software processes (routines) in conjunction with processors, etc.

[0059] The specific embodiments described in the embodiments are examples and do not limit the scope of the embodiments in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as "essential," "importantly," etc., a component may not be absolutely necessary for the application of the present invention.

[0060] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will understand that the present invention may be embodied in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

Claim 1 The method comprises: a step of acquiring infrared data from an infrared detector of an infrared seeker; a step of setting a global threshold by adding a specific value to the average of all pixel values ​​of an infrared image from the infrared data, and extracting pixels that fall outside the global threshold as dead pixels; a step of extracting dead pixels or bad pixels from the infrared data through a deep learning model trained to predict dead pixels or bad pixels; and a step of combining the result of extracting dead pixels based on the global threshold and the result of extracting dead pixels or bad pixels through the deep learning model, wherein the deep learning model is a deep learning model that takes an RGB 3-channel image converted from infrared data as input data and takes dead pixels or bad pixels as prediction data, and the step of extracting dead pixels or bad pixels through the deep learning model includes a step of converting the infrared data into RGB 3-channel image data; A method for extracting dead pixels in an infrared detector, comprising the step of inputting the RGB 3-channel image data into the deep learning model and extracting dead pixels or bad pixels through the deep learning model, wherein the deep learning model is a YOLO model, and the deep learning model is trained using training data in which a plurality of RGB 3-channel images converted from a plurality of infrared data are used as input data and dead pixels or bad pixels extracted by a user for each RGB 3-channel image are used as ground-truth data, and further comprising the step of performing duplicate removal on the combined extraction result, wherein the infrared data is acquired for each frame constituting an infrared video. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 An infrared data acquisition unit that acquires infrared data from an infrared detector of an infrared seeker; a global threshold-based dead pixel extraction unit that sets a global threshold by adding a specific value to the average of all pixel values ​​of an infrared image from the infrared data, and extracts pixels that fall outside the global threshold as dead pixels; and a deep learning model-based dead pixel extraction unit that extracts dead pixels or bad pixels from the infrared data through a deep learning model trained to predict dead pixels or bad pixels. The infrared method further comprises an extraction result combining unit that combines the global threshold-based dead pixel extraction result and the dead pixel or bad pixel extraction result obtained through the deep learning model, wherein the deep learning model is a deep learning model that takes an RGB 3-channel image converted from infrared data as input data and takes dead pixels or bad pixels as prediction data, wherein the deep learning model-based dead pixel extraction unit converts the infrared data into RGB 3-channel image data, inputs the RGB 3-channel image data into the deep learning model to extract dead pixels or bad pixels through the deep learning model, wherein the deep learning model is a YOLO model, and further comprises a deep learning model training unit that trains the deep learning model using training data in which a plurality of RGB 3-channel images converted from a plurality of infrared data are taken as input data and dead pixels or bad pixels extracted by a user for each RGB 3-channel image are used as ground-truth data, wherein the extraction result combining unit performs duplicate removal on the combined extraction result, and wherein the infrared data is acquired for each frame constituting the infrared video. Explorer dead pixel extraction device. Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete

Citation Information

Patent Citations

  • Imaging apparatus and its control method

    JP2017055308A

  • Image enhancement system, method, and recording medium recording a computer-readable program for executing the method

    KR1020220129837A