A multi-target-oriented infrared dim and weak moving target enhancement method

Abstract

The application provides a multi-target-oriented infrared dim and weak moving target enhancement method and relates to the technical field of image processing. The multi-target-oriented infrared dim and weak moving target enhancement system comprises an input sequence image, the input sequence image is connected with a spatial domain enhancement, the spatial domain enhancement is connected with time domain average denoising, the time domain average denoising is connected with threshold segmentation, the threshold segmentation is connected with stripe detection, the stripe detection is connected with windowing, the windowing is connected with optical flow detection, the optical flow detection is connected with 3D convolution, and the 3D convolution is connected with output of a local enhancement image. The application provides a multi-target-oriented infrared dim and weak moving target enhancement method. The method is combined with spatial domain and time domain information, pre-processes continuous multi-frame images under a low signal-to-noise ratio condition, realizes multi-target positioning with unknown number, unknown coordinates and unknown motion parameters in a field of view, improves the overall signal-to-noise ratio of images, and greatly reduces the calculation amount of global search.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method for enhancing infrared dark and weak moving targets for multiple targets. Background Technology

[0002] With the increasing prevalence of human space activities, space-based infrared target search and tracking technology is being applied more and more widely in both military and civilian fields. Its main task is to analyze and identify targets, which first requires detecting all targets in the image. On the one hand, due to the long imaging distance, space targets are usually presented as point targets of a few to a dozen pixels with weak intensity, resulting in a low signal-to-noise ratio in their infrared images. In addition, the inherent limitations of infrared detection systems and interference in the deep space environment also affect the imaging effect, increasing the difficulty of target detection. On the other hand, when multiple targets appear in the field of view simultaneously, their unknown number and different motion states make simultaneous detection extremely difficult. Therefore, in order to improve the overall signal-to-noise ratio of the image and further enhance the detection capability of space-based infrared target detection technology, enhancing the infrared image for dark, weak, and moving targets and achieving adaptability to multi-target situations becomes the key to solving the above problems.

[0003] Existing infrared target enhancement methods are mainly divided into two categories: indirectly enhancing the target by processing the background and directly enhancing the target.

[0004] Methods that indirectly enhance targets by processing the background are mainly based on the background characteristics of infrared images and are generally divided into two types: spatial domain processing and transform domain processing. Spatial domain background suppression methods include histogram equalization, gray-level linear transformation and two-dimensional minimum mean square filtering, maximum mean filtering and maximum median filtering, Butterworth high-pass filtering, and morphological filtering algorithms. Transform domain background suppression methods include ideal high-pass filtering, Butterworth high-pass filtering and Gaussian high-pass filtering, traditional frequency domain filtering, and wavelet transforms and their extended transforms—ridge wave transform, curve wave transform, and contour wave transform algorithms. These methods have the advantages of low computational cost and ease of engineering implementation, and they only consider background characteristics, making them adaptable to multi-target situations; however, they leave a large amount of residual noise after processing.

[0005] Methods that directly enhance targets are primarily based on target characteristics, including single-frame methods based on target intensity attributes, gradient attributes, and target-like Gaussian shapes, as well as multi-frame methods based on target motion characteristics. Methods based on target intensity and gradient attributes, such as target enhancement algorithms based on rough set theory, have limited effectiveness for images with very low signal-to-noise ratios. Methods based on target-like Gaussian shapes, such as target enhancement algorithms based on Human Visual Characteristics (HVS), do not consider global background interference. Furthermore, these single-frame methods, if not handled properly, can cause target points to be mistakenly identified as background or noise and removed, making recovery difficult in subsequent processing. Multi-frame methods based on target motion characteristics fully utilize inter-frame correlation and the continuity of target motion, resulting in better performance than the above algorithms. However, these algorithms typically require assumptions about the target's motion state, making them difficult to adapt to situations where targets with different motion states appear simultaneously in real-world scenes.

[0006] In summary, current methods for indirectly enhancing targets through background suppression and single-frame enhancement methods based on target characteristics suffer from performance degradation under low signal-to-noise ratio (SNR) conditions, while multi-frame enhancement methods based on target motion characteristics are mostly unable to handle multi-target scenarios. Therefore, there is an urgent need to design an infrared dark and weak moving target enhancement method and system suitable for low SNR conditions and applicable to multiple targets.

[0007] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0008] (1) In traditional infrared target enhancement methods, the method of indirectly enhancing the target through background suppression will leave a lot of noise after processing. Moreover, for deep space background, there is less background information used. Such methods are not suitable for enhancing space targets.

[0009] (2) In single-frame methods that directly enhance the target, methods based on target intensity and gradient attributes are difficult to implement due to the weakness of the target intensity, while methods based on the Gaussian shape of the target are prone to failure due to the lack of texture information of point targets. Furthermore, if the single-frame method is not processed properly, the target point may be mistaken for background or noise and removed, making it difficult to recover in subsequent processing.

[0010] (3) The multi-frame method that directly enhances the target utilizes the spatiotemporal characteristics of the target and has good performance. However, such algorithms usually need to make assumptions about the motion state of the target and achieve the purpose of enhancement through optimization. They are difficult to apply to the situation where multiple targets with different motion states appear at the same time in the field of view. Summary of the Invention

[0011] (a) Technical problems to be solved

[0012] To address the shortcomings of existing technologies, this invention provides an infrared dark and weak moving target enhancement method for multiple targets. It solves the problems that existing processing methods leave behind large noise, are difficult to achieve the weakness of the target intensity, and are not applicable to situations where multiple targets with different motion states appear simultaneously in the field of view.

[0013] (II) Technical Solution

[0014] To achieve the above objectives, the present invention is implemented through the following technical solution: an infrared dark and weak moving target enhancement system for multiple targets, comprising an input sequence image, wherein the input sequence image is connected to spatial domain enhancement, the spatial domain enhancement is connected to time domain averaging denoising, and the time domain averaging denoising is connected to threshold segmentation;

[0015] The threshold segmentation is connected to stripe detection, the stripe detection is connected to windowing, the windowing is connected to optical flow detection, the optical flow detection is connected to 3D convolution, and the 3D convolution is connected to output local enhancement image.

[0016] Preferably, the spatial enhancement includes energy harvesting and maximum filtering.

[0017] An infrared target enhancement method for multiple targets, including the following steps:

[0018] S1. Spatial domain filtering blind removal element

[0019] For each frame of the image, energy is collected in the m×m neighborhood and isolated and bright pixels are removed. Then, maximum filtering is performed in the n×n neighborhood to enhance and magnify the intensity and size of the target through spatial domain enhancement, while removing most of the blind pixels in the image.

[0020] S2. Time-domain average

[0021] Perform temporal averaging on K consecutive frames in the sequence to reduce the interference of random noise;

[0022] S3. Open the window.

[0023] The number of targets is determined and coarse target localization is obtained by threshold segmentation and connected component detection, and windowing is then performed based on this.

[0024] S4. Estimate speed and direction

[0025] The motion velocity and direction of the target in adjacent frames within each window are estimated using optical flow.

[0026] S5 3D Convolution

[0027] The convolution kernel parameters corresponding to different targets are determined based on the information in step S4, and 3D convolution is performed on the regions where each target is located in the sequence image to achieve enhancement of multiple targets.

[0028] Preferably, the filtering and blinding element in step S1 can lay the foundation for subsequent processing.

[0029] Preferably, the 3D convolution in step S5 can enhance multiple targets and improve the overall signal-to-noise ratio of the image.

[0030] (III) Beneficial Effects

[0031] This invention provides a method for enhancing infrared dark and weak moving targets for multiple targets. It has the following beneficial effects:

[0032] This invention provides an infrared target enhancement method for multiple targets in low-light and low-motion conditions. By combining spatial and temporal information, this invention preprocesses multiple consecutive frames of images under low signal-to-noise ratio conditions, enabling the localization of multiple targets with unknown numbers, coordinates, and motion parameters in the field of view. Furthermore, sparse optical flow detection within a window is performed to determine the motion information of each target. This, combined with 3D convolution, achieves adaptive energy accumulation for multiple targets, improving the overall signal-to-noise ratio of the image. At the same time, the multi-target localization in the preprocessing stage reduces the probability of missed detections and false detections in multi-target situations, while the subsequent windowing operation greatly reduces the computational load of the global search.

[0033] As supporting evidence of the inventive step of the claims of this invention, the following important aspects are also reflected:

[0034] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows: The space-based target surveillance system can monitor and perceive the situation of space targets in real time, and react promptly to dangerous targets when necessary, which has extremely important application value in the military. The multi-target enhancement technology proposed in this invention will greatly improve the detection capability and detection range of the infrared detection system, providing important technical support for the construction of my country's future space target detection system. In addition, this technology can also be widely used in satellite atmospheric infrared cloud image analysis, space remote sensing, infrared medical image pathology analysis, aircraft-captured ground infrared image geological analysis, urban infrared pollution analysis, and civilian fields such as maritime personnel search and rescue.

[0035] (2) The technical solution of this invention solves a long-standing technical problem that people have long desired to solve but have yet to succeed in: Currently, researchers at home and abroad have conducted extensive research on methods for enhancing infrared dark and weak moving targets, and have proposed a variety of effective methods. However, most existing single-frame algorithms have performance limitations or even fail under low signal-to-noise ratio conditions, while most multi-frame algorithms are only applicable to situations where there is only one target in the image. In real-world scenarios, multiple targets often appear in the field of view simultaneously. The infrared dark and weak moving target enhancement method based on optical flow and 3D convolution proposed in this invention solves both the low signal-to-noise ratio and multiple target problems. This algorithm can achieve adaptive enhancement of multiple targets under low signal-to-noise ratio conditions, providing a new direction for research in this field. Attached Figure Description

[0036] Figure 1 This is a flowchart of the infrared dark and weak moving target enhancement method for multiple targets according to the present invention;

[0037] Figure 2 This is a schematic diagram of 3D convolution in this invention;

[0038] Figure 3 The test image input for this invention;

[0039] Figure 4 This is a schematic diagram showing the results of spatial filtering and blind pixel suppression of the test image in this invention;

[0040] Figure 5 This is a schematic diagram of the result of temporal averaging denoising of the test image of this invention;

[0041] Figure 6 This is a processed image of the test image of the present invention;

[0042] Figure 7 This is the optical flow vector map detected by optical flow method within the target window of the original test image of this invention;

[0043] Figure 8 This is a 5-frame 3D convolution result of multiple target windows obtained by processing the image in this invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Example:

[0046] like Figure 1-8As shown, this embodiment of the invention provides an infrared dark and weak moving target enhancement system for multiple targets, including an input sequence image, the input sequence image being connected to spatial domain enhancement, the spatial domain enhancement being connected to time domain averaging denoising, and the time domain averaging denoising being connected to threshold segmentation;

[0047] The threshold segmentation is connected to stripe detection, the stripe detection is connected to windowing, the windowing is connected to optical flow detection, the optical flow detection is connected to 3D convolution, and the 3D convolution is connected to output local enhancement image.

[0048] The spatial enhancement includes energy harvesting and maximum filtering.

[0049] An infrared target enhancement method for multiple targets, including the following steps:

[0050] S1. Filtering and blinding element

[0051] For each frame of the image, energy is collected in the m×m neighborhood and isolated and bright pixels are removed. Then, maximum filtering of the n×n neighborhood is performed to enhance and magnify the intensity and size of the target through spatial domain enhancement, while removing most of the blind pixels in the image, laying the foundation for subsequent processing.

[0052] S2. Time-domain average

[0053] Perform temporal averaging on K consecutive frames in the sequence to reduce the interference of random noise;

[0054] S3. Open the window.

[0055] The number of targets is determined and coarse target localization is obtained by threshold segmentation and connected component detection, and windowing is then performed based on this.

[0056] S4. Estimate speed and direction

[0057] The motion velocity and direction of the target in adjacent frames within each window are estimated using optical flow.

[0058] S5 3D Convolution

[0059] The convolution kernel parameters corresponding to different targets are determined based on the information in step S4. 3D convolution is performed on the regions where each target is located in the sequence image to enhance multiple targets and improve the overall signal-to-noise ratio of the image.

[0060] Based on the method steps of the present invention, the following further details are provided:

[0061] Two filtering operations are performed in a single frame to magnify the target using spatial information and suppress it using the isolation of blind pixels. Then, noise reduction is performed by combining temporal information. After that, multiple targets in the field of view are windowed by coarse localization. The optical flow information of each target is detected within the window. Finally, based on this information, local 3D convolution is performed on consecutive frames to simultaneously enhance multiple targets in the image whose number, position, and velocity are unknown.

[0062] Furthermore, it also includes:

[0063] The algorithm collects target energy by performing m×m neighborhood filtering on each pixel in a single image, then removes isolated bright spots, and performs maximum filtering on the n×n neighborhood to magnify the target. Temporal averaging is performed on K consecutive frames to reduce noise. Coarse localization of multiple targets is obtained through connected component detection, and windowing is performed based on the localization information. Optical flow is used to estimate the target's motion direction and velocity within the window, thereby determining the convolution kernel parameters for 3D convolution. Adaptive enhancement of different targets is achieved by performing 3D convolution within local windows in consecutive frames.

[0064] Furthermore, the energy collection in the m×m neighborhood for each frame of image in step S1 includes:

[0065] f(x,y)=f(x,y)-mean(f Δ ),(x,y)∈Θ

[0066] f(x,y)=f(x,y)×w1×w2

[0067]

[0068]

[0069] Where Θ represents an m×m region centered at (x,y), Ψ represents a 3m×3m pixel region centered at (x,y), and Δ=Ψ-Θ.

[0070] The removal of isolated, bright pixels from the image includes:

[0071]

[0072] ifw>0.65,f′ center =min(f′)

[0073] Where f′ center This represents the pixel value at the center point of region Θ.

[0074] The maximum value filtering within the n×n neighborhood includes:

[0075] f"(x,y)=max(f′) Φ (x,y))

[0076] Where Φ represents an n×n region centered at (x,y).

[0077] The temporal averaging of K consecutive frames in the sequence in step S2 includes:

[0078]

[0079] Where k is the frame number of the image sequence.

[0080] Furthermore, the threshold segmentation and connected component detection performed in step S3 to obtain the window containing the target include:

[0081] (1) Adaptive threshold segmentation

[0082]

[0083] Where the threshold T = μ + k × σ, and the parameter k = 2-5.

[0084] (2) Connected component detection

[0085] After labeling the connected components, traverse all connected components {ψ1,ψ2,ψ3,…} and calculate the connected component ψ. n Internal dimensions and strength information:

[0086]

[0087]

[0088] in Representing the connected region ψ n The number of non-zero pixels contained Representing the connected region ψ n The pixel values ​​in the image.

[0089] Define confidence level:

[0090]

[0091] Determine whether a connected component is the target trajectory by using confidence level:

[0092]

[0093] Where Th is the threshold, and {φ} is retained. i} is the target trajectory, and from this, the window {Ω} where the target is located is obtained. i Location information of}.

[0094] Furthermore, the use of optical flow within each window in step S4 includes:

[0095] (u,v) i =Flow(Ω) i )

[0096] Where Flow() represents the LK optical flow method, (u,v) i The detected optical flow vector is actually a pixel displacement value, which can be regarded as the motion vector of the target in adjacent frames.

[0097] The 3D convolution in step S5:

[0098]

[0099] Where Conv3d() represents a 3D convolution function, kernel i This represents the convolution kernel determined in step S4. (output) Ωi The enhanced window for the target.

[0100] Figure 2 This is a schematic diagram of 3D convolution in step S5 provided in this embodiment of the invention;

[0101] Figure 3 The input test image provided in the embodiment of the present invention; wherein Figure (1) is a long-wave infrared background image; Figure (2) is a simulated multiple point targets; and Figure (3) is a test image formed by fusing the simulated targets and the long-wave infrared background image.

[0102] Figure 4 This is a schematic diagram of the result of the test image provided in the embodiment of the present invention being processed by spatial filtering in step S1 and having blind cells suppressed;

[0103] Figure 5 This is a schematic diagram of the result of time-domain denoising of the test image provided in the embodiment of the present invention through step S2;

[0104] Figure 6 The image shown is the result of processing the test image provided in this embodiment of the invention after step S3; where Figure (1) is the threshold segmentation result; and Figure (2) is the connected component detection result.

[0105] Figure 7 It is the optical flow vector within the target window of the original test image provided in this embodiment of the invention, detected by the optical flow method in step S4;

[0106] Figure 8 This is the 5-frame 3D convolution result of multiple target windows obtained by step S5 from the image provided in this embodiment of the invention.

[0107] The infrared dark and weak motion target enhancement system for multiple targets provided in this embodiment of the invention includes:

[0108] The spatiotemporal filtering module is used to perform m×m neighborhood filtering on each pixel in a single image to achieve spatial enhancement, while also performing blind pixel suppression and maximum value filtering on the n×n neighborhood of the image; then, K-frame temporal averaging is performed on the sequence of images.

[0109] The multi-target localization module performs threshold segmentation and connected component detection on the spatiotemporally processed image to determine the existence of targets and locate the window in which the targets are located.

[0110] The optical flow detection module performs optical flow detection on the motion trajectory within different target windows to obtain the target's motion vector.

[0111] The 3D convolution module determines the 3D convolution kernel parameters within different windows based on the aforementioned target motion vectors, performs 3D convolution operations on the sequence images, and obtains the enhanced window images of the target.

[0112] II. Application Examples. To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.

[0113] 1. By analyzing typical detection distances, the flight trajectories of space targets, detector resolution, and background and noise characteristics under space-based detection conditions, infrared images of actual scenarios were simulated. Considering the diffusion phenomenon caused by the optical system on point targets, simulation experiments were conducted under different numbers of targets, different target sizes, different signal-to-noise ratios, and different target movement speeds. All simulations achieved excellent enhancement effects, significantly improving the overall signal-to-noise ratio. Experiments showed that the multi-target enhancement algorithm based on optical flow detection and 3D convolution proposed in this invention is applicable to different conditions, including point target sizes of 3*3 and above, signal-to-noise ratios above 2, and target movement speeds within 1.5.

[0114] 2. A simulated infrared detection scenario of space targets was established, and a real long-wave infrared background image was obtained using a long-wave infrared camera. Multiple simulated moving targets were added to this image sequence and used as test images to verify the multi-target enhancement method based on optical flow detection and 3D convolution proposed in this invention. Experimental results show that the algorithm proposed in this invention can effectively enhance multiple targets simultaneously, suppress a large amount of background and noise, remove some interference from detector blind cells, and exhibits stability under different conditions.

[0115] III. Evidence of the Relevant Effects of the Embodiments. The embodiments of the present invention have achieved some positive effects during research and development or use, and indeed possess significant advantages compared to existing technologies. The following description, combined with data and charts from the experimental process, illustrates these advantages.

[0116] For space targets, due to the long detection distance, the target size is smaller than the sensor's smallest resolution unit, resulting in a speckled image, generally considered a point target. Furthermore, influenced by spaceborne optoelectronic imaging systems, point target imaging exhibits diffusion, represented by a point spread function. In contrast, the deep space background is a relatively pure 4K cold background, and cosmic background radiation can be ignored. Noise in infrared images that significantly affects targets is random noise, generally considered to approximately follow a Gaussian distribution. Moreover, in actual detection scenarios, multiple targets with different characteristics and motion patterns often appear simultaneously within the field of view. Based on the above analysis, the test images used in this invention are primarily sequence images obtained by adding multiple infrared simulated targets to a long-wave infrared background image.

[0117] Target simulation environment: Matlab2020b.

[0118] Test environment: Matlab2020b, Python3.6.

[0119] Test image: Long-wave infrared image, size 512*512.

[0120] Target information: Point target, with some diffusion; target size is determined by energy concentration.

[0121] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An infrared low-light and weak moving target enhancement system for multiple targets, comprising an input sequence of images, characterized in that: The input sequence image is connected to spatial domain enhancement, the spatial domain enhancement is connected to time domain averaging denoising, and the time domain averaging denoising is connected to threshold segmentation. The threshold segmentation is connected to stripe detection, the stripe detection is connected to windowing, the windowing is connected to optical flow detection, the optical flow detection is connected to 3D convolution, and the 3D convolution is connected to output local enhancement image.

2. The infrared dark and weak moving target enhancement system for multiple targets according to claim 1, characterized in that: The spatial enhancement includes energy harvesting and maximum filtering.

3. A method for enhancing infrared dark and weak moving targets for multiple targets, characterized in that, Specifically, the following steps are included: S1. Filtering and blinding element Energy is collected in a 3×3 neighborhood for each frame of the image, and isolated and bright pixels are removed. Then, maximum filtering is performed in a 5×5 neighborhood to enhance and enlarge the intensity and size of the target through spatial domain enhancement, while removing most of the blind pixels in the image. S2. Time-domain average A temporal average is performed on eight consecutive frames in the sequence to reduce the interference of random noise; S3. Open the window. The number of targets is determined and coarse target localization is obtained by threshold segmentation and connected component detection, and windowing is then performed based on this. S4. Estimate speed and direction Within each window, the motion velocity and direction of the target in adjacent frames are estimated using optical flow. S5 3D Convolution The convolution kernel parameters corresponding to different targets are determined based on the information in step S4, and 3D convolution is performed on the regions where each target is located in the sequence image to achieve enhancement of multiple targets.

4. The infrared dark and weak moving target enhancement method for multiple targets according to claim 3, characterized in that: The filtering and blinding element in step S1 lays the foundation for subsequent processing.

5. The infrared dark and weak moving target enhancement method for multiple targets according to claim 3, characterized in that: In step S5, 3D convolution can enhance multiple targets and improve the overall signal-to-noise ratio of the image.

Images