A method for detecting and tracking small infrared targets in complex backgrounds

Abstract

The present invention belongs to the field of embedded platform image processing technology, and in particular relates to a method for detecting and tracking small infrared targets in complex backgrounds. The method comprises: grayscale processing of several frames of images; extracting regions of interest for image registration to determine image offsets; performing sub-pixel translation on the grayscaled frames to convert them to the same coordinate system; determining a target grayscale value; and determining whether the current state is tracking or detecting based on the relationship between the target grayscale value and a set grayscale threshold; and then performing target detection and output. While ensuring algorithm accuracy, the method significantly reduces computational complexity and improves algorithm execution efficiency, providing a technical approach for detecting and tracking small infrared targets in complex embedded backgrounds.

Description

Technical Field

[0001] The present invention belongs to the technical field of embedded platform image processing, and in particular relates to a method for detecting and tracking infrared dim small targets against complex backgrounds. Background Art

[0002] With the development of image processing technology, target detection and tracking based on image processing technology has been applied in many fields, such as embedded airborne and missile-borne fields.

[0003] Existing methods are effective in detecting small targets in simple backgrounds, i.e., images with a stable spatial distribution of grayscale and invariant grayscale statistical mean and variance. For example, regional images composed of a single physical component such as the sky, sea surface, and desert are effective. However, the detection of small infrared targets with low signal-to-noise ratio in complex backgrounds such as sea-sky background and complex ground background is still an unresolved problem.

[0004] Embedded platforms have limited computing resources and relatively low computing power, which places higher demands on the computational efficiency of algorithms. In summary, it is of great significance to study infrared dim target detection and tracking technology for embedded computing platforms in complex backgrounds. Summary of the Invention

[0005] The present invention provides a method for detecting and tracking small infrared targets in complex backgrounds. Based on the collected images of several adjacent frames, the foreground image is extracted through grayscale conversion, image registration, sub-pixel translation, and multi-frame difference methods to achieve small target detection. The method also uses a dynamic region of interest (ROI) to achieve target tracking. Considering the small size of the target, which occupies only a few pixels, and the complexity and computational efficiency of the target background, a phase correlation method is used for image registration. Simultaneously, by employing a dynamic region of interest (ROI), the algorithm significantly reduces the computational effort and improves its execution efficiency while ensuring algorithm accuracy. This provides a technical approach for detecting and tracking small infrared targets in embedded complex backgrounds.

[0006] The technical solution of the present invention:

[0007] A method for detecting and tracking infrared small targets in complex backgrounds, comprising:

[0008] S1, collecting a plurality of adjacent frames of images of a small infrared target in real time through a camera, and performing grayscale processing on the plurality of frames of images;

[0009] S2, extracting regions of interest from the grayscale processed frames, and performing image registration on the extracted images of the regions of interest, thereby determining an image offset;

[0010] S3, performing sub-pixel translation on the plurality of frames of images after the grayscale processing according to the image offset, and converting the plurality of frames of images into the same coordinate system;

[0011] S4, determining a target grayscale value, and judging whether the current state is tracking state or detection state according to the relationship between the target grayscale value and a set grayscale threshold;

[0012] S5, if it is in the tracking state, extract the region of interest from several frames of images converted to the same coordinate system, perform target detection, and output the target detection result image;

[0013] S6: If it is in the detection state, target detection is directly performed on several frame images converted into a unified coordinate system, and a target detection result image is output.

[0014] Furthermore, in S1, for the collected RGB image, the pixel grayscale value is obtained according to Gray = R*0.299+G*0.587+B*0.114, where R, G, and B are the three-channel pixel values ​​of the RGB image, and Gray is the grayscale value of the pixel after grayscale conversion.

[0015] Furthermore, in S2, a phase-correlation-based image registration algorithm is used to perform image registration on the extracted region of interest image.

[0016] Furthermore, in S2, the image offset is determined as follows:

[0017] Suppose g(x,y) is the image obtained by translating f(x,y) by (x0,y0), and perform inverse discrete Fourier transform DFT on it, then it satisfies:

[0018]

[0019] where G(u,v) and F(u,v) are the discrete Fourier transforms of g(x,y) and f(x,y) respectively;

[0020] Then the cross power spectrum between the two images in frequency domain is

[0021] Among them G * (u,v) is the complex conjugate of G(u,v), |F(u,v)G * (u,v)| is F(u,v)G * Amplitude spectrum of (u,v);

[0022] Then the inverse discrete Fourier transform (IDFT) of P(u,v) is:

[0023]

[0024] The translation amount x0, y0 of the two frames of image can be obtained by finding the maximum value of the IDFT of the cross power spectrum.

[0025] Furthermore, S3 is specifically:

[0026] The image frame sequence is transformed into a unified coordinate system using sub-pixel translation based on bilinear interpolation; the grayscale value of bilinear interpolation is

[0027] f(i+x0,j+y0)=(1-x0)*(1-y0)f(i,j)

[0028] +(1-x0)*y0*f(i,j+1)+x0*(1-y0)*f(i+1,j)+x0*y0*f(i+1,j+1)

[0029] Where x0, y0 are pixel offset values.

[0030] Furthermore, S4 is specifically:

[0031] Determining the target grayscale value is to superimpose the foreground images of several adjacent frames of images, and then obtain the final target by threshold segmentation, and the grayscale value of the final target is the target grayscale value;

[0032] When the target grayscale value is less than or equal to the set grayscale threshold, it is in the detection state;

[0033] When the target grayscale value is greater than the set grayscale threshold, it is in the tracking state.

[0034] Furthermore, the method for extracting the region of interest in S5 is specifically as follows:

[0035] Referring to the position of the target in the previous frame image, the area around the target in the current frame sequence is extracted as the target detection region of interest.

[0036] Furthermore, after S3, the method further includes: performing image stitching based on a plurality of frames of images converted into the same coordinate system to obtain a panoramic image of the target.

[0037] The present invention has the following technical effects:

[0038] 1. The present invention provides a method for detecting and tracking infrared weak targets in complex backgrounds. According to the characteristics of complex backgrounds and weak infrared targets, a target detection algorithm is designed. Phase-correlation image registration, multi-frame difference target detection and other means are adopted to realize the detection of weak infrared targets in complex backgrounds, and at the same time, it has the function of real-time image stitching.

[0039] 2. The present invention provides a method for detecting and tracking infrared small targets in complex backgrounds. It is deployed on embedded platforms and applies the region of interest method to image registration and tracking process. It effectively improves the algorithm running speed while ensuring detection accuracy, and provides strong support for the application of the algorithm in scenarios with limited computing resources such as airborne and missile-borne platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is the overall flow chart of infrared dim target detection and tracking against complex backgrounds;

[0041] Figure 2 This is the image registration flow chart based on the phase correlation method;

[0042] Figure 3 This is a schematic diagram of target detection based on multi-frame difference method;

[0043] Figure 4 This is a schematic diagram of infrared dim target detection and tracking results against a complex background. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0045] The embodiment of the present invention provides a method for detecting and tracking infrared small targets in complex backgrounds. Figure 1 As shown, the following steps are included:

[0046] Step 1: Use the camera to collect several adjacent frames of images in real time, convert them into grayscale, and use the processed images for the next step of processing;

[0047] Step 2: Take the original multi-frame grayscale image converted in step 1 as input, extract the region of interest from the image frame considering the algorithm execution efficiency, and use the extracted image frame region as input for image registration;

[0048] Step 3: Based on the image registration output results in step 2, perform sub-pixel translation on the original multi-frame grayscale images. The translated multi-frame images are used for subsequent target detection and tracking.

[0049] Step 4: Determine the current state based on the relationship between the target grayscale and the threshold. If it is a detection state, use the translated multi-frame image as input, use the frame difference method to extract the foreground image, superimpose the multi-frame foreground images, and then use the target threshold to extract the target and output the target position in the image.

[0050] Step 5: According to the status judgment of step 4, if it is a tracking state, the region of interest around the target is extracted from the multi-frame image after registration and translation based on the target position of the previous frame, the extracted image frame area is used as input for target detection, and the position of the target in the image is output.

[0051] The number of image frames required for one calculation of the algorithm in step 1 can be adjusted according to the algorithm accuracy and execution speed. Theoretically, as the number of image frames calculated at one time increases, the algorithm accuracy will be improved, but the amount of calculation will also increase, reducing the algorithm execution efficiency. Therefore, the number of frames needs to be selected according to actual needs.

[0052] As a preferred solution of the present invention, the step 2 takes the continuous grayscale image frames output in step 1 as input. Taking into account the offset of the image background between each frame, a registration algorithm is required to realize the registration between each continuous frame, and convert them into the same coordinate system for subsequent foreground extraction, superposition and target detection. Theoretically, the computational complexity of the registration algorithm is proportional to the image size. In order to reduce the computational complexity of the image registration algorithm, the step 2 adopts the region of interest method to select the local area of ​​each image frame as input, which significantly reduces the registration time. At present, image registration algorithms can be roughly divided into region-based registration, feature-based registration, hybrid model-based registration and physical model-based registration. Taking into account the registration efficiency, this step selects an image registration algorithm based on the phase correlation method. The phase-correlation-based image registration algorithm is a region-based registration algorithm with extremely high registration efficiency, which can cope with image translation and brightness changes.

[0053] As a preferred embodiment of the present invention, step 3 uses a bilinear interpolation method based on the inter-frame translation amount obtained by the registration algorithm in step 2 to convert each frame image to a unified coordinate system through sub-pixel translation. Typical image sub-pixel translation methods include nearest neighbor interpolation, regional interpolation, bicubic interpolation, and bilinear interpolation. Considering both accuracy and efficiency, bilinear interpolation is selected as the sub-pixel translation method.

[0054] As a preferred embodiment of the present invention, the target detection method used in step 4 is target detection based on the multi-frame difference method. The image frames that have been aligned and sub-pixel-shifted and converted to a unified coordinate system are used as input. The foreground is extracted by difference between each adjacent frame and the first frame of the frame sequence. The foreground images are then superimposed to enhance the target area. Finally, the target is extracted based on the threshold to obtain the position of the target in the image. To improve the efficiency of the algorithm, the grayscale relationship between the target area and the threshold is used to determine whether it is in the detection state or the tracking state. If it is below the threshold, it is switched to the detection state, that is, the dynamic region of interest extraction is skipped and the entire frame image is used as input for target detection.

[0055] As a preferred solution of the present invention, the target tracking is realized by extracting the region of interest in step 5, that is, referring to the target position of the previous frame, extracting the area around the target in the current frame sequence as the target detection input, avoiding using the entire frame image as input, greatly reducing the target detection calculation amount, and significantly improving the detection speed in the tracking state. When the grayscale value of the target area is greater than the threshold, the tracking state is maintained, otherwise it is switched to the detection state. In the tracking state, the position of the target in the region of interest is obtained by multi-frame difference target detection, and the position of the target in the entire frame image is obtained by inverse transformation based on the position of the region of interest.

[0056] Specifically, the embodiment of the present invention provides a method for detecting and tracking infrared small targets in complex backgrounds, including image input, image frame grayscale conversion, image registration, image sub-pixel translation, image dynamic region of interest extraction, target detection and other processes, such as Figure 1 shown.

[0057] During the detection and tracking of infrared dim targets under complex backgrounds, the camera collects N frames of continuous images in real time and performs grayscale conversion on the collected images. For the collected RGB images, the pixel grayscale value is obtained according to Gray = R*0.299+G*0.587+B*0.114, where R, G, and B are the pixel values ​​of the three channels of the RGB image respectively, and Gray is the grayscale value of the pixel after grayscale conversion.

[0058] Taking grayscale image frames as input, we use image registration methods to transform each frame into a unified coordinate system. Considering registration efficiency, we choose a phase-correlation-based image registration algorithm. Phase-correlation-based image registration is a region-based registration algorithm with high efficiency and robustness to image translation and brightness changes.

[0059] Consider only the translation transformation of the two input frames. Let g(x,y) be the image obtained by translating f(x,y) by (x0,y0). Perform inverse discrete Fourier transform (DFT) on each of them, and then satisfy

[0060]

[0061] where G(u,v) and F(u,v) are the discrete Fourier transforms of g(x,y) and f(x,y) respectively.

[0062] Then the cross power spectrum between the two images in frequency domain is

[0063]

[0064] Among them G * (u,v) is the complex conjugate of G(u,v), |F(u,v)G * (u,v)| is F(u,v)G * The magnitude spectrum of (u,v).

[0065] Then the inverse discrete Fourier transform (IDFT) of P(u,v) is

[0066]

[0067] The translation amount of the two frames of images can be obtained by finding the maximum value of the IDFT of the cross power spectrum. The specific algorithm steps are as follows: Figure 2 In order to reduce the computational complexity of the registration algorithm, while ensuring the registration accuracy, the image of a partial area of ​​the image frame is used as input to further improve the registration efficiency.

[0068] Through the image registration algorithm, the inter-frame offset is obtained, and the image frame sequence is transformed into a unified coordinate system using sub-pixel translation based on bilinear interpolation. The grayscale value of bilinear interpolation is

[0069] f(i+x0,j+y0)=(1-x0)*(1-y0)f(i,j)

[0070] +(1-x0)*y0*f(i,j+1)+x0*(1-y0)*f(i+1,j)+x0*y0*f(i+1,j+1)

[0071] Where x0, y0 are pixel offset values.

[0072] After each frame image is translated and converted to a unified coordinate system, the target foreground image is extracted using the image frame difference. To further enhance the target, the foreground images of several adjacent frames are superimposed, and then the final target is obtained by threshold segmentation, as shown in the following example: Figure 3 As shown. In order to improve the detection efficiency, the area around the target position of the previous frame is used to extract the region of interest. The foreground image of the region of interest is extracted, superimposed and threshold segmented to replace the target detection of the entire image frame. The target tracking stage is constructed to reduce the amount of calculation and achieve improved detection performance. By comparing the grayscale value of the target area with the threshold, the switching between detection and tracking states is realized. When the grayscale of the target area of ​​the current frame is less than the set state switching threshold, it is switched to the detection state, that is, the extraction of the dynamic area is canceled, otherwise the tracking state is maintained. The target detection and tracking result is shown in the figure Figure 4 shown.

[0073] The present invention provides a method and system for detecting and tracking infrared small targets against complex backgrounds. Based on the characteristics of complex backgrounds and small infrared targets, a target detection algorithm is designed. Phase-correlated image registration and multi-frame difference target detection are used to detect small infrared targets against complex backgrounds, while also providing real-time image stitching capabilities. The present invention provides a method and system for detecting and tracking infrared small targets against complex backgrounds. Designed for deployment on embedded platforms, the system applies the region of interest method to image registration and tracking processes, effectively improving the algorithm's operating speed while ensuring detection accuracy. This provides strong support for the algorithm's application in scenarios with limited computing resources, such as those onboard aircraft and missiles.

[0074] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting and tracking infrared small targets in complex backgrounds, characterized in that: The method comprises: S1, collecting a plurality of adjacent frames of images of a small infrared target in real time through a camera, and performing grayscale processing on the plurality of frames of images; S2, extracting regions of interest from the grayscale processed frames, and performing image registration on the extracted images of the regions of interest, thereby determining an image offset; S3, performing sub-pixel translation on the plurality of frames of images after the grayscale processing according to the image offset, and converting the plurality of frames of images into the same coordinate system; S4, determining a target grayscale value, and judging whether the current state is tracking state or detection state according to the relationship between the target grayscale value and a set grayscale threshold; S5, if it is in the tracking state, extract the region of interest from several frames of images converted to the same coordinate system, perform target detection, and output the target detection result image; S6: If it is in the detection state, target detection is directly performed on several frame images converted into a unified coordinate system, and a target detection result image is output.

2. The method for detecting and tracking infrared small targets in complex backgrounds according to claim 1, wherein: In S1, for the collected RGB image, the pixel grayscale value is obtained according to Gray = R*0.299+G*0.587+B*0.114, where R, G, and B are the three-channel pixel values ​​of the RGB image respectively, and Gray is the grayscale value of the pixel after grayscale conversion.

3. The infrared small target detection and tracking method for complex background according to claim 1, characterized in that: In S2, the extracted region of interest images are registered using a phase correlation-based image registration algorithm.

4. The method for detecting and tracking infrared small targets in complex backgrounds according to claim 3, wherein: In S2, the image offset is determined as follows: Suppose g(x,y) is the image obtained by translating f(x,y) by (x0,y0), and perform inverse discrete Fourier transform DFT on it, then it satisfies: where G(u,v) and F(u,v) are the discrete Fourier transforms of g(x,y) and f(x,y) respectively; Then the cross power spectrum between the two images in frequency domain is Among them G * (u,v) is the complex conjugate of G(u,v), |F(u,v)G * (u,v)| is F(u,v)G * Amplitude spectrum of (u,v); Then the inverse discrete Fourier transform (IDFT) of P(u,v) is: The translation amount x0, y0 of the two frames of image can be obtained by finding the maximum value of the IDFT of the cross power spectrum.

5. The method for detecting and tracking infrared small targets in complex backgrounds according to claim 4, characterized in that: S3 specifically: The image frame sequence is transformed into a unified coordinate system using sub-pixel translation based on bilinear interpolation; the grayscale value of bilinear interpolation is f(i+x0,j+y0)=(1-x0)*(1-y0)f(i,j) +(1-x0)*y0*f(i,j+1)+x0*(1-y0)*f(i+1,j)+x0*y0*f(i+1,j+1) Where x0, y0 are pixel offset values.

6. The method for detecting and tracking infrared small targets in complex backgrounds according to claim 5, characterized in that: S4 is specifically: Determining the target grayscale value is to superimpose the foreground images of several adjacent frames of images, and then obtain the final target by threshold segmentation, and the grayscale value of the final target is the target grayscale value; When the target grayscale value is less than or equal to the set grayscale threshold, it is in the detection state; When the target grayscale value is greater than the set grayscale threshold, it is in the tracking state.

7. The method for detecting and tracking infrared small targets in complex backgrounds according to claim 1, wherein: The specific method of extracting the region of interest in S5 is: Referring to the position of the target in the previous frame image, the area around the target in the current frame sequence is extracted as the target detection region of interest.

8. The method for detecting and tracking infrared small targets in complex backgrounds according to claim 1, wherein: After S3, the method further includes: performing image stitching based on a plurality of frames of images converted into the same coordinate system to obtain a panoramic image of the target.

Images