A method for detecting moving targets in infrared video in a dynamic background

Abstract

The application discloses a kind of dynamic background under infrared video's moving target closed loop detection method, comprising: the noise type of infrared image is analyzed, using filtering algorithm to remove noise processing to image;Different interference angle point filtering method is taken for different frames, so as to carry out coarse elimination and fine elimination to angle point, and the final detection angle point is stored and recorded;Sparse optical flow method is used to track the above-mentioned angle point, and using bidirectional tracking to filter tracking point, according to the relationship between the two frame angle point set Homography matrix calculation, using homography transformation matrix to carry out background compensation to current frame image;The previous infrared image is differentiated with the current frame image after background compensation, and the adaptive gray threshold binary processing is carried out to difference result, and the final target position is obtained by morphological operation to binary image;According to the previous frame detection target position, form mask, feedback to next frame angle point detection, form a complete closed loop detection loop.

Description

Technical Field

[0001] This invention relates to the field of infrared moving target detection, and more particularly to a method for closed-loop detection of moving targets in infrared video under dynamic background. Background Technology

[0002] Moving target detection is one of the core research topics in computer vision. It is the foundation for target tracking, target recognition, and target behavior understanding, and has broad application prospects in military, security monitoring, industrial automation, and intelligent transportation. Depending on whether the shooting platform or camera is moving, moving target detection can be divided into moving target detection under static backgrounds and moving target detection under dynamic backgrounds. Moving target detection under static backgrounds occurs when the camera is stationary during shooting, and the resulting video sequence contains only the movement of the target. Moving target detection under dynamic backgrounds occurs when both the shooting platform or camera and the moving target change simultaneously during shooting, including translation, rotation, and scaling. The resulting video sequence contains not only the movement of the target but also the movement of the background, making it significantly more difficult than moving target detection under static backgrounds. The main method used for moving target detection under dynamic backgrounds is background compensation. This involves calculating the transformation matrix between the previous and current frames to compensate for the background in the current frame, and then using frame difference to detect the moving target. The accuracy of the background compensation directly affects the detection accuracy.

[0003] Compared to visible light images, infrared images have lower contrast, lower resolution, and more noise. This results in fewer and more concentrated corner points or feature points extracted from infrared images during background compensation compared to visible light images. This affects the accuracy of subsequent transformation matrix calculations and the overall accuracy of background compensation. Furthermore, while corner points or feature points in the background play a significant role in transformation matrix calculations, feature points in moving target areas can hinder accurate background registration. The proportion of feature points in moving target areas is higher in infrared images compared to visible light images. This may increase the probability that the PROSAC algorithm selects feature points from moving target areas during feature point selection, leading to inaccurate registration and affecting the final detection results. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention discloses a method for detecting closed-loop motion targets in infrared video under dynamic backgrounds, specifically including the following steps:

[0005] The types of noise in infrared images are analyzed, and filtering algorithms are used to remove noise from the images.

[0006] Different interference corner point filtering methods are adopted for different frames, thereby performing coarse and fine elimination of corner points, and the final detected corner points are stored and recorded;

[0007] The sparse optical flow method is used to track the corner points mentioned above, and the tracking points are filtered out using a bidirectional tracking method. The positional relationship of the corner points in the previous frame in the current frame is determined. Then, the current tracking points are used for reverse tracking, and the two sets of corner points are filtered to remove the corner points that fail to be tracked in reverse.

[0008] The homography matrix is ​​calculated based on the relationship between the corner point sets of the two consecutive frames, and the homography transformation matrix is ​​used to perform background compensation on the current frame image;

[0009] The previous frame infrared image is compared with the current frame image after background compensation. The difference result is subjected to adaptive grayscale threshold binarization. Morphological operations are performed on the binary image to obtain the final target position.

[0010] A mask is formed based on the target position detected in the previous frame, and then fed back to the corner detection in the next frame, forming a complete closed-loop detection circuit.

[0011] Furthermore, a Gaussian filter is used to filter out noise in the current frame of the infrared image, removing Gaussian white noise from the infrared image; a median filter is used to filter out random point noise in the infrared image.

[0012] Furthermore, it is determined whether the current frame is one of the previous five infrared images. If it is, coarse elimination is performed; otherwise, fine elimination is performed.

[0013] The infrared image is divided into sub-blocks of equal size, with sub-block sequence i1 to i... 25 ;

[0014] Starting from i1, the Shi-Tomasi algorithm is used to detect corners in each small block. After detection, the number of corners in each small block is sorted from smallest to largest.

[0015] Remove the 5 largest sub-blocks and the 5 smallest sub-blocks, and store the final corner points.

[0016] If it is determined that the image is not from the previous five frames, the detection result of the previous frame will be fed back to the current frame during detection. The mask generated by the moving target area in the previous frame will be used, and corner detection will not be performed on the current frame for areas with zero mask area.

[0017]

[0018] Shi-Tomasi corner detection is performed on other areas to complete fine elimination, enabling the entire detection method to achieve closed-loop detection;

[0019] The final corner detection results are stored.

[0020] Furthermore, the LK optical flow pyramid algorithm is used to track each corner point in the current frame to obtain the corner point P of the previous frame. i (x0, y0) is at position P in the current frame. i (x1, y1), repeat this step until all corner points have been calculated, and store all tracking points;

[0021] The LK optical flow pyramid algorithm is used again to reverse track the above set of tracking points to obtain the tracking point P in the current frame. i (x1, y1) is at position P in the previous frame. i (x2,y2), repeat this step until all tracking points have been calculated, and store all the tracking point pairs of reverse tracking;

[0022] Remove the positive tracking point pairs based on the output state vector;

[0023] If the output state vector is determined to be 1, the corner point of the previous frame and the corresponding tracking point of the current frame are stored; if the output state vector is determined to be 0, the corresponding point pair is removed.

[0024] Apply the same removal strategy to the set of reverse-tracked data;

[0025] Filter the sets of forward tracking point pairs and reverse tracking point pairs, and select point P. i (x1, y1) compares with P in its corresponding set. i (x0,y0) and P i If the x-coordinates and y-coordinates of two points (x2, y2) are the same, then bidirectional tracking is successful, and P is set to P. i (x0,y0) and P i (x1, y1) is added to the successful tracking set;

[0026] Repeat the above steps until all point pairs have been filtered.

[0027] Furthermore, the PROSAC algorithm is used to calculate the optimal homography transformation matrix H of the two sets of corner points corresponding to the previous frame and the current frame. For the corner point P of the previous frame... i (x0, y0) and the tracking point P corresponding to the current frame i (x1, y1) should satisfy the following relationship:

[0028]

[0029] Background compensation for the current frame is performed using the optimal homography transformation matrix H, and image correction is performed by interpolating in the x and y directions of the pixels using bilinear interpolation.

[0030] Perform a difference operation between the previous frame infrared image and the compensated infrared image, and then apply Gaussian filtering to the difference image to remove noise.

[0031] The Otsu algorithm is used to perform threshold segmentation on the difference image to obtain a binary image that is suspected to be a moving target;

[0032] Erosion is performed on the binary image suspected to be a moving target to remove discrete noise and linear noise interference. Then, dilation is performed. After dilation, small regions are marked and filtered out. Dilation is performed again to obtain the final binary image of the moving target.

[0033] Calculate the contour of the moving target based on the final binary image of the moving target, and store the contour;

[0034] Traverse each target contour and draw a bounding rectangle on the current frame infrared image based on the contour, and store the position, length and width of the rectangle;

[0035] Repeat the above steps until all contours have been traversed to obtain the final moving target detection result image.

[0036] Furthermore, create a single-channel mask image with the same size and type as the current frame's infrared image and set to white.

[0037] Take an unprocessed bounding rectangle from the stored set of rectangles, obtain the position and size of the rectangle, expand the length and width of the rectangle outward by m pixels, map the expanded rectangle onto the mask image, set the gray value of the pixels inside the rectangle in the mask image to 0, add the processed rectangle to the processed set, continue to process the next rectangle, and repeat the above steps until all rectangles have been processed.

[0038] A mask image of a moving target is obtained, and this mask image is used as initial information to feed back into the detection of the next frame to achieve loop closure detection.

[0039] By employing the above technical solutions, this invention provides a method for closed-loop detection of moving targets in infrared video under dynamic backgrounds. In the initial detection stage, corner point homogenization is used, and optical flow tracking is performed on the homogenized corner points to improve the accuracy of the transformation matrix calculation. During the optical flow tracking stage, a bidirectional tracking algorithm is used to remove the influence of erroneous tracking points on background compensation, further improving the accuracy of moving target detection. A mask of the moving object region from the previous frame is used to eliminate the influence of corner points in the current frame's moving object region. Only background points are used to iteratively calculate the optimal homography transformation matrix of the two infrared images, which not only improves the accuracy of background compensation but also enhances the algorithm's efficiency and real-time performance. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of the method disclosed in this invention;

[0042] Figure 2 This is a schematic diagram of the infrared image sequence in this invention;

[0043] Figure 3 This is a schematic diagram of the corner point homogenization image segmentation in this invention;

[0044] Figure 4 This is a schematic diagram of bidirectional tracking of sparse optical flow in this invention;

[0045] Figure 5 This is a schematic diagram of the difference results without background compensation in this invention;

[0046] Figure 6 This is a schematic diagram of the difference results without the use of bidirectional tracking and "fine elimination" in this invention;

[0047] Figure 7 This is a schematic diagram illustrating the difference results using bidirectional tracking and "fine elimination" in this invention;

[0048] Figure 8 This is a schematic diagram illustrating threshold segmentation of the differential results without the use of bidirectional tracking and "fine elimination" in this invention;

[0049] Figure 9 This is a schematic diagram illustrating the threshold segmentation of the difference results using bidirectional tracking and "fine elimination" in this invention;

[0050] Figure 10 Schematic diagram of the final motion area monitoring results;

[0051] Figure 11 This is a schematic diagram of the mask for the moving object region in this invention. Detailed Implementation

[0052] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

[0053] like Figure 1 The method for closed-loop detection of moving targets using infrared video in a dynamic background is shown below. During implementation, the infrared video images acquired by the device are as follows: Figure 2 As shown, corner point homogenization and either "coarse elimination" or "fine elimination" are selected based on the frame number. Then, bidirectional optical flow tracking is used to remove erroneous point pairs. Next, background compensation and image differencing are performed. Finally, a mask is generated based on the moving object region and fed back into the next frame detection, forming a closed-loop detection of infrared moving targets. The specific steps of the method disclosed in this invention are as follows:

[0054] S1: Preprocess the current frame of the infrared video: Based on the analysis of noise types in the infrared image, a filtering algorithm is used to process the image; specifically, the following method is adopted:

[0055] S11: Use a 7*7 Gaussian filter to filter out noise from the current frame of the infrared image, removing Gaussian white noise from the infrared image.

[0056] S12: Next, a 3*3 median filter is used to filter out random point noise in the infrared image. The noise in the processed infrared image is significantly reduced, while the detail information is also well preserved, and the image quality is improved to a certain extent, providing a good foundation for subsequent processing.

[0057] S2: Different interference corner point filtering strategies are adopted for different frames to achieve "coarse elimination" or "fine elimination" of corner points, and the final detected corner points are stored and recorded. Specifically, the following method is used:

[0058] S21: Determine if the current frame is one of the previous five frames. If it is, perform "coarse elimination"; otherwise, perform "fine elimination".

[0059] S22: If S21 determines that it is the first five frames, divide the infrared image into 5*5 equal sub-blocks, and the sub-block sequence i1 to i 25 Infrared image segmentation such as Figure 3 As shown.

[0060] S23: Starting from i1, use the Shi-Tomasi algorithm to detect corners in each small block. After detection, sort the number of corners in each small block from smallest to largest.

[0061] S24: Remove the 5 largest sub-blocks and the 5 smallest sub-blocks, and store the final corner points to make the detected corner points more uniform, thus completing the "coarse elimination".

[0062] S25: If S21 determines that it is not an image from the previous five frames, the detection result of the previous frame will be fed back to the current frame during detection. The mask generated by the moving target area in the previous frame will be used. No corner detection will be performed on the current frame for the areas where the mask area is zero, thus completing the "fine elimination" and the black area is the moving target area.

[0063]

[0064] S26: Perform Shi-Tomasi corner detection on other areas to complete "fine elimination", so that the entire detection method can achieve closed-loop detection.

[0065] S27: Store the final corner detection results.

[0066] S3: The sparse optical flow method is used to track corner points calculated in the previous frame, and bidirectional tracking is used to filter out the tracked points. The positional relationship of the corner points in the previous frame in the current frame is determined. Then, the current tracking points are used for reverse tracking, and the two sets of corner points are filtered to remove corner points that failed to be tracked in reverse. The specific method is as follows:

[0067] S31: Use the LK optical flow pyramid algorithm to track each corner point obtained in S27 in the current frame, and obtain the corner point P of the previous frame. i (x0, y0) is at position P in the current frame. i (x1, y1) In infrared video, since the video changes continuously over time, it is reasonable to assume that many points in the previous frame can be found in the next frame.

[0068] S32: Repeat step S31 until all corner points have been calculated, and store all tracking points.

[0069] S33: Use the LK optical flow pyramid algorithm again to reverse track the tracking point set from S31, and obtain the tracking point P in the current frame. i (x1, y1) is at position P in the previous frame. i (x2,y2).

[0070] S34: Repeat step S33 until all tracking points have been calculated, and store all reverse-tracked tracking point pairs.

[0071] S35: Remove the positive tracking point pairs based on the output state vector.

[0072] S36: If the output state vector is determined to be 1 from S32, then store the corner point of the previous frame and the corresponding tracking point of the current frame. If the output state vector is determined to be 0 from S32, then remove the corresponding point pair.

[0073] S37: Apply the same removal strategy to the set of reverse-tracked data.

[0074] S38: Filter the forward tracking point pair set and the reverse tracking point pair set, and select point P. i (x1, y1) compares with P in its corresponding set. i (x0,y0) and P iIf the x-coordinates and y-coordinates of two points (x2, y2) are the same, then bidirectional tracking is successful. A diagram illustrating bidirectional tracking is shown below. Figure 4 As shown, P i (x0,y0) and P i (x1, y1) is added to the successful tracking set.

[0075] S39: Repeat the above operation until all point pairs have been filtered.

[0076] S4: Calculate the homography matrix based on the relationship between the corner point sets of the two consecutive frames, and use the homography transformation matrix to perform background compensation on the current frame image. Specifically, the following method is used:

[0077] S41: Using the PROSAC algorithm, calculate the optimal homography transformation matrix H of the two sets of corner points corresponding to the previous and current frames obtained in S36. For the corner point P of the previous frame... i (x0, y0) and the tracking point P corresponding to the current frame i (x1, y1) should satisfy the following relationship:

[0078]

[0079] S42: Use the optimal homography transformation matrix H calculated in step S51 to perform background compensation on the current frame. The difference image without background compensation is shown below. Figure 5 As shown, in order to eliminate the error caused by pixel offset as much as possible during the compensation process, bilinear interpolation is used to perform interpolation in the x and y directions of the pixel point to correct the image.

[0080] S5: Difference is performed between the previous frame infrared image and the current frame image after background compensation. The difference result is then subjected to adaptive grayscale threshold binarization. Morphological operations are performed on the binary image to obtain the final target location. Specifically, the following method is used:

[0081] S51: Perform a difference operation between the previous frame infrared image and the compensated infrared image, apply Gaussian filtering to the difference image to remove noise. The background compensation difference image without bidirectional tracking and "fine-tuning" is shown below. Figure 6 As shown, the background compensation difference image undergoes bidirectional tracking and "fine removal" as follows: Figure 7 As shown.

[0082] S52: Use the Otsu algorithm to perform threshold segmentation on the difference image to obtain a binary image of the suspected moving target. The difference threshold segmentation image without bidirectional tracking and "fine-tuning" is shown below. Figure 8 As shown, the image undergoes bidirectional tracking and "fine elimination" differential thresholding segmentation. Figure 9 As shown.

[0083] S53: Perform an erosion operation on the binary image of the suspected moving target to remove discrete noise and linear noise interference, then perform a dilation operation. After dilation, small regions are marked and filtered out, and the dilation operation is performed again to obtain the final binary image of the moving target. The final motion region detection result is as follows: Figure 10 As shown.

[0084] S54: Calculate the contour of the moving target based on the final binary image of the moving target, and store the contour.

[0085] S55: Traverse each target contour, draw an outer rectangle on the current frame infrared image based on the contour, and store the position, length and width of the rectangle.

[0086] S56: Repeat the above steps until all contours have been traversed to obtain the final moving target detection result image.

[0087] S6: A mask is formed based on the target position detected in the previous frame, and fed back to the corner detection in the next frame, forming a complete closed-loop detection system. Specifically, the following method is used:

[0088] S61: Create a single-channel mask initial image with the same size and type as the current frame's infrared image and set to white.

[0089] S62: Take an unprocessed bounding rectangle from the set of rectangles stored in S55, obtain the position and size of the rectangle, expand the length and width of the rectangle outward by m pixels (m can be adjusted according to the actual situation), map the expanded rectangle onto the mask image, set the gray value of the pixels inside the rectangle in the mask image to 0, add the processed rectangle to the processed set, and continue processing the next rectangle.

[0090] S63: Repeat the above steps until all rectangles have been processed.

[0091] S64: After completing the above steps, a mask image of a moving target is obtained. This mask image is used as initial information to feed back into the detection of the next frame to achieve loop closure detection. The mask image is as follows: Figure 11 As shown.

[0092] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting closed loops of moving targets in infrared video under dynamic background, characterized in that... include: The types of noise in infrared images are analyzed, and filtering algorithms are used to remove noise from the images. Different interference corner point filtering methods are adopted for different frames, thereby performing coarse and fine elimination of corner points, and the final detected corner points are stored and recorded; The sparse optical flow method is used to track the corner points mentioned above, and the tracking points are filtered out using a bidirectional tracking method. The positional relationship of the corner points in the previous frame in the current frame is determined. Then, the current tracking points are used for reverse tracking, and the two sets of corner points are filtered to remove the corner points that fail to be tracked in reverse. The homography matrix is ​​calculated based on the relationship between the corner point sets of the two consecutive frames, and the homography transformation matrix is ​​used to perform background compensation on the current frame image; The previous frame infrared image is compared with the current frame image after background compensation. The difference result is subjected to adaptive grayscale threshold binarization. Morphological operations are performed on the binary image to obtain the final target position. A mask is formed based on the target position detected in the previous frame, and then fed back to the corner detection in the next frame, forming a complete closed-loop detection circuit.

2. The method for closed-loop detection of moving targets in infrared video under dynamic background according to claim 1, characterized in that: A Gaussian filter is used to remove noise from the current frame of the infrared image, eliminating Gaussian white noise; a median filter is used to remove random dot noise from the infrared image.

3. The method for closed-loop detection of moving targets in infrared video under dynamic background according to claim 1, characterized in that: Determine if the current frame is one of the previous five infrared images. If it is, perform coarse removal; otherwise, perform fine removal. The infrared image is divided into sub-blocks of equal size, with sub-block sequence i1 to i... 25 ; Starting from i1, the Shi-Tomasi algorithm is used to detect corners in each small block. After detection, the number of corners in each small block is sorted from smallest to largest. Remove the 5 largest sub-blocks and the 5 smallest sub-blocks, and store the final corner points. If it is determined that the image is not from the previous five frames, the detection result of the previous frame will be fed back to the current frame during detection. The mask generated by the moving target area in the previous frame will be used, and corner detection will not be performed on the current frame for areas with zero mask area. Shi-Tomasi corner detection is performed on other areas to complete fine elimination, enabling the entire detection method to achieve closed-loop detection; The final corner detection results are stored.

4. The method for detecting closed-loop motion targets in infrared video under dynamic background according to claim 2, characterized in that: The LK optical flow pyramid algorithm is used to track each corner point in the current frame to obtain the corner point P from the previous frame. i (x0, y0) is at position P in the current frame. i (x1, y1), repeat this step until all corner points have been calculated, and store all tracking points; The LK optical flow pyramid algorithm is used again to reverse track the above set of tracking points to obtain the tracking point P in the current frame. i (x1, y1) is at position P in the previous frame. i (x2,y2), repeat this step until all tracking points have been calculated, and store all the tracking point pairs of reverse tracking; Remove the positive tracking point pairs based on the output state vector; If the output state vector is determined to be 1, the corner point of the previous frame and the corresponding tracking point of the current frame are stored; if the output state vector is determined to be 0, the corresponding point pair is removed. Apply the same removal strategy to the set of reverse-tracked data; Filter the sets of forward tracking point pairs and reverse tracking point pairs, and select point P. i (x1, y1) compares with P in its corresponding set. i (x0,y0) and P i If the x-coordinates and y-coordinates of two points (x2, y2) are the same, then bidirectional tracking is successful, and P is set to P. i (x0,y0) and P i (x1, y1) is added to the successful tracking set; Repeat the above steps until all point pairs have been filtered.

5. The method for detecting closed-loop motion targets in infrared video under dynamic background according to claim 3, characterized in that: The PROSAC algorithm is used to calculate the optimal homography transformation matrix H of the two sets of corner points corresponding to the previous and current frames. For the corner point P of the previous frame... i (x0, y0) and the tracking point P corresponding to the current frame i (x1, y1) should satisfy the following relationship: Background compensation for the current frame is performed using the optimal homography transformation matrix H, and image correction is performed by interpolating in the x and y directions of the pixels using bilinear interpolation.

6. The method for detecting closed-loop motion targets in infrared video under dynamic background according to claim 4, characterized in that: Perform a difference operation between the previous frame infrared image and the compensated infrared image, and then apply Gaussian filtering to the difference image to remove noise. The Otsu algorithm is used to perform threshold segmentation on the difference image to obtain a binary image that is suspected to be a moving target; Erosion is performed on the binary image suspected to be a moving target to remove discrete noise and linear noise interference. Then, dilation is performed. After dilation, small regions are marked and filtered out. Dilation is performed again to obtain the final binary image of the moving target. Calculate the contour of the moving target based on the final binary image of the moving target, and store the contour; Traverse each target contour and draw a bounding rectangle on the current frame infrared image based on the contour, and store the position, length and width of the rectangle; Repeat the above steps until all contours have been traversed to obtain the final moving target detection result image.

7. The method for detecting closed-loop motion targets in infrared video under dynamic background according to claim 1, characterized in that: Create a single-channel mask image with the same size and type as the current frame's infrared image and set to white. Take an unprocessed bounding rectangle from the stored set of rectangles, obtain the position and size of the rectangle, expand the length and width of the rectangle outward by m pixels, map the expanded rectangle onto the mask image, set the gray value of the pixels inside the rectangle in the mask image to 0, add the processed rectangle to the processed set, continue to process the next rectangle, and repeat the above steps until all rectangles have been processed. A mask image of a moving target is obtained, and this mask image is used as initial information to feed back into the detection of the next frame to achieve loop closure detection.

Images