Compensation method suitable for moving target tracking image delay

Abstract

The invention discloses a compensation method suitable for moving target tracking image delay, and the method comprises the steps: Step 1, building a tracked target template sequence through a video tracker, namely, calculating the pixel features of a target in a tracking frame region range; step 2, the video tracker dynamically stores the graphic template with the aiming line as the center; 3, calling a graphic template stored by a video tracker according to the delay of the video stream; and Step 4, during the working period of the video tracker, generating a graphic template by using the video tracker, and dynamically storing a graphic template sequence in real time so as to be called when a moving target is captured, thereby providing an accurate template reference for continuously tracking the target. By introducing image frame delay compensation, the success rate of capturing and tracking a moving target by a photoelectric searching and tracking system is greatly improved, a man-machine efficacy mechanism is optimized, an operator can easily complete the moving target tracking process, and the market competitiveness of a product assembled with the design is improved.

Description

Technical Field

[0001] This invention belongs to the field of airborne optoelectronic search and tracking technology, and relates to a method for image delay compensation suitable for capturing moving targets and stabilizing the tracking image. The method mainly involves the video tracker canceling the delay generated by the input optical sensor video through parameter calculation, image conversion and video output to the integrated display, so that the control handle can accurately capture the target, and can realize the stable tracking of moving targets by the optoelectronic search and tracking system during aircraft flight. Background Technology

[0002] The technique of capturing moving targets and compensating for image delay in stable tracking is not widely used in airborne optoelectronic search and tracking systems. Typically, if the video tracker processes the input video signal quickly and there is no format conversion required for the video output to the host computer, a small video delay is insufficient to affect the control handle's capture of the moving target. Currently, the tracking algorithms implemented in airborne optoelectronic search and tracking systems are usually used to optimize the target's pixel features to improve tracking stability; or to implement memory tracking algorithms when the moving target is occluded, such as "Meanshift Tracking Algorithm Based on Adaptive Kalman Filter" published in *Manufacturing Automation*, Vol. 43, 2021; "Research on Video Moving Target Algorithm Based on Kalman Filter" published in *Information and Computer*, Vol. 339, 2015; and "Research on Video Moving Target Trajectory Tracking Based on State Dependency Detection" published in *Modern Electronics Technology*, Vol. 39, 2016. No applications related to accurate capture of moving targets before tracking were found in optoelectronic search and tracking systems.

[0003] In photoelectric search and tracking systems, to achieve accurate capture and tracking, the process of pressing the tracking button on the control handle, processing video images, calculating tracking deviations, and outputting the tracking frame involves video delay. If the delay is too large, it can cause the moving target to "escape" from the effective area of ​​the tracking frame, resulting in tracking failure. Previous photoelectric search and tracking systems primarily addressed situations where the target had already been captured but was obscured by a stationary or moving background during tracking, or where the target suddenly accelerated or decelerated, causing tracking instability or entering a memory tracking state where the target could not be recaptured. Examples include the article "A Review of Video Tracking Algorithms" published in *Computer Science*, Volume 40, 2013.

[0004] This patent proposes a method for image delay compensation to achieve stable tracking of moving targets. This method can effectively enable photoelectric search and tracking systems to accurately capture moving targets during aircraft flight and achieve tracking functionality. The method mainly utilizes the video delay characteristics of the tracker and applies an algorithm. Its algorithmic form is simple and can effectively improve the tracking performance of photoelectric search and tracking systems for moving targets.

[0005] No similar patents or documents were found at present. Summary of the Invention

[0006] (I) Purpose of the Invention

[0007] The purpose of this invention is to provide a method for compensating for image delay in moving target tracking. This method addresses the video delay output from the video conversion and tracking parameter calculation processes of the video tracker module to the integrated display on the carrier aircraft, thereby achieving accurate capture of moving targets and stable tracking.

[0008] (II) Technical Solution

[0009] To address the aforementioned technical problems, this invention provides a method for compensating for image delay in moving target tracking, comprising the following steps:

[0010] Step 1: The video tracker establishes a template sequence for the tracked target. This involves calculating the pixel features of the target within the tracking bounding box region. The specific method is as follows:

[0011] Step 1.1: Construct a target feature region at the center of the crosshair of the aiming line. This region is the same size as the tracking box. The video tracker calculates and generates a real-time graphic template in the feature region based on the features of the tracked target. The center of this graphic template always coincides with the center of the crosshair of the aiming line.

[0012] Step 1.2: When the tracked target moves, the operator uses the control handle to attach the aiming line "+" to the moving target in real time. After acquiring the pixel features in the target feature area, the video tracker performs statistical reconstruction based on each frame of image input from the optical sensor (such as a visible light camera, infrared imager, etc.). The so-called statistical reconstruction is that the video tracker performs statistical learning and reconstructs the graphic template based on the graphic information near the aiming line "+". The identifiable range of statistical reconstruction is a local area that is 2 to 3 times the horizontal and vertical dimensions of the tracking frame.

[0013] Step 1.3: The statistical reconstruction process is defined as Size Invariant Feature Transform (SIFT). For each frame of image input from the optical sensor, the video tracker uses the SIFT detection method to identify the graphic features within the tracking box near the aiming line and extract local appearance feature descriptors. Within the field of view of the optical sensor, due to the time delay characteristics of the video stream, the target observed moving near the crosshair of the aiming line has actually left the tracking box area, as shown in the attached figure. Figure 2As shown. In reality, the aiming line moves in real time during the system's target acquisition and tracking process, and the area (tracking box) identified by the tracker for the target must also follow the aiming line. At this time, the video tracker needs to calculate and update the pixel offset of the center of the aiming line in real time. The calculation is based on the pixel movement in two dimensions: azimuth and elevation of the system's aiming line. A Toplitz matrix a = [a1, a2, ..., a...] is defined. n ] T (a1,a2,…,a n (For appearance feature descriptors) as basis vectors and a permutation matrix The coordinate matrix for each unit leftward movement of the aiming line's azimuth direction can be represented as: A az =a T ×P=[a2,a3,···,a n [,a1];The coordinate matrix for each unit downward movement of the aiming line in the elevation direction can be represented as: A el =P×a=[a n ,a1,a2,···,a n-1 From this, we can obtain the cyclic matrix of pixel movement:

[0014] The video tracker statistically reconstructs the tracking graphic template in the center region of the aiming line for each frame of the image.

[0015] Step 2: The video tracker dynamically stores graphic templates centered on the aiming line. For each frame of image input from the optical sensor, the video tracker stores the graphic template near the aiming line once. The number of usable graphic templates in the system is set to m, and the number of graphic templates stored by the video tracker is F = (f0, f1, ... f...). m-2 ,f m-1 (Note: f) m-1 f m-2 These are the graphic templates for the current frame and the previous frame, respectively, and so on. Each time a frame of video is refreshed, the graphic template for the current frame is stored, while the previously stored graphic template f0 for the first frame is discarded. At this point, the graphic template stored by the video tracker is: F = (f1, f2, ..., f0). m ).

[0016] Step 3: Retrieve the graphic template stored in the video tracker based on the video stream delay. During the process of the optical sensor generating video and outputting it to the integrated display on the aircraft, there is a delay in the video tracker's image processing and transmission. That is, when capturing a moving target, the "target" at the crosshair in the integrated display's output image has already deviated from the crosshair in the actual scene. This "target" image information is the graphic template from the video tracker n frames ago. Based on the inherent delay characteristics of the video tracker and through testing, the value of n can be easily determined. When the system tracks a moving target, the crosshair is actively aligned with the target, and the tracking button is pressed. The video tracker's tracking frame retrieves the graphic template from the nth frame image, based on the current frame, to achieve accurate capture.

[0017] Step 4: In short, during the operation of the video tracker, a graphic template is generated using the video tracker and the graphic template sequence is dynamically stored in real time for retrieval when capturing moving targets, providing an accurate template reference for continuous target tracking.

[0018] (III) Beneficial Effects

[0019] The above-mentioned technical solution provides a method for compensating for image delay in moving target tracking. By introducing an image frame delay compensation software algorithm, it greatly improves the success rate of photoelectric search and tracking systems in capturing and tracking moving targets. It also optimizes the human-machine interface mechanism, making it easier for operators to complete the moving target tracking process and enhancing the market competitiveness of products equipped with this design. Attached Figure Description

[0020] Figure 1 This diagram illustrates the capture and tracking of a moving target by an optoelectronic search and tracking system.

[0021] Figure 2 This is a diagram illustrating how video stream delays can cause the tracking box to fail to capture a moving target.

[0022] Figure 3 This is a schematic diagram of video stream control during the tracking process. Detailed Implementation

[0023] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0024] This invention discloses an image delay compensation method suitable for capturing and stably tracking moving targets. The key is that the video tracker compensates for the delay generated during image conversion, tracking parameter calculation, and video output to the integrated display from the input optical sensor video, enabling the control handle to accurately capture the target. By addressing the delay in video output from the video tracker module's video conversion and tracking parameter calculation to the carrier's integrated display, accurate capture of moving targets and stable tracking are achieved. The specific implementation steps are as follows:

[0025] Step 1: The video tracker establishes a template sequence for the tracked target. This involves calculating the pixel features of the target within the tracking bounding box region. The specific method is as follows:

[0026] Step 1.1: Construct a target feature region at the center of the crosshair of the aiming line. This region is the same size as the tracking box. The video tracker calculates and generates a real-time graphic template in the feature region based on the features of the tracked target. The center of this graphic template always coincides with the center of the crosshair of the aiming line.

[0027] Step 1.2: When the tracked target moves, the operator uses the control handle to attach the aiming line "+" to the moving target in real time. After acquiring the pixel features in the target feature area, the video tracker performs statistical reconstruction based on each frame of image input from the optical sensor (such as a visible light camera, infrared imager, etc.). The so-called statistical reconstruction is that the video tracker performs statistical learning and reconstructs the graphic template based on the graphic information near the aiming line "+". The identifiable range of statistical reconstruction is a local area that is 2 to 3 times the horizontal and vertical dimensions of the tracking frame.

[0028] Step 1.3: The statistical reconstruction process is defined as Size Invariant Feature Transform (SIFT). For each frame of image input from the optical sensor, the video tracker uses the SIFT detection method to identify the graphic features within the tracking box near the aiming line and extract local appearance feature descriptors. Within the field of view of the optical sensor, due to the time delay characteristics of the video stream, the target observed moving near the crosshair of the aiming line has actually left the tracking box area, as shown in the attached figure. Figure 2 As shown. In reality, the aiming line moves in real time during the system's target acquisition and tracking process, and the area (tracking box) identified by the tracker for the target must also follow the aiming line. At this time, the video tracker needs to calculate and update the pixel offset of the center of the aiming line in real time. The calculation is based on the pixel movement in two dimensions: azimuth and elevation of the system's aiming line. A Toplitz matrix a = [a1, a2, ..., a...] is defined. n ] T (a1,a2,…,a n (For appearance feature descriptors) as basis vectors and a permutation matrix The coordinate matrix for each unit leftward movement of the aiming line's azimuth direction can be represented as: A az =a T ×P=[a2,a3,···,a n [,a1];The coordinate matrix for each unit downward movement of the aiming line in the elevation direction can be represented as: A el =P×a=[a n ,a1,a2,···,a n-1 From this, we can obtain the cyclic matrix of pixel movement:

[0029] The video tracker statistically reconstructs the tracking graphic template in the center region of the aiming line for each frame of the image.

[0030] Step 2: The video tracker dynamically stores graphic templates centered on the aiming line. For each frame of image input from the optical sensor, the video tracker stores the graphic template near the aiming line once. The number of usable graphic templates in the system is set to m, and the number of graphic templates stored by the video tracker is F = (f0, f1, ... f...). m-2 ,f m-1 (Note: f) m-1 f m-2 These are the graphic templates for the current frame and the previous frame, respectively, and so on. Each time a frame of video is refreshed, the graphic template for the current frame is stored, while the previously stored graphic template f0 for the first frame is discarded. At this point, the graphic template stored by the video tracker is: F = (f1, f2, ..., f0). m ).

[0031] In this embodiment, the system video stream is delayed by 6 frames from the optical sensor to the integrated display. Therefore, the graphic template that the video tracker needs to store is: F = (f0, f1, f2, f3, f4, f5), where f5 is the graphic template at the current moment and f0 is the graphic template of the first frame before this moment.

[0032] Step 3: Retrieve the graphic template stored in the video tracker based on the video stream delay. During the process of the optical sensor generating video and outputting it to the onboard integrated display, the video stream... Figure 3As shown, there is a delay in the processing and transmission of images by the video tracker. That is, when capturing a moving target, the "target" at the crosshair in the overall display output image has already deviated from the crosshair in the actual scene. This "target" image information is the graphic template from n frames ago in the video tracker. Based on the inherent delay characteristics of the video tracker and through test experiments, the value of n can be easily obtained. When the system tracks a moving target, the crosshair is actively aligned with the target and the tracking button is pressed. The video tracker's tracking frame retrieves the graphic template from the nth frame image based on the current frame, thus achieving accurate capture.

[0033] When the system tracks a moving target, the aiming line is actively aligned with the target in a crosshair and the tracking button is pressed. If the video tracker does not retrieve a suitable target graphic template at this moment, the tracking frame will largely detach from the moving target, resulting in a "target lost" state. The root cause of this "lost" moving target is the delay in the video output from the optical sensor to the video tracker and finally to the integrated display at the moment of active target acquisition. The moving target displayed on the integrated display lags behind the actual moving target's trajectory in the time domain. If the optical sensor (30Hz frame rate, 33.3ms per frame) itself has a 1-frame delay, and the video tracker has a 5-frame delay, the calculated video stream delay is nearly 200ms. If a target (car) moving at 60 km / h moves laterally in the scene, the video stream delay results in an instantaneous capture of the scene 3.3 meters behind the target. The video processor cannot recognize the graphic template at this location and therefore cannot achieve normal tracking. Therefore, a graphic template centered on the crosshair of the aiming line is stored in real time. When the tracking button is activated to capture the moving target, the video tracker retrieves the image information of the 5th frame before that moment as the graphic template for capture to enter the tracking process.

[0034] Experimental tests showed that the image delay compensation algorithm significantly outperformed conventional capture methods in terms of target acquisition and tracking success rate. (See Table 1 for details.)

[0035] Table 1 Comparison of Delay Compensation Algorithm and Conventional Acquisition and Tracking Results Table 1 Comparison of Delay Compensation Algorithm and Conventional Acquisition and Tracking Results

[0036]

[0037] Note: Target distance 3.5 km, vehicle speed 60 km / h, number of successful captures per 100 attempts.

[0038] Step 4: In short, during the operation of the video tracker, a graphic template is generated using the video tracker and the graphic template sequence is dynamically stored in real time for retrieval when capturing moving targets, providing an accurate template reference for continuous target tracking.

[0039] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A compensation method for image lag in moving target tracking, characterized in that, The method comprises the following steps: Step 1: the video tracker establishes a template sequence of the tracked target, i.e. calculates the pixel features of the target in the tracking frame area; Step 2: the video tracker dynamically stores the graphic template centered on the aiming line; Step 3: the video tracker retrieves the graphic template stored by the video tracker according to the delay of the video stream; Step 4: during the operation of the video tracker, the video tracker generates a graphic template and dynamically stores the graphic template sequence in real time, so as to be ready for retrieval when the moving target is captured, and to provide an accurate template reference for continuous tracking of the target.

2. The method for compensating for image lag suitable for moving target tracking according to claim 1, wherein, Step 1 comprises the following sub-steps: Step 1.1: a target feature area is constructed at the center of the aiming line "cross", the area is consistent with the size of the tracking frame, and the video tracker generates a real-time graphic template according to the features of the tracked target in the feature area, the center position of the graphic template is always coincident with the center of the aiming line "cross"; Step 1.2: when the tracked target moves, the operator controls the handle to make the aiming line "cross" adhere to the moving target in real time, and after the pixel features in the target feature area are obtained, the video tracker reconstructs according to each frame of image input by the optical sensor; Step 1.3: the process of statistical reconstruction is defined as SIFT (Scale-Invariant Feature Transform), the video tracker uses the SIFT detection method to identify the graphic features in the tracking frame area near the aiming line, and extracts the local appearance feature descriptor; in the field of view of the optical sensor, the system captures and tracks the real-time moving aiming line of the target, the tracking area of the target is followed by the aiming line, and the video tracker calculates and updates the pixel offset of the center of the aiming line in real time.

3. The method for compensating for image lag suitable for moving target tracking according to claim 2, wherein, In Step 1.2, the optical sensor is a visible light camera or an infrared imager.

4. The method for compensating for image lag suitable for moving target tracking according to claim 3, wherein, In Step 1.2, during statistical reconstruction, the video tracker statistically learns and reconstructs the graphic template from the graphic information near the aiming line "cross", and the identifiable range is 2-3 times the local area of the tracking frame in the horizontal and vertical directions.

5. The method for compensating for image lag suitable for moving target tracking according to claim 4, wherein, In Step 1.3, the pixel offset of the aiming line center is calculated based on the pixel movement in both the azimuth and elevation dimensions of the system's aiming line. A Toplitz matrix a = [a1, a2, ..., a...] is defined. n ] T As a basis vector and a permutation matrix The coordinate matrix for each unit leftward movement of the aiming line azimuth direction is represented as: A az =a T ×P=[a2,a3,···,a n [,a1]; The coordinate matrix for each unit downward movement of the aiming line in the elevation direction is represented as: A el =P×a=[a n ,a1,a2,···,a n-1 This yields the cyclic matrix of pixel movement: The video tracker counts and reconstructs a tracking pattern template in the center region of the aiming line for each frame of image; wherein a1, a2, …, a n is an appearance feature descriptor.

6. The method for compensating for image lag suitable for moving target tracking according to claim 5, wherein, In Step 2, the optical sensor inputs every 1 frame image, the video tracker stores 1 time pattern near the aiming line, sets the number of available pattern as m, the video tracker stores the pattern F=(f0, f1, … f m-2 , m-1 f m-1 , m-2 f m , respectively, the current frame, the previous frame pattern, every 1 frame refresh, the current frame pattern is stored at the same time, the first frame pattern f0 stored before is discarded, the video tracker stores the pattern F=(f1, f2, …, f m ).

7. The method for compensating for image lag suitable for moving target tracking according to claim 6, wherein, In Step 3, during the process of generating video by the optical sensor and outputting video to the integrated display of the carrier, there is a delay in the processing and transmission of the image by the video tracker, and when the moving target is captured, the "target" at the aiming line "cross" in the image output by the integrated display deviates from the aiming line in the actual scene, and the image information of the "target" is the graphic template of n frames before the current frame; When the system tracks the moving target, the aiming line "cross" is actively aligned with the target and the tracking key is pressed, the video tracker retrieves the graphic template of the n-th frame of image based on the current frame, and accurate capture is achieved.

8. The method for compensating for image lag suitable for moving target tracking according to claim 7, wherein, In Step 3, when the tracking frame is separated from the moving target and the target is lost, image delay compensation is added to capture the target and implement tracking.

9. The method for compensating for image lag suitable for moving target tracking according to claim 8, wherein, In step 3, the optical sensor itself delays 1 frame of picture, the video tracker delays 5 frames, and the motion target is captured, the 5th frame of picture information before this moment is called by the video tracker as a graphic template to enter the tracking process.

10. Application of the compensation method for image delay suitable for moving target tracking according to any one of claims 1-8 in the field of airborne photoelectric search and tracking technology.

Images