A theodolite stable tracking method based on multi-frame information fusion

Abstract

The present invention relates to a theodolite stable tracking method based on multi-frame information fusion, which belongs to the technical field of photoelectric theodolites and solves the problem of insufficient stability and accuracy of the theodolite during target tracking. The present invention comprises the steps of: extracting a gate image using a centroid tracking method; determining a stable tracking state based on multi-frame information fusion; predicting the target trajectory using a Kalman filter based on a comprehensive angle value for the tracking state to be evaluated in the stable tracking state; converting the predicted comprehensive angle value into a predicted target miss amount, comparing the deviation value between the predicted target miss amount and the actual miss amount with a threshold value, and adaptively correcting the miss amount. The present invention determines the stable tracking state of the theodolite through multi-frame information fusion, and performs trajectory prediction in combination with a Kalman filter. It has the advantages of enhancing tracking robustness and accuracy, achieving adaptive correction of the miss amount, and improving the tracking performance of the theodolite in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of photoelectric theodolites, and in particular to a theodolite stable tracking method based on multi-frame information fusion. Background Art

[0002] Photoelectric theodolites are primarily used for trajectory measurement and live recording of ballistic targets such as missiles and launch vehicles. The observed target is kept in the center of the field of view through the coordinated efforts of the image processing subsystem, which extracts the target miss distance in real time, and the servo subsystem, which controls the drive. However, target tracking against a complex sky background is susceptible to interference from dynamic processes such as cloud movement, bird flight, and target separation. These can lead to anomalies in miss distance extraction and sudden gate changes, compromising the system's automatic tracking and measurement performance and causing the target to move out of the field of view.

[0003] Current photoelectric theodolites primarily consist of an optical subsystem (visible, infrared, etc.), a tracking frame, a servo subsystem, an operation and control subsystem, an image tracking subsystem, an image storage subsystem, a video interpretation subsystem, and a time-based terminal. These subsystems are used to track, measure, and record the movements of flying targets, such as missiles and launch vehicles, along predetermined trajectories. The servo and image tracking subsystems are key components in the target tracking process. During high-speed target tracking, the area pointed by the photoelectric theodolite forms a projected image on the target surface of the detector. The image tracking subsystem extracts the target miss distance in real time and sends it to the servo control system as a basis for controlling the theodolite's motion. The servo subsystem then controls the turntable to track the target while simultaneously capturing the next frame of image. This cycle forms a closed-loop automatic tracking control system. However, image-based automatic tracking strategies are inevitably affected by the accuracy of the tracking algorithm. When the target is located against a complex sky background, the accuracy of the tracking algorithm is easily disrupted by dynamic processes such as cloud movement, bird flight, and target separation, leading to abnormal miss distance extraction, resulting in the target leaving the field of view and tracking measurement failure. Therefore, how to improve the stability and accuracy of target miss distance extraction has always been a key issue that cannot be ignored in the tracking process of photoelectric theodolite.

[0004] There are currently three mainstream solutions to the problem of insufficient stability and accuracy of photoelectric theodolites during target tracking: template matching, trajectory prediction, and deep learning. However, each of these methods has certain limitations:

[0005] Template matching: When tracking an object, the performance of template matching depends significantly on the selection of the initial template and is highly sensitive to variations in the object's appearance and background dynamics. Template matching accuracy is often negatively impacted when the object undergoes transformations such as scaling, rotation, and occlusion.

[0006] Trajectory prediction: This method analyzes the target's miss distance data from previous frames to build a prediction model to estimate the target's position in the next frame. However, this strategy does not fully consider the accuracy of the miss distance data from previous frames. If there are errors in the data from previous frames, these errors will be directly transmitted to the prediction model, affecting the accuracy of the trajectory prediction.

[0007] Deep learning approaches: This technology relies on building large datasets for network training, and its performance is significantly affected by the target's texture and feature information. Especially when processing small infrared targets, detection accuracy often falls short of requirements due to a lack of sufficient texture feature information. Furthermore, the high computational complexity of deep learning algorithms makes achieving high frame rate, real-time processing, challenging. Summary of the Invention

[0008] In view of the above problems, the present invention proposes a theodolite stable tracking method based on multi-frame information fusion. On the basis of the traditional theodolite centroid tracking algorithm, the method performs multi-frame associated multi-modal information statistics, and proposes a tracking state determination method based on the fusion of target grayscale and position information to evaluate the correct credibility of target miss amount extraction. At the same time, a Kalman filter target trajectory prediction method based on comprehensive angle value is proposed, which converts the predicted comprehensive angle value into a predicted target miss amount, and uses the predicted miss amount to correct the actual miss amount, ultimately achieving stable and accurate output of the target miss amount.

[0009] The technical solution adopted by the present invention is as follows:

[0010] A theodolite stable tracking method based on multi-frame information fusion, the method comprising the following steps:

[0011] Step 1: Extract the wave gate image using the centroid tracking method;

[0012] Step 2: Determine the stable tracking state based on multi-frame information fusion, specifically including the following steps:

[0013] Step 2.1: Calculate the target grayscale evaluation value of each frame image and continuously count The average value of the target grayscale evaluation value of the frame image is calculated based on the average value;

[0014] Step 2.2: Calculate the intersection-over-combination ratio of the target detection gate of the current frame image and the target detection gate of the previous frame image;

[0015] Step 2.3: Determine whether the current theodolite is in a stable tracking state. If so, calculate the value of the continuous statistics after the current time. The tracking state of the theodolite is divided into one of a stable tracking state, a tracking state to be evaluated, and an unstable tracking state based on the intersection-over-union ratio of the frame image, the target grayscale evaluation value, and the comprehensive target grayscale evaluation value;

[0016] Step 3: Tracking state to be evaluated in stable tracking state, and predicting target trajectory based on Kalman filter of comprehensive angle value, specifically including the following steps:

[0017] Step 3.1: Calculate the comprehensive angle value of the target based on the azimuth and elevation angles when the theodolite collects the image and the target miss distance obtained after the image is detected;

[0018] Step 3.2: Construct a Kalman filter and use the Kalman filter to predict the trajectory state of the target in the next frame of image, wherein the trajectory state includes the predicted comprehensive angle value of the target;

[0019] Step 4: Convert the predicted comprehensive angle value into the predicted target miss amount, and output the adaptively corrected miss amount by comparing the deviation value between the predicted target miss amount and the actual miss amount detected by the current frame image with the threshold.

[0020] The multi-frame information fusion-based theodolite stable tracking method proposed in the present invention realizes the stable tracking state determination of the theodolite through multi-frame information fusion and combines it with Kalman filtering for trajectory prediction. The method has the following advantages: the robustness and accuracy of theodolite tracking are enhanced, and the tracking strategy is optimized through state classification. Kalman filtering improves the prediction stability and accuracy, realizes the adaptive correction of the miss distance, ensures real-time performance and anti-interference capability, and improves the tracking performance of the photoelectric theodolite in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a theodolite stable tracking method based on multi-frame information fusion according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the comprehensive angle value. DETAILED DESCRIPTION

[0023] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.

[0024] The automatic tracking process of the photoelectric theodolite mainly includes: the image processing system captures the target in the full field of view based on the current frame image received. If the capture is successful, the wave gate is drawn and the miss distance information of the current target is sent to the servo control system. The servo control system drives the theodolite to rotate according to the received miss distance information to perform real-time target tracking and update the next frame image. If the capture fails, the servo control system guides the tracking according to the external guidance data. At this time, the image processing system enters the capture state and continues to capture the target of interest. This cycle forms a closed-loop automatic tracking.

[0025] like Figure 1 As shown, this embodiment provides a theodolite stable tracking method based on multi-frame information fusion, which includes the following steps 1 to 4.

[0026] Step 1: Use the classic centroid tracking method to extract the wave gate image.

[0027] The centroid tracking method is generally implemented based on the threshold segmentation principle, so the selection of the threshold is very important. This embodiment uses the threshold segmentation method based on grayscale statistics to determine the threshold , where the main idea of ​​the threshold segmentation method based on grayscale statistics is expressed as follows:

[0028] ;

[0029] ;

[0030] in, 、 The length and width of the processed image respectively; is the average gray value of the entire image; is the grayscale standard deviation of the entire image; is a constant. Generally, Take 3; is the adjustable threshold deviation value; The target point is located in the original image Row, No. Grayscale value of the column pixels.

[0031] Threshold After confirmation, the gray value is greater than the threshold The pixels with grayscale values ​​less than the threshold are divided into target areas. The pixels are divided into the background area, which is expressed as follows:

[0032] ;

[0033] in, Represents the image after threshold segmentation Row, No. Grayscale value of the column pixel, Represents the original image Row, No. Grayscale value of the column pixels.

[0034] Based on the set of all target pixels obtained from the above classification decision, the center of mass coordinates of the target are calculated as follows:

[0035] ;

[0036] ;

[0037] in: The image after threshold segmentation Row, No. Grayscale value of the column pixel, is the target's centroid row coordinate, The coordinates of the target's center of mass.

[0038] The gate image is extracted based on the target area and the target's centroid coordinates after segmentation. The specific operation is: extract the minimum bounding rectangle of the target area, and the upper left corner coordinates are , the lower right corner coordinates , then the width and height of the gate image are:

[0039] ;

[0040] ;

[0041] in, It is usually a constant. The coordinates of the upper left corner of the gate image are .

[0042] Step 2: Determine the stable tracking state based on multi-frame information fusion, specifically including the following steps 2.1 to 2.3.

[0043] Step 2.1: Calculate the target grayscale evaluation value and the comprehensive target grayscale evaluation value.

[0044] The grayscale of each detected gate image is calculated to obtain the target grayscale evaluation value. If the detection fails, the target grayscale evaluation value is calculated for the center window of half of the image. The calculation formula of the target grayscale evaluation value is as follows:

[0045] ;

[0046] in, is the grayscale mean of the current wave gate image, is the maximum grayscale value of the current wave gate image.

[0047] Continuous Statistics The average value of the target grayscale evaluation value of the frame gate image is used to judge whether the grayscale information evaluation index of the next frame target is successfully detected, that is, according to The comprehensive target grayscale evaluation value is obtained by calculating the average value of the target grayscale evaluation value of the frame wave gate image , the calculation formula is as follows:

[0048] ;

[0049] in, , is an adjustable grayscale threshold.

[0050] Step 2.2: Calculate the intersection-over-union ratio.

[0051] Calculate the intersection-and-union ratio of the target detection gate of the current frame image and the target detection gate of the previous frame image. This is a position information evaluation indicator used to determine whether the target tracking is stable. The calculation formula of the intersection-and-union ratio is as follows:

[0052] ;

[0053] in, and Respectively represent the target detection gate of the current frame and the target detection gate of the previous frame, represents the intersection area of ​​two rectangular wave gate frames, and Represents the union area of ​​two rectangular wave gate frames; Indicates the area of ​​the region.

[0054] Step 2.3: Determine the stable tracking state.

[0055] Determine whether the current theodolite is in a stable tracking state. If it is not in a stable tracking state, determine whether it meets the requirements of three consecutive frames of images. and If the condition is satisfied, the current theodolite is judged to be in a stable tracking state. If not, the current theodolite is judged to be in an unstable tracking state and the target is lost. If the current theodolite is in a stable tracking state, the following state judgment strategy is executed:

[0056] Count the continuous The intersection-over-union ratio of the frame image, and each frame satisfies and When the target gate extraction is stable, the confidence of target extraction is high, and the target miss distance can be output. At this time, the stable tracking state can be entered. Under normal circumstances, 3 is acceptable;

[0057] In the stable tracking state, when When , it is considered that the accuracy of target gate extraction needs to be evaluated and there is a risk of interference from other targets. At this time, the tracking state is entered;

[0058] In the stable tracking state, if , it is believed that the target extraction gate jump is serious and the confidence level of target extraction is low, so the target is determined to be lost and the tracking state is unstable.

[0059] Step 3: Tracking state to be evaluated in stable tracking state, and predicting target trajectory based on Kalman filter of comprehensive angle value, specifically including the following steps 3.1-3.2.

[0060] Step 3.1: Calculate the integrated angle value.

[0061] like Figure 2 As shown in the figure, the azimuth and elevation angles of the theodolite when collecting images are 、 , in the imaging coordinate system with the center of the field of view P, the tracking algorithm captures the target. At this time, the miss distance of the target detection is and , then the azimuth offset of the target relative to the center of the field of view is , the pitch angle offset is , at this time, the comprehensive angle value of the target is:

[0062] ;

[0063] ;

[0064] in, is the pixel size, is the focal length of the system, is the field of view angle occupied by one pixel.

[0065] Step 3.2: Construct a Kalman filter and use it to predict the trajectory state of the target in the next frame of image. The trajectory state includes the predicted comprehensive angle value of the target.

[0066] In general, the flight trajectory of the target in the mission is smooth, so the state estimation and trajectory prediction of the target's comprehensive angle value in the next frame image are performed to assist the target tracking strategy of the theodolite. The present invention uses the Kalman filter of uniform acceleration and linear observation model to predict the trajectory state of the target in the next frame image. The state vector at time It can be modeled as:

[0067] ;

[0068] in, 、 Respectively indicate the target The comprehensive angle value at the moment, 、 The targets are The rate of change of the integrated angle value at a given moment.

[0069] In step 3.2, using the Kalman filter to predict the trajectory state of the target in the next frame of image includes the following steps:

[0070] Step 3.2.1: Determine whether the Kalman filter is in a stable prediction state. If so, execute step 3.2.2; otherwise, execute step 3.2.4.

[0071] In the stable tracking state, the comprehensive angle value of the trajectory predicted by the Kalman filter at the current moment is and The actual comprehensive angle value of the target detection in the current frame at the current moment is and , then the comprehensive angle prediction deviation value is defined as:

[0072] ;

[0073] Continuous Statistics Frame comprehensive angle prediction deviation value, if Frames all meet , it is considered that the Kalman filter is in a stable prediction state. In general, .

[0074] Step 3.2.2: Determine whether the theodolite is in the tracking evaluation state. If so, proceed to step 3.2.3; otherwise, proceed to step 3.2.4.

[0075] Step 3.2.3: Predict the trajectory state of the target in the current frame image, including the predicted comprehensive angle value of the target.

[0076] In the prediction stage, the time update equation is used to utilize the previous frame The comprehensive angle value of the target in the image predicts the current frame The target state of the image, where the time update equation is:

[0077] ;

[0078] ;

[0079] in, is the state transition matrix, is the covariance matrix of the current measurement error predicted based on the covariance of the previous moment, is the system noise, is the measurement error covariance matrix of the optimal estimate of the previous frame.

[0080] Step 3.2.4: Update the Kalman filter.

[0081] In the update phase, the comprehensive angle value of the target in the current frame image is used to correct the predicted state of the Kalman filter system. After correction, the miss distance is output and the next iteration is entered to realize the prediction of the comprehensive angle value of the target in the next frame image. The state update equation is:

[0082] ;

[0083] ;

[0084] ;

[0085] in, is the Kalman gain, is the measurement error, is the observation matrix, is the optimal estimate of the target trajectory state, is the predicted value of the target trajectory state, is the actual measurement value of the target trajectory state, is the identity matrix, is the measurement error covariance matrix.

[0086] Step 4: Convert the predicted comprehensive angle value into the predicted target miss distance and output the adaptively corrected miss distance.

[0087] If the current state is in tracking evaluation, the Kalman filter is used to predict the trajectory of the current frame. The comprehensive angle value of the predicted target is and , the azimuth of the current optical measurement device and pitch angle , the predicted target off-target amount at this time and They are:

[0088] ;

[0089] ;

[0090] The actual miss distance of the target extracted at this time is and , if the deviation between the predicted target miss amount and the actual miss amount detected by the current frame image is less than the threshold, that is , it is considered that the target miss amount detected in the current frame is relatively consistent with the predicted miss amount, and the actual miss amount of the target is output and If the deviation between the predicted target miss amount and the actual miss amount detected by the current frame image is greater than the threshold, then , it is considered that the target miss amount detected in the current frame is significantly different from the predicted target miss amount, and the actual target miss amount detected in the current frame has a large error detection risk, so the predicted target miss amount is output. and , to achieve adaptive correction of output miss distance.

[0091] This paper proposes a theodolite stable tracking method based on multi-frame information fusion. Through multi-frame correlation statistics, the method uses the intersection-over-union ratio of the multi-frame integrated target grayscale evaluation value and the target gate image to determine the theodolite's stable tracking state. The theodolite's tracking state is divided into stable tracking state, tracking state to be evaluated, and unstable tracking state to assess the accuracy and reliability of target miss distance extraction.

[0092] In the stable tracking state, a Kalman filter target trajectory prediction method based on integrated angle values ​​is constructed. This method proposes a strategy for determining the stable prediction state of the target trajectory. In the stable prediction state, the miss distance extracted from the tracking state to be evaluated is analyzed and determined. The predicted integrated angle value is converted into a predicted miss distance. By comparing it with the threshold of the actual miss distance, the output miss distance is adaptively corrected.

[0093] The present invention has the following advantages:

[0094] 1) Compared with the template matching method: the method of the present invention does not rely on the choice of template and will not be affected by the variation of the target appearance;

[0095] 2) Compared with the trajectory prediction method: the Kalman filter constructed by the method of the present invention predicts the target trajectory in a stable tracking state and a stable prediction state, which has a higher prediction accuracy and is more reliable;

[0096] 3) Compared with deep learning methods: the method of the present invention does not require the construction of a large number of data sets, does not rely on the texture feature information of the target, has low computational complexity, and has strong applicability.

[0097] The multi-frame information fusion-based theodolite stable tracking method proposed in the present invention realizes stable tracking state determination through multi-frame information fusion and combines with Kalman filtering for trajectory prediction. The method has the following advantages: the robustness and accuracy of theodolite tracking are enhanced, and the tracking strategy is optimized through state classification. Kalman filtering improves prediction stability and accuracy, realizes adaptive correction of miss distance, ensures real-time performance and anti-interference capability, and improves the tracking performance of photoelectric theodolite in complex environments.

[0098] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A theodolite stable tracking method based on multi-frame information fusion, characterized in that: The following steps are involved: Step 1: Extract the wave gate image using the centroid tracking method; Step 2: Determine the stable tracking state based on multi-frame information fusion, specifically including the following steps: Step 2.1: Calculate the target grayscale evaluation value of each frame image and continuously count The average value of the target grayscale evaluation value of the frame image is calculated based on the average value; Step 2.2: Calculate the intersection-over-combination ratio of the target detection gate of the current frame image and the target detection gate of the previous frame image; Step 2.3: Determine whether the current theodolite is in a stable tracking state. If so, The tracking state of the theodolite is divided into one of a stable tracking state, a tracking state to be evaluated, and an unstable tracking state based on the intersection-over-union ratio of the frame image, the target grayscale evaluation value, and the comprehensive target grayscale evaluation value; Step 3: Tracking state to be evaluated in stable tracking state, and predicting target trajectory based on Kalman filter of comprehensive angle value, specifically including the following steps: Step 3.1: Calculate the comprehensive angle value of the target based on the azimuth and elevation angles when the theodolite collects the image and the target miss distance obtained after the image is detected; Step 3.2: Construct a Kalman filter and use the Kalman filter to predict the trajectory state of the target in the next frame of image, wherein the trajectory state includes the predicted comprehensive angle value of the target; Step 4: Convert the predicted comprehensive angle value into the predicted target miss amount, compare the deviation value between the predicted target miss amount and the actual miss amount detected by the current frame image with the threshold, and output the adaptively corrected miss amount.

2. The method for theodolite stable tracking based on multi-frame information fusion according to claim 1, characterized in that: In step 1, the threshold value in the centroid tracking method The calculation formula is as follows: ； ； in, 、 The length and width of the processed image respectively; is the average gray value of the entire image; is the grayscale standard deviation of the entire image; is a constant; is the adjustable threshold deviation value; The target point is located in the original image Row, No. Grayscale value of the column pixels.

3. A theodolite stable tracking method based on multi-frame information fusion according to claim 1 or 2, characterized in that: In step 2.1, the target grayscale evaluation value is calculated as follows: ； in, is the grayscale mean of the current wave gate image, is the maximum grayscale value of the current wave gate image; The calculation formula of the comprehensive target grayscale evaluation value is as follows: ； in, , is an adjustable grayscale threshold.

4. The method for theodolite stable tracking based on multi-frame information fusion according to claim 1 or 2, characterized in that: In step 2.2, the calculation formula for the intersection-over-union ratio is as follows; ； in, and Represent the target detection gate of the current frame image and the target detection gate of the previous frame image respectively, Represents the intersection area of ​​two rectangular wave gate frames, Represents the union area of ​​two rectangular wave gate frames; Indicates the area of ​​the region.

5. The method for theodolite stable tracking based on multi-frame information fusion according to claim 1 or 2, characterized in that: In step 3.1, the calculation formula of the comprehensive angle value of the target is as follows: ； ； in, is the pixel size, is the system focal length, is the field of view angle occupied by one pixel; and are the azimuth and elevation angles of the theodolite respectively; and are the azimuth offset and elevation offset of the target relative to the center of the field of view, respectively.

6. The method for theodolite stable tracking based on multi-frame information fusion according to claim 1 or 2, characterized in that: In step 3.2, using the Kalman filter to predict the trajectory state of the target in the next frame of image includes the following steps: Step 3.2.1: Determine whether the Kalman filter is in a stable prediction state. If so, proceed to step 3.2.2; otherwise, proceed to step 3.2.

4. Step 3.2.2: Determine whether the theodolite is in the tracking evaluation state. If so, proceed to step 3.2.3; otherwise, proceed to step 3.2.

4. Step 3.2.3: Prediction phase: Use the time update equation to use the comprehensive angle value of the target in the previous frame image to predict the trajectory state of the target in the current frame image; Step 3.2.4: Update phase: Use the comprehensive angle value of the target in the current frame image to correct the predicted state of the system and enter the next iteration to realize the prediction of the comprehensive angle value of the target in the next frame image.

7. The method for theodolite stable tracking based on multi-frame information fusion according to claim 6, characterized in that: In step 3.2.1, determine whether the Kalman filter is in a stable prediction state by the following method: The comprehensive angle prediction deviation value is defined as: ； in, and are the true comprehensive angle values ​​of the target detection in the current frame, and are the comprehensive angle values ​​obtained by trajectory prediction through the Kalman filter at the current moment; Continuous Statistics The comprehensive angle prediction deviation value of the frame image, if continuous Frame image All conditions are met: , then it is determined that the Kalman filter is in a stable prediction state.

8. The method for theodolite stable tracking based on multi-frame information fusion according to claim 6, characterized in that: The time update equation in step 3.2.3 is: ； ； in, is the optimal estimate of the target trajectory state, is the state transition matrix, is the covariance matrix of the current measurement error predicted based on the covariance of the previous moment, is the system noise, is the measurement error covariance matrix of the optimal estimate of the previous frame.

9. The method for theodolite stable tracking based on multi-frame information fusion according to claim 6, characterized in that: The state update equation in step 3.2.4 is: ； ； ； in, is the Kalman gain, is the measurement error, is the observation matrix, is the optimal estimate of the target’s trajectory state, is the predicted value of the target trajectory state, is the actual measurement value of the target trajectory state, is the identity matrix, is the measurement error covariance matrix, is the covariance matrix of the measurement error at the current moment predicted based on the covariance at the previous moment.

10. The theodolite stable tracking method based on multi-frame information fusion according to claim 1 or 2, characterized in that: In step 4, the predicted integrated angle value is converted to the predicted target miss distance by the following formula: ； ； in, is the pixel size, is the focal length of the system, and is the predicted comprehensive angle value, and is the azimuth and elevation angle of the current theodolite; If the deviation between the predicted target miss distance and the actual miss distance detected by the current frame image is Less than threshold , then the actual target miss distance is output and ; If the deviation between the predicted target miss distance and the actual miss distance detected by the current frame image is Greater than threshold , then the predicted target miss amount is output and .

Images