Visual tracking fusion compensation method and system based on external information of unmanned aerial vehicle nacelle

Abstract

The application belongs to the technical field of image processing, and particularly relates to a visual tracking fusion compensation method and system based on unmanned aerial vehicle pod exogenous information. The method comprises the following steps: receiving an image frame, a target candidate frame, a visual confidence and pod exogenous information; uniformly converting the candidate frame into a four-dimensional visual observation vector; based on the integrity of the exogenous information and the geometric solving effectiveness, adaptively selecting an angle domain model or a Cartesian domain model to perform target state prediction and mapping; in the unified visual observation space, fusing the motion consistency and the visual confidence to score and associate multiple candidates, selecting the best candidate to update the state or keeping the prediction; and finally, controlling the output of the tracking result according to the stable release condition. The application decouples the platform and the target motion through the exogenous information, adaptively switches the double models to ensure the tracking continuity, and significantly enhances the visual tracking stability and reliability of the unmanned aerial vehicle pod in complex scenes such as strong disturbance and multiple candidates.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a visual tracking fusion compensation method and system based on external source information from a UAV pod. Background Technology

[0002] In UAV pod tracking scenarios, the target displacement in an image is typically formed by the superposition of two parts: the target's actual motion relative to the ground, and the field-of-view disturbance caused by changes in the attitude of the UAV platform and pod. Most existing visual tracking methods directly update and filter the target bounding box within the image pixel plane. While simple to implement, this approach is prone to failure in the following scenarios:

[0003] 1) When the pod rotates rapidly or the platform maneuvers, the target's displacement in the image contains a large number of pseudo-motions introduced by changes in viewpoint, causing the pure image domain motion model to fail.

[0004] 2) The visual results of a single frame are usually not unique. There are multiple candidate boxes, occlusion, jumps, short-term loss and false detections, etc. It is difficult to reliably complete the association by relying solely on visual confidence.

[0005] 3) When it is necessary to publish results to downstream control, instruction or decision modules, continuous frame-by-frame output often contains jitter and error abrupt changes, which is not conducive to engineering applications;

[0006] 4) In some scenarios, the external information is relatively complete, which can support physical space back projection and modeling. However, existing methods lack a mechanism for adaptive switching between high-precision physical modeling and low-priority tracking.

[0007] Therefore, a visual tracking fusion compensation scheme is needed that can make full use of external information from the pod, elevate target observation from the pixel domain to a more stable modeling space, and maintain continuous and robust output under conditions of multiple candidates, strong perturbations, and incomplete priors. Summary of the Invention

[0008] To address the aforementioned issues, this application provides a visual tracking fusion compensation method and system based on external information from an unmanned aerial vehicle (UAV) pod.

[0009] The first aspect of this application provides a visual tracking fusion compensation method based on external information from an unmanned aerial vehicle (UAV) pod, mainly including:

[0010] Step S1: Receive the current image frame, at least one target candidate box and its visual confidence output by the front-end visual tracker, and time-aligned pod external source information;

[0011] Step S2: Preprocess the external source information of the pod, and establish the camera model of the current frame based on the preprocessed information;

[0012] Step S3: Convert each of the target candidate boxes into a four-dimensional visual observation vector. The visual observation vector includes at least the center position, aspect ratio, and scale information of the target candidate box.

[0013] Step S4: Based on the completeness of the external source information of the pod and the effectiveness of the geometric solution, adaptively select the angle domain model or the Cartesian domain model to predict the target state;

[0014] Specifically, if the angle domain model is selected, the target candidate box is back-projected and transformed to the reference attitude coordinate system to obtain the target's azimuth, pitch, and angular scale measurements, and state prediction is performed in the angle domain state space; if the Cartesian domain model is selected, the target candidate box is back-projected and intersected with the target height plane to obtain the target's physical position and physical size measurements, and state prediction is performed in the Cartesian domain state space.

[0015] Step S5: Map the state prediction results of the selected model to a unified visual observation space to obtain the predicted observations;

[0016] Step S6: Calculate the motion consistency metric between the visual observation and the predicted observation of each candidate box, and fuse the motion consistency metric with the corresponding visual confidence score to obtain the comprehensive score of each candidate box;

[0017] Step S7: Select the candidate box with the highest comprehensive score as the best candidate box, and update the state of the selected model using the best candidate box when it meets the update conditions; otherwise, only retain the state prediction results.

[0018] Step S8: Based on the updated or predicted target state, generate the target tracking result for the current frame, and based on preset stable release conditions, decide whether to release the target tracking result to the outside world.

[0019] Preferably, the external information of the pod includes camera intrinsic parameters, pod position, pod attitude, camera field of view, and prior information on target height.

[0020] Preferably, the preprocessing of the external information of the pod includes at least one of temporal smoothing, filtering, interpolation, temporal alignment, outlier removal, or sensor fusion.

[0021] Preferably, the step of adaptively selecting the angle domain model or the Cartesian domain model includes:

[0022] Determine whether the external source information of the pod in the current frame is complete, and whether the current line of sight can form a valid intersection with the target plane determined a priori based on the target height;

[0023] If the above two conditions are met, the Cartesian domain model is selected; otherwise, the angular domain model is selected.

[0024] Preferably, in step S6, the first step is calculated using the following formula. The overall score of each candidate box :

[0025] ;

[0026] in, To integrate weights, For the first Visual confidence of each candidate bounding box For the first A measure of motion consistency between visual observations of candidate boxes and predicted observations. This is the consistency threshold. The visual magnification index is a high-resolution visual enhancement index.

[0027] in, ; For the first Innovation vector of each candidate box For the first Innovation covariance of each candidate box;

[0028] in, ; ;

[0029] in, For the first Visual observation of candidate boxes, The mean of the predicted observations is obtained by mapping the state prediction results to a unified visual observation space. For visual observation noise covariance, To predict the observation noise covariance.

[0030] Preferably, in step S8, the decision on whether to release based on preset stable release conditions includes:

[0031] Statistical length is The percentage of frames that meet the high-quality criteria within the sliding window. ;

[0032] Stability tracking results will be released to the public when all of the following conditions are met:

[0033] Window length reaches The time interval since the last release is not less than the set value. ;

[0034] The proportion of high-quality conditions Not lower than the preset threshold .

[0035] The second aspect of this application provides a visual tracking fusion compensation system based on external information from an unmanned aerial vehicle (UAV) pod, mainly comprising:

[0036] The parameter acquisition module is used to receive the current image frame, at least one target candidate box and its visual confidence output by the front-end visual tracker, and time-aligned pod external source information;

[0037] The pod external source information preprocessing module is used to preprocess the pod external source information and establish a camera model for the current frame based on the preprocessed information.

[0038] The target parameter unification module is used to convert each of the target candidate boxes into a four-dimensional visual observation vector, wherein the visual observation vector includes at least the center position, aspect ratio and scale information of the target candidate box;

[0039] The algorithm adaptive selection module is used to adaptively select either the angle domain model or the Cartesian domain model for target state prediction based on the completeness of the external source information of the pod and the effectiveness of the geometric solution.

[0040] Specifically, if the angle domain model is selected, the target candidate box is back-projected and transformed to the reference attitude coordinate system to obtain the target's azimuth, pitch, and angular scale measurements, and state prediction is performed in the angle domain state space; if the Cartesian domain model is selected, the target candidate box is back-projected and intersected with the target height plane to obtain the target's physical position and physical size measurements, and state prediction is performed in the Cartesian domain state space.

[0041] The predictive observation acquisition module is used to map the state prediction results of the selected model to a unified visual observation space to obtain predictive observations.

[0042] The candidate box comprehensive score calculation module is used to calculate the motion consistency metric between the visual observation and the predicted observation of each candidate box, and to fuse the motion consistency metric with the corresponding visual confidence to obtain the comprehensive score of each candidate box.

[0043] The model state update module is used to select the candidate box with the highest comprehensive score as the best candidate box, and update the state of the selected model using the best candidate box when it meets the update conditions; otherwise, only the state prediction result is retained.

[0044] The target tracking result publishing module is used to generate the target tracking result of the current frame based on the updated or predicted target state, and decide whether to publish the target tracking result based on preset stable publishing conditions.

[0045] Preferably, the external information of the pod includes camera intrinsic parameters, pod position, pod attitude, camera field of view, and prior information on target height.

[0046] Preferably, the step of adaptively selecting the angle domain model or the Cartesian domain model includes:

[0047] Determine whether the external source information of the pod in the current frame is complete, and whether the current line of sight can form a valid intersection with the target plane determined a priori based on the target height;

[0048] If the above two conditions are met, the Cartesian domain model is selected; otherwise, the angular domain model is selected.

[0049] Preferably, in the candidate box comprehensive score calculation module, the first score is calculated using the following formula. The overall score of each candidate box :

[0050] ;

[0051] in, To integrate weights, For the first Visual confidence of each candidate bounding box For the first A measure of motion consistency between visual observations of candidate boxes and predicted observations. This is the consistency threshold. Visual high-resolution magnification index;

[0052] in, ; For the first Innovation vector of each candidate box For the first Innovation covariance of each candidate box;

[0053] in, ; ;

[0054] in, For the first Visual observation of candidate boxes, The mean of the predicted observations is obtained by mapping the state prediction results to a unified visual observation space. For visual observation noise covariance, To predict the observation noise covariance.

[0055] This application effectively decouples platform motion from the real motion of the target by fusing external information from the pod and using adaptive switching between angle domain and Cartesian domain dual models. Even under conditions of multiple candidates, strong disturbances, and incomplete prior knowledge, it can still achieve stable, continuous, and high-quality target tracking and result output. Attached Figure Description

[0056] Figure 1 This is a flowchart of a preferred embodiment of the visual tracking fusion compensation method based on external information from an unmanned aerial vehicle (UAV) pod, as described in this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of this application, not all of them. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0058] The first aspect of this application provides a visual tracking fusion compensation method based on external information from an unmanned aerial vehicle (UAV) pod, mainly including:

[0059] Step S1: Receive the current image frame, at least one target candidate box and its visual confidence output by the front-end visual tracker, and time-aligned pod external source information;

[0060] Step S2: Preprocess the external source information of the pod, and establish the camera model of the current frame based on the preprocessed information;

[0061] Step S3: Convert each of the target candidate boxes into a four-dimensional visual observation vector. The visual observation vector includes at least the center position, aspect ratio, and scale information of the target candidate box.

[0062] Step S4: Based on the completeness of the external source information of the pod and the effectiveness of the geometric solution, adaptively select the angle domain model or the Cartesian domain model to predict the target state;

[0063] Specifically, if the angle domain model is selected, the target candidate box is back-projected and transformed to the reference attitude coordinate system to obtain the target's azimuth, pitch, and angular scale measurements, and state prediction is performed in the angle domain state space; if the Cartesian domain model is selected, the target candidate box is back-projected and intersected with the target height plane to obtain the target's physical position and physical size measurements, and state prediction is performed in the Cartesian domain state space.

[0064] Step S5: Map the state prediction results of the selected model to a unified visual observation space to obtain the predicted observations;

[0065] Step S6: Calculate the motion consistency metric between the visual observation and the predicted observation of each candidate box, and fuse the motion consistency metric with the corresponding visual confidence score to obtain the comprehensive score of each candidate box;

[0066] Step S7: Select the candidate box with the highest comprehensive score as the best candidate box, and update the state of the selected model using the best candidate box when it meets the update conditions; otherwise, only retain the state prediction results.

[0067] Step S8: Based on the updated or predicted target state, generate the target tracking result for the current frame, and based on preset stable release conditions, decide whether to release the target tracking result to the outside world.

[0068] In step S1, this application receives the current image frame, at least one target candidate bounding box and its visual confidence score output by the front-end visual tracker, and time-aligned pod external source information. Subsequent steps are used to perform unified observation representation, state-space modeling, dynamic prediction, multi-candidate association, measurement update, and stable release determination on the candidate bounding boxes based on the pod external source information. In some optional embodiments, the pod external source information includes camera intrinsic parameters, pod position, pod attitude, camera field of view, and prior information on target height.

[0069] Subsequently, in step S2, the external source information of the pod is first preprocessed. In some optional embodiments, the preprocessing of the external source information of the pod includes at least one of temporal smoothing, filtering, interpolation, temporal alignment, outlier removal, or sensor fusion.

[0070] Understandably, in order to reduce the amplification effect of pod measurement noise, transmission jitter, time asynchrony, or short-term anomalies on geometric mapping and state estimation, this embodiment may perform prior smoothing, filtering, compensation, correction, or fusion processing on the pod's external information in step S2. The external information may include not only position and attitude information, but also velocity, angular velocity, field of view, focal length, target priors, platform state information, and other parameters that can assist target tracking.

[0071] In practical implementation, any suitable time-domain processing method can be used to preprocess the external information, depending on the system's accuracy requirements, computational resources, information sampling frequency, and noise characteristics. For example, moving average, exponential smoothing, median filtering, low-pass filtering, autoregressive filtering, complementary filtering, Kalman filtering, extended Kalman filtering, unscented Kalman filtering, particle filtering, interpolation smoothing, attitude interpolation, outlier removal, time alignment, sensor fusion, or other equivalent information smoothing and correction methods can be used. These methods can be used individually or in combination; a unified processing strategy can be applied to the same type of information, or different processing strategies can be applied to different types of information.

[0072] For example, for positional information, high-frequency noise can be suppressed through linear or nonlinear filtering; for attitude information, the stability of orientation estimation can be improved through interpolation, filtering, or constraint optimization; for field of view, focal length, or other imaging parameters, parameter consistency can be improved through temporal smoothing, lookup table correction, model compensation, or joint estimation. The above algorithms and processing methods are merely examples and not intended to limit the scope of this invention.

[0073] Furthermore, the prior processing can be completed independently based on a single external source of information, or it can be performed collaboratively by multiple information sources; it can be executed in real time in the current frame, or it can be recursively updated by combining historical frames, prediction results, or system feedback. The processed external source information can serve as the basic auxiliary information for subsequent geometric mapping, state prediction, candidate association, result correction, and output control.

[0074] Next, in step S3, all the target candidate boxes are uniformly converted into four-dimensional visual observation vectors. Let the current image width be... The height is The horizontal field of view is The vertical field of view is Then the camera's equivalent focal length It can be written as:

[0075] ;

[0076] Camera intrinsic parameter matrix Recorded as:

[0077] ;

[0078] in, and The coordinates of the main point.

[0079] Let the candidate boxes output by the front-end visual tracker be or .in, and The x and y coordinates of the bottom left corner of the candidate box. and The x and y coordinates are the center positions of the candidate boxes. The candidate box width, The height of the candidate bounding box is used as the basis for the unified association in the backend. To facilitate subsequent association, all candidates and predictions are uniformly represented as a four-dimensional visual observation vector. :

[0080] ;

[0081] in, The aspect ratio of the candidate bounding box. and Same as the candidate box height, T is the matrix transpose operation, specifically:

[0082] , , , .

[0083] This unified observation representation decouples the center location, shape, and scale, facilitating uncertainty propagation and multi-candidate consistency comparison.

[0084] Subsequently, in step S4, instead of using a single state space, two models—an angle-domain baseline model and a Cartesian domain model—are used for target state prediction. In some optional embodiments, the step of adaptively selecting the angle-domain model or the Cartesian domain model includes: determining whether the external source information of the pod in the current frame is complete, and whether the current line of sight and the target plane determined a priori based on the target height can form a valid intersection point; if the above two conditions are met, the Cartesian domain model is selected; otherwise, the angle-domain model is selected.

[0085] In this embodiment, the Cartesian domain model is preferentially attempted during runtime; if the current frame lacks latitude, longitude, and altitude, the target height is prior, or there is no valid intersection between the current line of sight and the target plane, the system automatically reverts to the angle domain model to continue tracking. The Cartesian domain is only enabled when the external information is complete and the geometric solution is successful.

[0086] The angle domain model and the Cartesian domain model will be explained below.

[0087] The angle domain model is used to describe the changes in the line of sight and angular scale of the target relative to the reference attitude. It does not directly recover the true three-dimensional position of the target, so it has a low dependence on external information and is suitable for scenarios where the target height is unknown, the plane assumption does not hold, or the Cartesian domain geometry solution fails.

[0088] Let the reference attitude coordinate system be the camera attitude coordinate system fixed during tracking setup. The line-of-sight direction after back-projection of the target center pixel in the current frame is uniformly expressed in this reference attitude coordinate system after attitude transformation, thereby removing the pseudo-motion introduced by the pod rotation from the image observation.

[0089] Angle domain state vector Defined as:

[0090] ;

[0091] in, Let be the azimuth angle of the target relative to the reference attitude coordinate system. Let be the pitch angle of the target relative to the reference attitude coordinate system. and These are the angular velocities for azimuth and pitch, respectively. Aspect ratio, For height.

[0092] Based on this, the angle domain state vector at the current k-th frame can be calculated using the angular velocity model commonly used in the angle domain algorithm for the two frames before and after the discrete state transition equation.

[0093] Next, angular domain measurements are constructed from the pixel bounding boxes. The center of the current candidate bounding box satisfies:

[0094] , ;

[0095] The viewing vector is obtained by back-projecting the pixel center onto the current camera coordinate system. :

[0096] ;

[0097] The above results are normalized to obtain the unit direction vector. Let the current posture be... The reference posture is Therefore, the direction vector in the reference attitude coordinate system can be obtained. Let the obtained for Then the azimuth angle With pitch angle for:

[0098] ;

[0099] ;

[0100] Corner width corresponding to the width and height of the candidate box and angle height for:

[0101] ;

[0102] ;

[0103] Therefore, we can conclude that:

[0104] , ;

[0105] Thus constructing angle domain measurement .

[0106] Finally, the angle domain state is projected back into the image domain, and the orientation vector under the reference pose can be recovered based on the predicted state:

[0107] .

[0108] The coordinates of the predicted center pixel are then obtained by rotating and transforming the current pose back to the current camera coordinate system and projecting it onto the image plane. .

[0109] If the predicted angular scale is The predicted aspect ratio is Then the angle width is:

[0110] ;

[0111] The corresponding pixel width and height are:

[0112] ;

[0113] Therefore, the prediction box can be calculated:

[0114] .

[0115] It should also be noted that the azimuth angle is a periodic variable, and the shortest angle difference should be used during the update to avoid generating false large residuals when crossing ±π.

[0116] The Cartesian domain is used to restore pixel observations to their position and physical size in the world coordinate system and to perform dynamic estimation within that physical space. It is suitable for scenarios where the external source information of the pod is complete and the target can be approximately located on a plane with a known height.

[0117] Regarding the definition of world coordinates, a local geographic tangent plane coordinate system is preferred, and the Cartesian domain state vector is defined as follows:

[0118] ;

[0119] The corresponding measurement vector is defined as ;

[0120] in, and These represent the position components in two orthogonal directions within the local plane. The vertical component is... and The velocity components of the target within the target plane. and These are the target physical width and physical height, respectively.

[0121] Based on this, the Cartesian domain state vector at the k-th frame can be calculated using the planar velocity model commonly used in the Cartesian domain algorithm for the two frames before and after the discrete state transition equation.

[0122] Next, Cartesian domain measurements are constructed from the pixel bounding boxes. The viewing direction in the current camera coordinate system is obtained by backprojecting the center pixel of the candidate bounding box using the camera intrinsic parameters. Then, the orientation vector in the world coordinate system is obtained through attitude transformation. :

[0123] ;

[0124] in, This is the transformation matrix between the world coordinate system and the camera coordinate system.

[0125] Let the position of the camera's optical center in the world coordinate system be... The normal vector of the target plane is A point on the plane is Then the intersection point t is:

[0126] ;

[0127] like or If the intersection of the planes in the current frame fails, the Cartesian domain measurement cannot be constructed in the current frame, and the frame should be reverted to the angle domain.

[0128] When the intersection is valid, the target measurement location is:

[0129] ;

[0130] Let the distance from the camera to the intersection point be:

[0131] .

[0132] The candidate box's pixel width and height are then converted to physical width and height:

[0133] ;

[0134] ;

[0135] From this, we can obtain the Cartesian domain measurement. .

[0136] Finally, the Cartesian domain state is projected back to the image domain. The world coordinates of the target in the predicted state are transformed to the camera coordinate system through the current camera extrinsic parameters, and then the predicted center pixel coordinates are obtained through perspective projection. Finally, the predicted bounding box is calculated based on the distance from the current predicted target to the camera.

[0137] Then, in step S6, the overall score of each candidate box is calculated.

[0138] In some alternative implementations, in step S6, the first step is calculated using the following formula. The overall score of each candidate box :

[0139] ;

[0140] in, To integrate weights, For the first Visual confidence of each candidate bounding box For the first A measure of motion consistency between visual observations of candidate boxes and predicted observations. This is the consistency threshold. The visual magnification index is a high-resolution visual enhancement index.

[0141] in, ; For the first Innovation vector of each candidate box For the first Innovation covariance of each candidate box;

[0142] in, ; ;

[0143] in, For the first Visual observation of candidate boxes, The mean of the predicted observations is obtained by mapping the state prediction results to a unified visual observation space. For visual observation noise covariance, To predict the observation noise covariance.

[0144] In this embodiment, to unify the candidate comparison method between the angular domain and the Cartesian domain, the present invention reprojects the prediction result of any model onto a unified visual observation space to obtain the predicted observation mean. Covariance of predicted observation noise Since the mapping from state space to visual observation space is usually nonlinear, it is preferable to use unscented transformation for covariance propagation.

[0145] Then, based on the above comprehensive evaluation, the candidate box with the highest score is selected as the best candidate box. In step S7, filtering and updating are performed; when the score of the best candidate box exceeds the update threshold... If the candidate box is accepted, it is used for state updates; otherwise, the current visual observation is rejected, and only the prediction result is retained.

[0146] It should be noted that in step S7, after the best candidate passes the gating, it is transformed from the unified visual observation space to the measurement space corresponding to the currently selected state space, and then the update is performed.

[0147] This application preferably uses state-based scheduling of parameters according to the tracking phase, including but not limited to the initialization phase, stabilization phase, recovery phase, and loss phase. Different update thresholds can be configured for different phases. Consistency threshold fusion weight And the stable release window length.

[0148] Finally, in step S8, to avoid directly transmitting internal short-term jitter to external systems, this application further sets up a high-quality external release mechanism.

[0149] In some optional implementations, step S8, which involves determining whether to release the product based on preset stable release conditions, includes:

[0150] Statistical length is The percentage of frames that meet the high-quality criteria within the sliding window. ;

[0151] Stability tracking results will be released to the public when all of the following conditions are met:

[0152] Window length reaches The time interval since the last release is not less than the set value. ;

[0153] The proportion of high-quality conditions Not lower than the preset threshold .

[0154] In this embodiment, the length is set to... Within the sliding window, the first Whether a frame meets the high-quality condition is denoted as Then the window quality percentage is:

[0155] .

[0156] This mechanism allows internal filtering to run continuously, while external results are only published when they are sufficiently stable, making it more suitable for downstream control and indication scenarios.

[0157] Compared with existing methods that primarily smooth target boxes directly in the image pixel domain, the present invention has at least the following advantages:

[0158] 1) Fully utilize external information such as pod attitude, position, and field of view to decouple changes in platform perspective from the actual motion of the target, significantly improving target tracking stability;

[0159] 2) By using a dual-model complementary structure of angle domain baseline protection and Cartesian domain optimization, continuous tracking is guaranteed when information is insufficient, and higher accuracy physical space prediction is obtained when information is sufficient;

[0160] 3) Improve false detection suppression, occlusion recovery, and recapture capabilities by using a unified visual observation representation and a multi-candidate joint scoring mechanism;

[0161] 4) By performing only predictive extension when gating fails, state pollution caused by erroneous observations is reduced;

[0162] 5) By setting up a high-quality external release mechanism, a hierarchical control of internal continuous estimation and external conservative output can be achieved, which is more conducive to engineering system integration;

[0163] 6) The proposed solution does not depend on a specific front-end visual tracker and has good module compatibility and engineering scalability.

[0164] The second aspect of this application provides a visual tracking fusion compensation system based on external information from an unmanned aerial vehicle (UAV) pod, corresponding to the above-described method, mainly comprising:

[0165] The parameter acquisition module is used to receive the current image frame, at least one target candidate box and its visual confidence output by the front-end visual tracker, and time-aligned pod external source information;

[0166] The pod external source information preprocessing module is used to preprocess the pod external source information and establish a camera model for the current frame based on the preprocessed information.

[0167] The target parameter unification module is used to convert each of the target candidate boxes into a four-dimensional visual observation vector, wherein the visual observation vector includes at least the center position, aspect ratio and scale information of the target candidate box;

[0168] The algorithm adaptive selection module is used to adaptively select either the angle domain model or the Cartesian domain model for target state prediction based on the completeness of the external source information of the pod and the effectiveness of the geometric solution.

[0169] Specifically, if the angle domain model is selected, the target candidate box is back-projected and transformed to the reference attitude coordinate system to obtain the target's azimuth, pitch, and angular scale measurements, and state prediction is performed in the angle domain state space; if the Cartesian domain model is selected, the target candidate box is back-projected and intersected with the target height plane to obtain the target's physical position and physical size measurements, and state prediction is performed in the Cartesian domain state space.

[0170] The predictive observation acquisition module is used to map the state prediction results of the selected model to a unified visual observation space to obtain predictive observations.

[0171] The candidate box comprehensive score calculation module is used to calculate the motion consistency metric between the visual observation and the predicted observation of each candidate box, and to fuse the motion consistency metric with the corresponding visual confidence to obtain the comprehensive score of each candidate box.

[0172] The model state update module is used to select the candidate box with the highest comprehensive score as the best candidate box, and update the state of the selected model using the best candidate box when it meets the update conditions; otherwise, only the state prediction result is retained.

[0173] The target tracking result publishing module is used to generate the target tracking result of the current frame based on the updated or predicted target state, and decide whether to publish the target tracking result based on preset stable publishing conditions.

[0174] In some alternative implementations, the external information of the pod includes camera intrinsic parameters, pod position, pod attitude, camera field of view, and prior information on target height.

[0175] In some alternative implementations, the step of adaptively selecting the angle domain model or the Cartesian domain model includes:

[0176] Determine whether the external source information of the pod in the current frame is complete, and whether the current line of sight can form a valid intersection with the target plane determined a priori based on the target height;

[0177] If the above two conditions are met, the Cartesian domain model is selected; otherwise, the angular domain model is selected.

[0178] In some optional implementations, in the candidate box comprehensive score calculation module, the first score is calculated using the following formula. The overall score of each candidate box :

[0179] ;

[0180] in, To integrate weights, For the first Visual confidence of each candidate bounding box For the first A measure of motion consistency between visual observations of candidate boxes and predicted observations. This is the consistency threshold. Visual high-resolution magnification index;

[0181] in, ; For the first Innovation vector of each candidate box For the first Innovation covariance of each candidate box;

[0182] in, ; ;

[0183] in, For the first Visual observation of candidate boxes, The mean of the predicted observations is obtained by mapping the state prediction results to a unified visual observation space. For visual observation noise covariance, To predict the observation noise covariance.

[0184] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A visual tracking fusion compensation method based on external information from an unmanned aerial vehicle (UAV) pod, characterized in that, include: Step S1: Receive the current image frame, at least one target candidate box and its visual confidence output by the front-end visual tracker, and time-aligned pod external source information; Step S2: Preprocess the external source information of the pod, and establish the camera model of the current frame based on the preprocessed information; Step S3: Convert each of the target candidate boxes into a four-dimensional visual observation vector. The visual observation vector includes at least the center position, aspect ratio, and scale information of the target candidate box. Step S4: Based on the completeness of the external source information of the pod and the effectiveness of the geometric solution, adaptively select the angle domain model or the Cartesian domain model to predict the target state; Specifically, if the angle domain model is selected, the target candidate box is back-projected and transformed to the reference attitude coordinate system to obtain the target's azimuth, pitch, and angular scale measurements, and state prediction is performed in the angle domain state space; if the Cartesian domain model is selected, the target candidate box is back-projected and intersected with the target height plane to obtain the target's physical position and physical size measurements, and state prediction is performed in the Cartesian domain state space. Step S5: Map the state prediction results of the selected model to a unified visual observation space to obtain the predicted observations; Step S6: Calculate the motion consistency metric between the visual observation and the predicted observation of each candidate box, and fuse the motion consistency metric with the corresponding visual confidence score to obtain the comprehensive score of each candidate box; Step S7: Select the candidate box with the highest comprehensive score as the best candidate box, and update the state of the selected model using the best candidate box when it meets the update conditions; otherwise, only retain the state prediction results. Step S8: Based on the updated or predicted target state, generate the target tracking result for the current frame, and based on preset stable release conditions, decide whether to release the target tracking result to the outside world.

2. The visual tracking fusion compensation method based on external information from an unmanned aerial vehicle (UAV) pod according to claim 1, characterized in that, The external information of the pod includes camera intrinsic parameters, pod position, pod attitude, camera field of view, and prior information on target height.

3. The visual tracking fusion compensation method based on external information from an unmanned aerial vehicle (UAV) pod according to claim 1, characterized in that, The preprocessing of external information from the pod includes at least one of temporal smoothing, filtering, interpolation, temporal alignment, outlier removal, or sensor fusion.

4. The visual tracking fusion compensation method based on external source information from an unmanned aerial vehicle (UAV) pod according to claim 1, characterized in that, The steps for adaptively selecting the angle domain model or the Cartesian domain model include: Determine whether the external source information of the pod in the current frame is complete, and whether the current line of sight can form a valid intersection with the target plane determined a priori based on the target height; If the above two conditions are met, the Cartesian domain model is selected; otherwise, the angular domain model is selected.

5. The visual tracking fusion compensation method based on external source information from an unmanned aerial vehicle (UAV) pod according to claim 1, characterized in that, In step S6, the first step is calculated using the following formula. The overall score of each candidate box : ； in, To integrate weights, For the first Visual confidence of each candidate bounding box For the first A measure of motion consistency between visual observations of candidate boxes and predicted observations. This is the consistency threshold. Visual high-resolution magnification index; in, ; For the first Innovation vector of each candidate box For the first Innovation covariance of each candidate box; in, ; ; in, For the first Visual observation of candidate boxes, The mean of the predicted observations is obtained by mapping the state prediction results to a unified visual observation space. For visual observation noise covariance, To predict the observation noise covariance.

6. The visual tracking fusion compensation method based on external source information from an unmanned aerial vehicle (UAV) pod according to claim 1, characterized in that, In step S8, determining whether to release to the public based on preset stable release conditions includes: Statistical length is The percentage of frames that meet the high-quality criteria within the sliding window. ; Stability tracking results will be released to the public when all of the following conditions are met: Window length reaches The time interval since the last release is no less than the set value. ; The proportion of high-quality conditions Not lower than the preset threshold .

7. A visual tracking fusion compensation system based on external information from an unmanned aerial vehicle (UAV) pod, characterized in that, include: The parameter acquisition module is used to receive the current image frame, at least one target candidate box and its visual confidence output by the front-end visual tracker, and time-aligned pod external source information; The pod external source information preprocessing module is used to preprocess the pod external source information and establish a camera model for the current frame based on the preprocessed information. The target parameter unification module is used to convert each of the target candidate boxes into a four-dimensional visual observation vector, wherein the visual observation vector includes at least the center position, aspect ratio and scale information of the target candidate box. The algorithm adaptive selection module is used to adaptively select either the angle domain model or the Cartesian domain model for target state prediction based on the completeness of the external source information of the pod and the effectiveness of the geometric solution. Specifically, if the angle domain model is selected, the target candidate box is back-projected and transformed to the reference attitude coordinate system to obtain the target's azimuth, pitch, and angular scale measurements, and state prediction is performed in the angle domain state space; if the Cartesian domain model is selected, the target candidate box is back-projected and intersected with the target height plane to obtain the target's physical position and physical size measurements, and state prediction is performed in the Cartesian domain state space. The predictive observation acquisition module is used to map the state prediction results of the selected model to a unified visual observation space to obtain predictive observations. The candidate box comprehensive score calculation module is used to calculate the motion consistency metric between the visual observation and the predicted observation of each candidate box, and to fuse the motion consistency metric with the corresponding visual confidence to obtain the comprehensive score of each candidate box. The model state update module is used to select the candidate box with the highest comprehensive score as the best candidate box, and update the state of the selected model using the best candidate box when it meets the update conditions; otherwise, only the state prediction result is retained. The target tracking result publishing module is used to generate the target tracking result of the current frame based on the updated or predicted target state, and decide whether to publish the target tracking result based on preset stable publishing conditions.

8. The visual tracking fusion compensation system based on external information from an unmanned aerial vehicle (UAV) pod according to claim 7, characterized in that, The external information of the pod includes camera intrinsic parameters, pod position, pod attitude, camera field of view, and prior information on target height.

9. The visual tracking fusion compensation system based on external information from an unmanned aerial vehicle (UAV) pod according to claim 7, characterized in that, The steps for adaptively selecting the angle domain model or the Cartesian domain model include: Determine whether the external source information of the pod in the current frame is complete, and whether the current line of sight can form a valid intersection with the target plane determined a priori based on the target height; If the above two conditions are met, the Cartesian domain model is selected; otherwise, the angular domain model is selected.

10. The visual tracking fusion compensation system based on external information from an unmanned aerial vehicle (UAV) pod according to claim 7, characterized in that, In the candidate box comprehensive score calculation module, the first score is calculated using the following formula. The overall score of each candidate box : ； in, To integrate weights, For the first Visual confidence of each candidate bounding box For the first A measure of motion consistency between visual observations of candidate boxes and predicted observations. This is the consistency threshold. Visual high-resolution magnification index; in, ; For the first Innovation vector of each candidate box For the first Innovation covariance of each candidate box; in, ; ; in, For the first Visual observation of candidate boxes, The mean of the predicted observations is obtained by mapping the state prediction results to a unified visual observation space. For visual observation noise covariance, To predict the observation noise covariance.

Images