A method and system for locking and tracking aircraft targets in complex dynamic environments

Abstract

The present invention provides a method and system for locking and tracking an aircraft target in a complex dynamic environment, including: the aircraft obtains the flight altitude through an altitude sensor; after performing image preprocessing on the ground target, target recognition is carried out. If a possible target is recognized, it enters the target preselection state; all possible target images and information are sent to the ground end; Adaptive-SORT is used to perform image tracking and recognition on the predetermined target to give target information; according to the obtained target information and flight altitude, the relative three-dimensional position between the target and the aircraft is calculated, and the target position parameters are updated; according to the target position parameters, the desired flight attitude is obtained, and the flight attitude of the aircraft is controlled to achieve the aircraft's locking and tracking of the predetermined target. The aircraft of the present invention can autonomously track and recognize a specified target in a complex dynamic environment, and enable the aircraft to predict the target movement trajectory when encountering obstacles or other interferences, ensuring the stability of tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and particularly to a method and system for locking and tracking an aircraft target in a complex dynamic environment. Background Art

[0002] With the rapid development of aircraft technology, especially unmanned aerial vehicles (UAVs), due to their low cost, simple operation, light weight and portability, UAVs are increasingly widely used in military, civilian, commercial and other fields. And high-speed target-tracking UAVs, because of their excellent maneuverability and rapid deployment capabilities, have broad application prospects in public safety fields such as military reconnaissance, fugitive tracking, fire rescue, etc., and civilian fields such as automatic follow-up shooting.

[0003] However, for existing high-speed tracking UAVs, although they can accurately identify and track in real time, in a complex environment, in the case of interference such as being blocked by an obstacle or the target suddenly disappearing, it is extremely easy to cause the loss of target tracking, thus interrupting the tracking task. Moreover, existing aircraft target locking and tracking methods are extremely prone to problems of target locking and tracking failure when multiple approximate targets appear simultaneously, and there are practical problems such as unstable performance, poor anti-interference ability, and low robustness.

[0004] Therefore, developing a method for locking and tracking an aircraft target in a complex dynamic environment has great application value and significance. Summary of the Invention

[0005] The present invention provides a method and system for locking and tracking an aircraft target in a complex dynamic environment, to solve the defects existing in the prior art in the field of aircraft target tracking, realize locking and tracking of a target by an aircraft in a complex environment, and predict the target movement trajectory in real time to prevent the loss of target tracking.

[0006] In a first aspect, the present invention provides a method for locking and tracking an aircraft target in a complex dynamic environment, including:

[0007] After the aircraft hovers stably in the air, obtain the height of the aircraft through a height sensor;

[0008] Collect a target image of a ground target through a camera, and preprocess the target image to obtain a preprocessed image;

[0009] Perform target recognition on the preprocessed image. If it is determined that a pending target is recognized and obtained, enter the target pre-selection state;

[0010] Send the pending target to the ground end, and receive the selected target returned by the ground end;

[0011] Using the Adaptive-SORT algorithm, perform image tracking and recognition on the selected target, and output target information;

[0012] According to the target information and the altitude of the aircraft, calculate the relative three-dimensional position between the selected target and the aircraft, and update the target position parameters based on the relative three-dimensional position;

[0013] Obtain the desired flight attitude from the target position parameters, and control the flight attitude of the aircraft through the flight control system, so that the aircraft completes real-time target locking and tracking.

[0014] According to a method for locking and tracking aircraft targets in a complex dynamic environment provided by the present invention, after using the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target and output target information, it further includes:

[0015] If it is determined that there is interference information or object occlusion causing the target to be lost, then use the Adaptive-SORT algorithm to predict the object's motion trajectory, and estimate the predicted target information according to the object's motion trajectory.

[0016] According to a method for locking and tracking aircraft targets in a complex dynamic environment provided by the present invention, collect the target image of the ground target through a camera, and preprocess the target image to obtain the preprocessed image, including:

[0017] Obtain the principal point coordinates of the target image and the lens focal length of the camera, determine the image coordinate system corresponding to the target image, and use the principal point coordinates and the lens focal length to convert the image coordinate system to the camera coordinate system corresponding to the camera;

[0018] Determine the coupling coefficient of the camera coordinate system, and adjust the abscissa of the camera coordinate system according to the coupling coefficient to obtain the adjusted abscissa;

[0019] Obtain the radial distortion coefficient and the tangential distortion coefficient, and use the radial distortion coefficient, the tangential distortion coefficient, and the adjusted abscissa to perform distortion correction on the camera coordinate system to obtain the corrected camera coordinates;

[0020] Based on the principal point coordinates and the lens focal length, reversely convert the corrected camera coordinates to the image coordinate system;

[0021] Repeat the above operations of converting the image coordinate system to the camera coordinate system, coordinate distortion correction, and coordinate reverse conversion several times to obtain the distortion-corrected image;

[0022] Optimize the clarity or contrast of the distortion-corrected image to obtain the preprocessed image.

[0023] A method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, which obtains radial distortion coefficients and tangential distortion coefficients, and corrects the distortion of the camera coordinate system by using the radial distortion coefficients, the tangential distortion coefficients, and the adjusted abscissa to obtain the corrected camera coordinates, includes:

[0024] Calculating a distortion radius by using the adjusted abscissa and ordinate;

[0025] Calculating a radial distortion correction factor from the distortion radius and the radial distortion coefficients;

[0026] Calculating a tangential distortion correction factor according to the tangential distortion coefficients, the distortion radius, the adjusted abscissa, and the ordinate;

[0027] Based on the radial distortion correction factor and the tangential distortion correction factor, correcting the adjusted abscissa and the ordinate to obtain the corrected camera coordinates.

[0028] A method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, which performs target recognition on the preprocessed image, includes:

[0029] Adopting an improved YOLOv11 model and combining a large convolutional kernel attention mechanism to construct a target recognition model;

[0030] Inputting the preprocessed image into the target recognition model and outputting a target recognition result.

[0031] A method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, which uses an Adaptive-SORT algorithm for image tracking and recognition of the selected target and outputs target information, includes:

[0032] Determining a moving video sequence frame corresponding to the selected target, and obtaining a subsequent frame image of the previous frame image in the moving video sequence frame by using a Hungarian matching algorithm, where there is an optimal matching estimate between n detection boxes in the previous frame image and m detection boxes in the subsequent frame image;

[0033] Calculating the intersection-over-union ratio of any one of the n detection boxes and any one of the m detection boxes, and constructing a dimensional cost matrix from multiple intersection-over-union ratios;

[0034] Constructing a Kalman filter state model for the selected target according to the target center position, the length and width of the target detection box, the moving speed of the target center position, the time change rate of the length and width of the target detection box, and the area of the target detection box;

[0035] Construct the state transition equation of the selected target Kalman filter state model. The state transition equation includes a state transition matrix, which predicts the state variables at time k based on the state variables at time k-1 and process noise. Among them, the process noise is Gaussian white noise, and the noise covariance matrix corresponding to the Gaussian white noise is composed of the variances of the process noise of each state variable;

[0036] Determine the prior state estimate value, prior state estimate covariance matrix, posterior state estimate value, and posterior state estimate covariance matrix of the state variables at time k, obtain the Kalman filter prediction equations for target tracking, predict and output the prior estimate vector from the Kalman filter prediction equations, convert the respective state values of the prior estimate vector into a measurement matrix, and calculate the Kalman gain based on the measurement matrix to minimize the mean square error of the system state estimate;

[0037] Collect the residual errors of each frame in the sliding window through the sliding window, calculate the root mean square error value of the residuals in the N-frame sliding window, and perform transform adaptive correction on the root mean square error value of the residuals using an error function to obtain the noise covariance matrix;

[0038] Obtain a new moving video sequence frame, update the new moving video sequence frame based on the noise covariance matrix, and output the target information.

[0039] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, based on the target information and the altitude of the aircraft, calculate and obtain the relative three-dimensional position of the selected target and the aircraft, and update the target position parameters based on the relative three-dimensional position, including:

[0040] Obtain the pixel coordinates of the target information in the image plane;

[0041] Scale the pixel coordinates according to the scaling ratio, image length, and image width to obtain image coordinates;

[0042] Convert the image coordinates into camera coordinates based on perspective shadow and the focal length of the camera;

[0043] Rotate the camera coordinates according to the rotation matrix and offset vector of the camera to obtain the rotated coordinates. The rotation matrix includes the pitch angle, yaw angle, and roll angle of the aircraft;

[0044] Obtain the angle between the line connecting the target object and the imaging two points and the central axis of the camera, and combine the altitude of the aircraft and the pitch angle to convert the rotated coordinates into the final coordinates of the target in the world coordinate system with the aircraft as the origin.

[0045] A method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, obtaining an expected flight attitude from the target position parameters, and controlling the flight attitude of the aircraft through a flight control system to enable the aircraft to complete real-time target locking and tracking, including:

[0046] Input the target position parameters into the inner and outer loop controllers of the aircraft position and attitude, and convert them into control quantities of the flight control system;

[0047] Complete real-time target locking and tracking based on the control quantities of the flight control system.

[0048] A method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, after target recognition of the preprocessed image, further includes:

[0049] If it is determined that no valid target is recognized, the aircraft performs self-stabilized hovering or flies according to the ground control command.

[0050] In a second aspect, the present invention further provides a system for locking and tracking an aircraft target in a complex dynamic environment, including:

[0051] An acquisition module, configured to obtain the height of the aircraft through a height sensor after the aircraft hovers stably in the air;

[0052] A preprocessing module, configured to collect a target image of a ground target through a camera and preprocess the target image to obtain a preprocessed image;

[0053] A determination module, configured to perform target recognition on the preprocessed image, and if it is determined that a pending target is recognized, enter the target preselection state;

[0054] A receiving module, configured to send the pending target to the ground end and receive the selected target returned by the ground end;

[0055] A tracking module, configured to use the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target and output target information;

[0056] A calculation module, configured to calculate the relative three-dimensional position between the selected target and the aircraft according to the target information and the height of the aircraft, and update the target position parameters based on the relative three-dimensional position;

[0057] A tracking module, configured to obtain an expected flight attitude from the target position parameters, and control the flight attitude of the aircraft through a flight control system to enable the aircraft to complete real-time target locking and tracking.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] On the one hand, the present invention realizes continuous and stable tracking of a high-speed tracking UAV in a complex dynamic environment, and can predict the motion trajectory of a locked target in real time, enabling the UAV to dynamically adjust its own position according to the predicted trajectory even in the face of long-term occlusion, target loss, etc., so that after the interference environment is removed, it can continue to lock and track the established target, improving the stability of target locking and tracking;

[0060] On the other hand, the present invention adopts a new Adaptive-SORT technology, which can continue to stably lock and track the established target when multiple approximate targets appear simultaneously, and has strong anti-interference ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0062] Figure 1 is a schematic flow chart of the aircraft target locking and tracking for a complex dynamic environment provided by the present invention;

[0063] Figure 2 is a schematic diagram of an image de-distortion camera calibration board provided by the present invention;

[0064] Figure 3 is a schematic flow chart of the aircraft control provided by the present invention;

[0065] Figure 4 is a schematic diagram of the change of the measurement noise covariance matrix update function of the Kalman filter provided by the present invention with respect to the parameter λ;

[0066] Figure 5 is a relative position diagram of the UAV image center and the target provided by the present invention;

[0067] Figure 6 is a schematic diagram of the principle of target position calculation provided by the present invention;

[0068] Figure 7 is a schematic structural diagram of the aircraft target locking and tracking system device for a complex dynamic environment provided by the present invention;

[0069] Figure 8 is a schematic structural diagram of the aircraft target locking and tracking system for a complex dynamic environment provided by the present invention;

[0070] Figure 9 is a schematic structural diagram of the electronic device provided by the present invention. Detailed Implementation Manner

[0071] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0072] Figure 1 is a schematic flowchart of the aircraft target locking and tracking for a complex dynamic environment provided by an embodiment of the present invention. As Figure 1 shown, it includes:

[0073] Step 100: After the aircraft hovers stably in the air, obtain the height of the aircraft through a height sensor;

[0074] Step 200: Collect a target image of the ground target through a camera, and perform preprocessing on the target image to obtain a preprocessed image;

[0075] Step 300: Perform target recognition on the preprocessed image. If it is determined that a pending target is recognized and obtained, enter the target pre-selection state;

[0076] Step 400: Send the pending target to the ground terminal, and receive the selected target returned by the ground terminal;

[0077] Step 500: Use the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target, and output target information;

[0078] Step 600: Calculate the relative three-dimensional position between the selected target and the aircraft based on the target information and the height of the aircraft, and update the target position parameters based on the relative three-dimensional position;

[0079] Step 700: Obtain the desired flight attitude from the target position parameters, and control the flight attitude of the aircraft through a flight control system to enable the aircraft to complete real-time target locking and tracking.

[0080] Specifically, in the embodiments of the present invention, the aircraft obtains the flight altitude through a height sensor; acquires and preprocesses images of ground targets through a camera; performs target recognition on the preprocessed images. If a possible target is recognized, it enters the preselected target state; sends all possible target images and information to the ground terminal, and the ground terminal operator manually selects the tracking target; uses the Adaptive-SORT technology to perform image tracking and recognition on the predetermined target and gives target information; calculates the relative three-dimensional position between the target and the aircraft based on the obtained target information and flight altitude, and updates the target position parameters; obtains the desired flight attitude based on the target position parameters, and controls the flight attitude of the aircraft through the flight control system to achieve the aircraft's locking and tracking of the predetermined target.

[0081] The aircraft in the present invention can autonomously track and recognize a specified target in a complex dynamic environment, and enable the aircraft to predict the target's movement trajectory when encountering obstacles or other interferences to ensure the stability of tracking.

[0082] In one embodiment, the aircraft in step 100 realizes self-stabilized hovering in the air. By using the tight-coupled LiDAR-Inertial Odometry (LIO) technology, the UAV can achieve self-stabilized hovering without the Global Navigation Satellite System (GNSS) signal.

[0083] In one embodiment, the preprocessing of the image in step 200 includes image de-distortion operations, including the following operations:

[0084] First is the coordinate transformation, which transforms the coordinates in the image coordinate system to the camera coordinate system , and is achieved through the following formula:

[0085]

[0086]

[0087] where principal is the principal point of the image, is the origin in the coordinates corresponding to the principal point of the image, focal is the focal length of the lens, is the origin in the coordinates corresponding to the focal length of the lens, is any pixel point in the image.

[0088] Then, adjust the X[i] coordinate according to the coupling coefficient (skew) to remove the influence of the non-orthogonality of the x-axis and y-axis of the camera. The adjusted coordinate is:

[0089]

[0090] Further, the radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2) are used for distortion correction, which is carried out through the following steps:

[0091] (a) Calculate the distortion radius temp:

[0092]

[0093] (b) Calculate the radial distortion correction factor tempKr:

[0094]

[0095] (c) Calculate the tangential distortion and correct the coordinates (deltax, deltay) according to the tangential distortion coefficients:

[0096]

[0097]

[0098] (d) Perform distortion correction on X[i] and Y[i] to obtain the coordinates in the corrected camera coordinate system ( , ):

[0099]

[0100]

[0101] Finally, the coordinates in the corrected camera coordinate system are converted back to the image coordinate system, which is completed through the following formula:

[0102]

[0103]

[0104] After completing the coordinate conversion, one or more image processing algorithms are used to optimize the clarity or contrast of the corrected image, and the specific implementation of this algorithm can be an existing mature image optimization algorithm.

[0105] In the embodiments of the present invention, the above operations of converting from the image coordinate system to the camera coordinate system, distortion correction, and coordinate inverse conversion are all repeatedly executed n times, and n usually takes a value from 3 to 5 to enhance the distortion correction effect.

[0106] It should be noted that the above camera principal point coordinates principal, lens focal length focal, distortion coefficients (k1, k2, k3), and tangential distortion coefficients (p1, p2) are achieved through camera calibration and can be obtained by photographing a specific calibration board, such asFigure 2 As shown, Figure 2 The black and white flat plate with a fixed-pitch pattern array in the middle is the calibration plate. By photographing the flat plate with a fixed-pitch pattern array with a camera and calculating with the calibration algorithm, we can derive the camera's geometric model and other related parameters, thereby obtaining high-precision measurement and reconstruction results.

[0107] In one embodiment, step 300 includes:

[0108] The improved YOLOv11 model is used in combination with the large convolution kernel attention mechanism to build a target recognition model;

[0109] The preprocessed image is input into the target recognition model, and the target recognition result is output.

[0110] In an embodiment of the present invention, a target recognition model based on deep learning is applied to the preprocessed image data. The target recognition model is an improved YOLOv11 model, and a large convolution kernel attention mechanism is introduced. The large convolution kernel attention mechanism is adopted to enable the model to effectively focus on important features in the image, enhance the model's recognition ability for small targets at a distance and its adaptability to complex backgrounds. Based on the YOLOv11 model, the present invention introduces Single-Head Self-Attention (SHSA) to solve problems such as small targets and occlusions in the target recognition process, thereby improving the target recognition performance of the YOLOv11 model.

[0111] It should be noted that the YOLOv11 model is the latest YOLO version released by the Ultralytics team in September 2024. Compared with other historical YOLO versions, it has further enhanced the feature extraction performance, adopted an improved back-end and neck architecture, and enhanced the feature extraction capability, thereby achieving more accurate object detection and more complex task execution; optimized processing efficiency and speed, introduced improved architecture design and optimized training process, achieved faster processing speed, and maintained the best balance between accuracy and performance; adopted more precise and fewer parameters, had strong cross-environment adaptability, and could be seamlessly deployed in various environments, supporting multiple tasks, whether it is target detection, instance segmentation, image classification, posture estimation or directional object detection, all have excellent performance.

[0112] In one embodiment, in step 400, the identified pending target is sent to the ground terminal, and the communication module used includes a control terminal and a data transmission terminal, both of which support multiple protocols, such as Wi-Fi, 4G, 5G and LoRa, to adapt to different communication application scenarios.

[0113] In one embodiment, step 500 includes:

[0114] 1. Multi-object detection results for matching the front and rear moving video sequence frames corresponding to the selected target

[0115] First, use the tracking model to process the video sequence frames, generate a bounding box for each potential dynamic target object detected by the target recognition model in the foregoing embodiments, and determine a specific bounding box for subsequent tracking by manually selecting a predetermined target as described above. Here, the Hungarian algorithm is introduced to obtain the optimal estimate of the correspondence between multiple bounding boxes of objects in the previous and subsequent frame images, and then determine the predicted bounding box in the frame corresponding to the specific bounding box in the frame, as shown in the steps of a series of processing flow branches after the target is recognized in Figure 3 . Here, Figure 3 The complete control process of the aircraft is described in detail. First, the UAV performs image acquisition. If a target is recognized, it enters the subsequent target recognition control process. Otherwise, the UAV enters self-stabilized hovering or flies according to the instructions of the ground control terminal. In the target recognition control process, the Hungarian algorithm is used to match the current state of the UAV with the corresponding target, and the target to be tracked in the corresponding target is manually selected by the ground end, so as to construct a Kalman filter state model for the selected target, and determine whether the UAV detects the target to be tracked. If the target is detected, a dynamic estimation noise covariance matrix R is established. If the target is not detected, it is determined that the target is occluded or not successfully tracked. Therefore, a prediction is made based on the previous state in which the target can be detected, the current target state estimate is compensated, and the current target state estimate is updated using the dynamic estimation noise covariance matrix R. Then, based on the target information and the flight altitude, the relative position between the target and the UAV is calculated, and finally the target is locked and tracked, and information is sent to the UAV to control the flight attitude of the UAV, and the flight position of the UAV is updated and controlled by the flight control module on the UAV.

[0116] If there are n detection bounding boxes in the previous frame image , and there are m detection bounding boxes in the subsequent frame image , construct cost matrix , where each element represents the cost of matching the i-th detection bounding box in the previous frame with the j-th detection bounding box in the subsequent frame . The cost is measured using the intersection over union (IOU) between the detection bounding boxes, that is:

[0117]

[0118] At this time, the cost matrix The zero elements in it correspond to using the optimal solution obtained by the Hungarian matching algorithm as the basis for subsequent Kalman filtering, ensuring that each state variable corresponds to a corresponding target one by one. If no target is detected, the new target state is predicted according to the previous target state through the linear Kalman filter framework and compensated into the current image.

[0119] It can be understood that the zero elements in the cost matrix represent "minimum cost" matches. Through a series of row subtractions, column subtractions, and other matrix operations, attempts are made to create and maintain as many zero elements as possible in the matrix. The more zero elements, the more optimal matches are available. Eventually, when a non-conflicting set of matches is selected from the zero elements, that is, zero element pairs without shared rows or columns are used for matching, the algorithm ends.

[0120] II. Establish a state model based on Kalman filtering

[0121] The target state model is described as follows:

[0122]

[0123] Where , represents the center position of the target, , represents the length and width of the target detection box, , represents the moving speed of the center position of the target, , represents the time change rate of the length and width of the target detection box, and represents the area of the target detection box. According to the variables in the target state model, the state equation system can be obtained:

[0124]

[0125] corresponds to the variables at the current moment in the target state model, corresponds to the variables at the previous moment in the target state model, is the state transition time variable of the target state model, , represent the center position of the target time variable, , represent the length and width of the target detection box time variable, , represent the moving speed of the center position of the target time variable, , represent the time change rate of the length and width of the target detection box Time variable Indicating the area of the target detection box Time variable; 、 Indicating the center position of the target Time variable at -1 、 Indicating the length and width of the target detection box Time variable at -1 、 Indicating the moving speed of the target center position Time variable at -1 、 Indicating the time change rate of the length and width of the target detection box Time variable at -1 Indicating the area of the target detection box Time variable at -1.

[0126] The state transition matrix can be constructed based on the state variables constructed linearly :

[0127]

[0128] The state transition equation in matrix form is:

[0129]

[0130] Among them, Is the state variable at time k, Is the state variable at time k - 1.

[0131] The state transition matrix will predict the state variable at time k. Since the update of the state in the target tracking process only depends on the previous state and the process noise, the control input matrix in the original Kalman filter framework is set to 0. To simulate the random perturbations that may occur during the time growth process, introduce Representing the process noise vector in the state update equation. Is usually assumed to be white noise, with Gaussian distribution characteristics, its mean is zero, and the covariance is . Is the noise covariance matrix. Assume that the variances of the process noise of each state variable are respectively , , …… , then And Satisfy:

[0132]

[0133]

[0134] The above 1-9 subscripts correspond to 9 variables in the target state model.

[0135] Without considering the observed value at the current moment, predict the specific target state at the current moment, and construct the Kalman filter framework as follows:

[0136] Denote the prior state estimate value at time k as , that is, the result at time k predicted according to the optimal estimate at the previous moment (time k-1). Denote the posterior state estimate value at time k as . The prior estimation covariance matrix of the state variable predicting from time k-1 to time k, that is, The covariance matrix of is denoted as . Represent the posterior estimation covariance matrix of the state variable, that is, The covariance matrix of is denoted as .

[0137] The prediction equation set of the Kalman filter for target tracking is:

[0138]

[0139] is the posterior state estimate value at time k-1, that is, the updated state value;

[0140] Convert each state value of the prior estimation vector predicted and output in the current stage into the corresponding measurement value of the sensor column vector . The mapping relationship is denoted as the measurement matrix :

[0141]

[0142] Calculate the Kalman gain K to minimize the mean square error of the system state estimation:

[0143]

[0144] Where is the sensor covariance noise matrix, is the measurement matrix at time k.

[0145] III. Dynamically estimate the measurement noise covariance matrix

[0146] The embodiment of the present invention adopts an adaptive filtering method to dynamically estimate the measurement noise covariance matrix R through residual analysis to enhance the state estimation of the Kalman filter. This process uses a sliding window to collect residuals and calculates the residuals using the function given below :

[0147]

[0148] Among them , , , represents the residual vector, represents the measurement matrix.

[0149] Calculate the of the residuals in the N-frame sliding window, which reflects the average error magnitude on the sliding window, and the calculation is as follows:

[0150]

[0151] represents the residual of the residual vector in any frame of the sliding window.

[0152] The measurement noise covariance matrix R is adaptively corrected by transforming . The measurement noise covariance matrix update function is determined as:

[0153]

[0154] Where:

[0155]

[0156] Among them, the parameter λ adjusts the steepness of the function transition, and the parameter β determines the amplitude of the output. As shown in Figure 4 the variation of the measurement noise covariance matrix update function with respect to the parameter λ. The larger the value of λ, the steeper the curve.

[0157] The above error function erf(x) can smoothly scale the (Root Mean Square Error, RMSE) value, preventing sudden and significant changes, reducing the risk of the filter overreacting to these anomalies, enabling the Kalman filter to adapt to changes in the observation noise, thereby improving the robustness and accuracy of the entire filtering process.

[0158] IV. Update the target state estimate

[0159] As shown in Figure 3 after completing the dynamic estimation noise covariance matrix , update the current target state estimate, adjust and dynamically adjust the error covariance matrix according to the new measurement data, and update the state estimate variable. The update equation is as follows:

[0160]

[0161] is the identity matrix, with the same dimension as the state vector, mainly used for covariance update. is the observation vector, which contains the observations provided by the measurement system at the current moment.

[0162] Through the above equations, nine variables such as the position, velocity, and size of the detection box can be predicted in the multi-object detection task. When the environmental conditions change or occlusion factors cause unstable object detection in the image, the prediction results can be directly used to compensate for the object detection.

[0163] The Adaptive-SORT technology adopted in the present invention can effectively prevent sudden and significant changes in the measurement noise covariance matrix in the Kalman filter, reduce the risk of the filter overreacting to these anomalies, enable the Kalman filter to adapt to changes in the observation noise, and thus improve the robustness and accuracy of the entire filtering process.

[0164] In one embodiment, the three-dimensional spatial position calculation of the obtained target in step 600 includes:

[0165] Obtain the pixel coordinates of the target in the image plane ( ), and scale the pixel coordinates to obtain the image coordinates of the target ( ). As shown in the relative position diagram of the center of the UAV image and the target in Figure 5 , it is realized through the following formula:

[0166]

[0167] In the formula, represents the image length, represents the image width.

[0168] Then, according to knowledge such as perspective shadow, convert the coordinates to the camera coordinates of the target ( ), which is realized through the following formula:

[0169]

[0170] In the formula, represents the focal length of the camera.

[0171] According to data information such as the pitch angle of the camera carried by the UAV, convert the known position coordinates. The conversion is realized through the following formula:

[0172]

[0173] In the formula, represents the rotation matrix, including the pitch angle, yaw angle, and roll angle of the UAV; is the offset vector. In the embodiment of the present invention, the offset of the origin of the two coordinate systems is defaulted to 0.

[0174] Calculate the final coordinates of the target with the UAV as the origin in the world coordinate system ( ), as shown in the schematic diagram of target position resolution Figure 6 shown, which is realized through the following formula:

[0175]

[0176] In the formula, H represents the flight altitude of the UAV, represents the camera pitch angle, represents the angle between the line connecting the target object and the imaging point and the central axis of the camera.

[0177] In one embodiment, step 700 includes:

[0178] Input the target position parameters into the inner and outer loop controllers of the aircraft position and attitude, convert them into the control quantities of the flight control system, and achieve accurate, stable and real-time target locking and tracking according to the control quantities of the flight control system.

[0179] In addition, when the aircraft fails to identify a valid target, it will enter the self-stabilizing hover state or fly according to the instructions of the control terminal on the ground end.

[0180] In one embodiment, as Figure 7 shown, it further includes an aircraft target locking and tracking device for complex dynamic environments, specifically including an aircraft end 201, a communication module 202 and a ground end 203;

[0181] The aircraft end 201 includes:

[0182] An image acquisition module 2011, which is used to collect images of ground targets through a camera during the flight of the aircraft;

[0183] An image preprocessing module 2012, which is used to perform operations on the images collected by the image acquisition module 2011, including but not limited to de-distortion, noise reduction, and clarity optimization;

[0184] A target recognition module 2013, which is used to perform target recognition operations on the preprocessed images, send all possible target images and information to the communication module, and enter the target pre-selection state;

[0185] A target locking and tracking module 2014, which uses the Adaptive-SORT technology to perform accurate and stable image tracking and recognition on the predetermined target and give target information;

[0186] A position resolution module 2015, which is used to calculate the relative three-dimensional position between the target and the aircraft according to the obtained target information and flight altitude, update the target position parameters, and send the information to the flight attitude control module.

[0187] The flight attitude control module 2016 is used to achieve the self-stabilized flight of the aircraft and control the aircraft to track the target according to the target position parameters.

[0188] The flight attitude control module 2016 specifically includes a self-stabilized flight module 20161 and a flight tracking module 20162.

[0189] Furthermore, the communication module 202 is used to send all possible target images and information from the aircraft side to the ground side, and send the ground side selection result to the aircraft side.

[0190] The ground side 203 includes a ground side display module 2031 and a ground side target selection module 2032, which are used to display all possible target images and information, manually select the tracking target by the operator, and send the selected target information to the communication module.

[0191] The aircraft target locking and tracking system for a complex dynamic environment provided by the present invention will be described below. The aircraft target locking and tracking system for a complex dynamic environment described below can be correspondingly referred to the aircraft target locking and tracking method for a complex dynamic environment described above.

[0192] Figure 8 It is a schematic structural diagram of the aircraft target locking and tracking system for a complex dynamic environment provided by an embodiment of the present invention. As Figure 8 shown, it includes: an acquisition module 81, a preprocessing module 82, a determination module 83, a receiving module 84, a tracking module 85, a calculation module 86, and a tracking module 87, where:

[0193] The acquisition module 81 is used to obtain the height of the aircraft through a height sensor after the aircraft hovers stably in the air; the preprocessing module 82 is used to collect a target image of the ground target through a camera and preprocess the target image to obtain a preprocessed image; the determination module 83 is used to perform target recognition on the preprocessed image. If it is determined that a pending target is recognized and obtained, it enters the target preselection state; the receiving module 84 is used to send the pending target to the ground side and receive the selected target returned by the ground side; the tracking module 85 is used to adopt the Adaptive-SORT algorithm to perform image tracking recognition on the selected target and output target information; the calculation module 86 is used to calculate the relative three-dimensional position between the selected target and the aircraft according to the target information and the height of the aircraft, and update the target position parameters based on the relative three-dimensional position; the tracking module 87 is used to obtain the desired flight attitude from the target position parameters and control the flight attitude of the aircraft through a flight control system, so that the aircraft completes real-time target locking and tracking.

[0194] Figure 9 Illustrates a schematic diagram of the physical structure of an electronic device, such as Figure 9 shown. The electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940. Among them, the processor 910, the communications interface 920, and the memory 930 complete communication with each other through the communication bus 940. The processor 910 may call the logical instructions in the memory 930 to execute a method for locking and tracking a flying vehicle target in a complex dynamic environment. The method includes: after the flying vehicle hovers stably in the air, obtaining the altitude of the flying vehicle through an altitude sensor; collecting a target image of a ground target through a camera, and preprocessing the target image to obtain a preprocessed image; performing target recognition on the preprocessed image. If it is determined that a pending target is recognized and obtained, enter the target pre-selection state; sending the pending target to the ground end, and receiving the selected target returned by the ground end; using the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target, and outputting target information; according to the target information and the altitude of the flying vehicle, calculating to obtain the relative three-dimensional position between the selected target and the flying vehicle, and updating the target position parameters based on the relative three-dimensional position; obtaining an expected flight attitude from the target position parameters, and controlling the flight attitude of the flying vehicle through a flight control system, so that the flying vehicle completes real-time target locking and tracking.

[0195] In addition, when the logical instructions in the above-mentioned memory 930 can be implemented in the form of software function units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. And the foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0196] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0197] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for locking and tracking an aircraft target in a complex dynamic environment, characterized in that: include: After the aircraft is hovering in the air, the altitude of the aircraft is obtained through the altitude sensor; Capturing a target image of a ground target by a camera, and preprocessing the target image to obtain a preprocessed image; Performing target recognition on the preprocessed image, and if it is determined that the pending target is recognized, entering a target preselection state; Sending the pending target to a ground terminal, and receiving the selected target returned by the ground terminal; Adopting an adaptive sorting algorithm, performing image tracking and recognition on the selected target, and outputting target information; According to the target information and the height of the aircraft, a relative three-dimensional position between the selected target and the aircraft is calculated, and a target position parameter is updated based on the relative three-dimensional position; Obtaining a desired flight attitude from the target position parameters, and controlling the flight attitude of the aircraft through a flight control system so that the aircraft can complete real-time target locking and tracking; Wherein, the Adaptive-SORT algorithm includes: Constructing a Kalman filter state model of the selected target according to the target center position, the length and width of the target detection frame, the moving speed of the target center position, the time change rate of the length and width of the target detection frame, and the area of ​​the target detection frame of the selected target; A new moving video sequence frame is acquired, the new moving video sequence frame is updated based on a noise covariance matrix, and the target information is output.

2. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: Adaptive-SORT algorithm is used to perform image tracking and recognition on the selected target, and after outputting the target information, the method further includes: If it is determined that interference information exists or the target is lost due to object occlusion, the Adaptive-SORT algorithm is used to predict the object's motion trajectory, and the predicted target information is estimated based on the object's motion trajectory.

3. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: The target image of the ground target is collected by a camera, and the target image is preprocessed to obtain a preprocessed image, including: Obtaining the principal point coordinates of the target image and the focal length of the lens of the camera, determining an image coordinate system corresponding to the target image, and converting the image coordinate system into a camera coordinate system corresponding to the camera using the principal point coordinates and the focal length of the lens; Determine a coupling coefficient of the camera coordinate system, and adjust the horizontal coordinate of the camera coordinate system according to the coupling coefficient to obtain an adjusted horizontal coordinate; Acquire a radial distortion coefficient and a tangential distortion coefficient, and perform distortion correction on the camera coordinate system using the radial distortion coefficient, the tangential distortion coefficient, and the adjusted horizontal coordinate to obtain a corrected camera coordinate; Based on the principal point coordinates and the lens focal length, reversely transforming the corrected camera coordinates into the image coordinate system; Repeat the above operations of image coordinate system to camera coordinate system conversion, coordinate distortion correction and coordinate inverse conversion several times to obtain a distortion-corrected image; The clarity or contrast of the distortion-corrected image is optimized to obtain the preprocessed image.

4. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 3, characterized in that: Acquiring a radial distortion coefficient and a tangential distortion coefficient, and performing distortion correction on the camera coordinate system using the radial distortion coefficient, the tangential distortion coefficient, and the adjusted horizontal coordinate to obtain a corrected camera coordinate, including: The distortion radius is calculated by using the adjusted abscissa and ordinate; Calculating a radial distortion correction factor based on the distortion radius and the radial distortion coefficient; Calculating a tangential distortion correction factor according to the tangential distortion coefficient, the distortion radius, the adjusted abscissa and the ordinate; Based on the radial distortion correction factor and the tangential distortion correction factor, the adjusted horizontal coordinate and the vertical coordinate are corrected to obtain the corrected camera coordinates.

5. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: Performing target recognition on the preprocessed image includes: The improved YOLOv11 model is used in combination with the large convolution kernel attention mechanism to build a target recognition model; The preprocessed image is input into the target recognition model, and the target recognition result is output.

6. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: Adaptive-SORT algorithm is used to perform image tracking and recognition on the selected target, and target information is output, including: Determine the moving video sequence frame corresponding to the selected target, and use the Hungarian matching algorithm to obtain a subsequent frame image of the previous frame image in the moving video sequence frame, wherein n detection frames in the previous frame image and m detection frames in the subsequent frame image have an optimal matching estimate; Calculate the intersection and union ratio of any detection box in n detection boxes and any detection box in m detection boxes, and construct it from multiple intersection and union ratios dimensional cost matrix; Constructing a Kalman filter state model of the selected target according to the target center position, the length and width of the target detection frame, the moving speed of the target center position, the time change rate of the length and width of the target detection frame, and the area of ​​the target detection frame of the selected target; Constructing a state transfer equation of the selected target Kalman filter state model, wherein the state transfer equation includes a state transfer matrix, and the state transfer matrix predicts the state variable at time k according to the state variable at time k-1 and the process noise, wherein the process noise is Gaussian distributed white noise, and the noise covariance matrix corresponding to the Gaussian distributed white noise is composed of the variance of the process noise of each state variable; Determine the a priori state estimation value, the a priori state estimation covariance matrix, the a posteriori state estimation value and the a posteriori state estimation covariance matrix of the state variable at the k moment, obtain the Kalman filter prediction equation group for target tracking, predict the output a priori estimation vector by the Kalman filter prediction equation group, convert each state value of the a priori estimation vector into a measurement matrix, calculate the Kalman gain based on the measurement matrix, and minimize the mean square error of the system state estimation; Collect sliding window residuals of each frame through a sliding window, calculate the residual root mean square error value in the N-frame sliding window, and use an error function to transform and adaptively correct the residual root mean square error value to obtain a noise covariance matrix; A new moving video sequence frame is acquired, the new moving video sequence frame is updated based on the noise covariance matrix, and the target information is output.

7. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: The method further comprises: calculating a relative three-dimensional position between the selected target and the aircraft according to the target information and the height of the aircraft, and updating the target position parameters based on the relative three-dimensional position, including: Obtaining pixel coordinates of the target information in the image plane; Scaling the pixel coordinates according to the scaling ratio, the image length and the image width to obtain image coordinates; Converting the image coordinates to camera coordinates based on perspective shading and camera focal length; According to the rotation matrix and offset vector of the camera, the camera coordinates are rotated to obtain the rotated coordinates, wherein the rotation matrix includes the pitch angle, yaw angle and roll angle of the aircraft; The angle between the line connecting the target object and the imaging point and the central axis of the camera is obtained, and the rotated coordinates are converted into the final coordinates of the target in the world coordinate system with the aircraft as the origin, in combination with the aircraft height and the pitch angle.

8. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: The desired flight attitude is obtained from the target position parameters, and the flight attitude of the aircraft is controlled by a flight control system so that the aircraft can complete real-time target locking and tracking, including: Input the target position parameters into the inner and outer loop controllers of the aircraft position and attitude, and convert them into flight control system control quantities; Real-time target locking and tracking is completed based on the flight control system control quantity.

9. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: After performing target recognition on the preprocessed image, the method further includes: If it is determined that no valid target is identified, the aircraft performs self-stabilizing hovering, or flies according to the control instructions of the ground end.

10. An aircraft target locking and tracking system for complex dynamic environments, based on the aircraft target locking and tracking method for complex dynamic environments as claimed in any one of claims 1 to 9, characterized in that: include: The acquisition module is used to obtain the altitude of the aircraft through the altitude sensor after the aircraft is hovering in the air in a self-stable manner; A preprocessing module, used to collect a target image of a ground target through a camera, and preprocess the target image to obtain a preprocessed image; A determination module is used to perform target recognition on the preprocessed image, and if a pending target is determined to be recognized, enter a target preselection state; A receiving module, used for sending the pending target to a ground terminal and receiving a selected target returned by the ground terminal; A tracking module, used to use the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target and output target information; A calculation module, used for calculating the relative three-dimensional position of the selected target and the aircraft according to the target information and the height of the aircraft, and updating the target position parameters based on the relative three-dimensional position; The tracking module is used to obtain the desired flight attitude from the target position parameters, and control the flight attitude of the aircraft through a flight control system so that the aircraft can complete real-time target locking and tracking.

Images