Aircraft target locking and tracking method and system for complex dynamic environment

Abstract

The invention provides an aircraft target locking and tracking method and system for a complex dynamic environment. The method comprises the steps that an aircraft obtains the flight height through a height sensor; carrying out image preprocessing on a ground target and then carrying out target identification, and if a possible target is identified, entering a target pre-selection state; sending all possible target images and information to a ground end; the method comprises the following steps: carrying out image tracking identification on a predetermined target by adopting Adaptive-SORT to give target information; calculating a relative three-dimensional position of the target and the aircraft according to the obtained target information and the flight height, and updating a target position parameter; and obtaining an expected flight attitude according to the target position parameter, controlling the flight attitude of the aircraft, and locking and tracking the predetermined target by the aircraft. The aircraft provided by the invention can autonomously track and identify a specified target in a complex dynamic environment, and can predict the motion track of the target and ensure the tracking stability when the aircraft encounters a shelter or other interferences.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to an aircraft target locking and tracking method and system for a complex dynamic environment. Background Art

[0002] With the rapid development of aircraft technology, especially UAVs, UAVs are increasingly widely used in military, civil, and commercial fields due to their low cost, simple operation, light weight and portability. High-speed target tracking UAVs have broad application prospects in public security fields such as military reconnaissance, fugitive tracking, fire rescue, and automatic follow-up shooting, due to their excellent maneuverability and rapid deployment capabilities.

[0003] However, although the existing high-speed tracking drones can achieve accurate identification and real-time tracking, they are very likely to lose target tracking in complex environments when encountering interference such as obstructions or sudden disappearance of the target, thus interrupting the tracking task. In addition, the existing aircraft target locking and tracking methods are very likely to fail to lock and track targets when multiple similar targets appear at the same time, and there are practical problems such as unstable performance, poor anti-interference ability, and low robustness.

[0004] Therefore, developing an aircraft target locking and tracking method for complex dynamic environments has great application value and significance. Summary of the invention

[0005] The present invention provides a method and system for locking and tracking an aircraft target in a complex dynamic environment, so as to solve the defects of aircraft in the field of target tracking in the prior art, realize locking and tracking of the target by the aircraft in a complex environment, and predict the target motion trajectory in real time to prevent the target tracking from being lost.

[0006] In a first aspect, the present invention provides a method for locking and tracking an aircraft target in a complex dynamic environment, comprising: After the aircraft is hovering in the air, the altitude of the aircraft is obtained through the altitude sensor; Capturing a target image of a ground target by a camera, and preprocessing the target image to obtain a preprocessed image; Performing target recognition on the preprocessed image, and if it is determined that the pending target is recognized, entering a target preselection state; Sending the pending target to a ground terminal, and receiving the selected target returned by the ground terminal; Adopting the Adaptive-SORT algorithm, performing image tracking and recognition on the selected target, and outputting target information; According to the target information and the height of the aircraft, a relative three-dimensional position between the selected target and the aircraft is calculated, and a target position parameter is updated based on the relative three-dimensional position; The desired flight attitude is obtained from the target position parameters, and the flight attitude of the aircraft is controlled by a flight control system so that the aircraft can complete real-time target locking and tracking.

[0007] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, an Adaptive-SORT algorithm is used to perform image tracking and recognition on the selected target, and after outputting the target information, the method further includes: If it is determined that interference information exists or the target is lost due to object occlusion, the Adaptive-SORT algorithm is used to predict the object's motion trajectory, and the predicted target information is estimated based on the object's motion trajectory.

[0008] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, a target image of a ground target is collected by a camera, and the target image is preprocessed to obtain a preprocessed image, including: Obtaining the principal point coordinates of the target image and the focal length of the lens of the camera, determining an image coordinate system corresponding to the target image, and converting the image coordinate system into a camera coordinate system corresponding to the camera using the principal point coordinates and the focal length of the lens; Determine a coupling coefficient of the camera coordinate system, and adjust the horizontal coordinate of the camera coordinate system according to the coupling coefficient to obtain an adjusted horizontal coordinate; Acquire a radial distortion coefficient and a tangential distortion coefficient, and perform distortion correction on the camera coordinate system using the radial distortion coefficient, the tangential distortion coefficient, and the adjusted horizontal coordinate to obtain a corrected camera coordinate; Based on the principal point coordinates and the lens focal length, reversely transforming the corrected camera coordinates into the image coordinate system; Repeat the above operations of image coordinate system to camera coordinate system conversion, coordinate distortion correction and coordinate inverse conversion several times to obtain a distortion-corrected image; The clarity or contrast of the distortion-corrected image is optimized to obtain the preprocessed image.

[0009] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, a radial distortion coefficient and a tangential distortion coefficient are obtained, and the camera coordinate system is subjected to distortion correction using the radial distortion coefficient, the tangential distortion coefficient and the adjusted horizontal coordinate to obtain the corrected camera coordinate, including: The distortion radius is calculated by using the adjusted abscissa and ordinate; Calculating a radial distortion correction factor based on the distortion radius and the radial distortion coefficient; Calculating a tangential distortion correction factor according to the tangential distortion coefficient, the distortion radius, the adjusted abscissa and the ordinate; Based on the radial distortion correction factor and the tangential distortion correction factor, the adjusted horizontal coordinate and the vertical coordinate are corrected to obtain the corrected camera coordinates.

[0010] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, target recognition is performed on the preprocessed image, including: The improved YOLOv11 model is used in combination with the large convolution kernel attention mechanism to build a target recognition model; The preprocessed image is input into the target recognition model, and the target recognition result is output.

[0011] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, an adaptive sorting Adaptive-SORT algorithm is used to perform image tracking and recognition on the selected target and output target information, including: Determine the moving video sequence frame corresponding to the selected target, and use the Hungarian matching algorithm to obtain a subsequent frame image of the previous frame image in the moving video sequence frame, wherein n detection frames in the previous frame image and m detection frames in the subsequent frame image have an optimal matching estimate; Calculate the intersection and union ratio of any detection box in n detection boxes and any detection box in m detection boxes, and construct it from multiple intersection and union ratios dimensional cost matrix; Constructing a Kalman filter state model of the selected target according to the target center position, target detection frame length and width, target center position moving speed, target detection frame length and width time change rate and target detection frame area of ​​the selected target; Constructing a state transfer equation of the selected target Kalman filter state model, wherein the state transfer equation includes a state transfer matrix, and the state transfer matrix predicts the state variable at time k according to the state variable at time k-1 and the process noise, wherein the process noise is Gaussian distributed white noise, and the noise covariance matrix corresponding to the Gaussian distributed white noise is composed of the variance of the process noise of each state variable; Determine the a priori state estimation value, the a priori state estimation covariance matrix, the a posteriori state estimation value and the a posteriori state estimation covariance matrix of the state variable at the k moment, obtain the Kalman filter prediction equation group for target tracking, predict the output a priori estimation vector by the Kalman filter prediction equation group, convert each state value of the a priori estimation vector into a measurement matrix, calculate the Kalman gain based on the measurement matrix, and minimize the mean square error of the system state estimation; Collect sliding window residuals of each frame through a sliding window, calculate the residual root mean square error value in the N-frame sliding window, and use an error function to transform and adaptively correct the residual root mean square error value to obtain a noise covariance matrix; A new moving video sequence frame is acquired, the new moving video sequence frame is updated based on the noise covariance matrix, and the target information is output.

[0012] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, the relative three-dimensional position between the selected target and the aircraft is calculated according to the target information and the aircraft altitude, and the target position parameters are updated based on the relative three-dimensional position, including: Obtaining pixel coordinates of the target information in the image plane; Scaling the pixel coordinates according to the scaling ratio, the image length and the image width to obtain image coordinates; Converting the image coordinates to camera coordinates based on perspective shading and camera focal length; According to the rotation matrix and offset vector of the camera, the camera coordinates are rotated to obtain the rotated coordinates, wherein the rotation matrix includes the pitch angle, yaw angle and roll angle of the aircraft; The angle between the line connecting the target object and the imaging point and the central axis of the camera is obtained, and the rotated coordinates are converted into the final coordinates of the target in the world coordinate system with the aircraft as the origin, in combination with the aircraft height and the pitch angle.

[0013] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, a desired flight attitude is obtained from the target position parameters, and the flight attitude of the aircraft is controlled by a flight control system so that the aircraft completes real-time target locking and tracking, including: Input the target position parameters into the inner and outer loop controllers of the aircraft position and attitude, and convert them into flight control system control quantities; Real-time target locking and tracking is completed based on the flight control system control quantity.

[0014] According to a method for locking and tracking an aircraft target in a complex dynamic environment provided by the present invention, after performing target recognition on the preprocessed image, the method further includes: If it is determined that no valid target is identified, the aircraft performs self-stabilizing hovering, or flies according to the control instructions of the ground end.

[0015] In a second aspect, the present invention further provides an aircraft target locking and tracking system for a complex dynamic environment, comprising: The acquisition module is used to obtain the altitude of the aircraft through the altitude sensor after the aircraft is hovering in the air in a self-stable manner; A preprocessing module, used to collect a target image of a ground target through a camera, and preprocess the target image to obtain a preprocessed image; A determination module is used to perform target recognition on the preprocessed image, and if a pending target is determined to be recognized, enter a target preselection state; A receiving module, used for sending the pending target to a ground terminal and receiving a selected target returned by the ground terminal; A tracking module, used to use the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target and output target information; A calculation module, used for calculating the relative three-dimensional position of the selected target and the aircraft according to the target information and the height of the aircraft, and updating the target position parameters based on the relative three-dimensional position; The tracking module is used to obtain the desired flight attitude from the target position parameters, and control the flight attitude of the aircraft through a flight control system so that the aircraft can complete real-time target locking and tracking.

[0016] Compared with the prior art, the present invention has the following beneficial effects: On the one hand, the present invention realizes the continuous and stable tracking of the high-speed tracking UAV in a complex dynamic environment, and can predict the motion trajectory of the locked target in real time, so that the UAV can still dynamically adjust its own position according to the predicted trajectory in the face of long-term occlusion, target loss, etc., so that it can continue to lock and track the predetermined target after contacting the interference environment, thereby improving the stability of target locking and tracking; On the other hand, the present invention adopts the new Adaptive-SORT technology, which can continue to stably lock and track the established target when multiple similar targets appear at the same time, and has strong anti-interference ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of the process of locking and tracking an aircraft target in a complex dynamic environment provided by the present invention; Figure 2 It is a schematic diagram of the image dedistortion camera calibration plate provided by the present invention; Figure 3 is an aircraft control flow chart provided by the present invention; Figure 4It is a schematic diagram of the change of the measurement noise covariance matrix update function of the Kalman filter provided by the present invention relative to the parameter λ; Figure 5 It is a relative position diagram of the drone image center and the target provided by the present invention; Figure 6 It is a target position solution principle diagram provided by the present invention; Figure 7 It is a structural schematic diagram of an aircraft target locking and tracking system device for complex dynamic environments provided by the present invention; Figure 8 It is a structural schematic diagram of an aircraft target locking and tracking system for complex dynamic environments provided by the present invention; Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 FIG. 1 is a flow chart of an aircraft target locking and tracking in a complex dynamic environment provided by an embodiment of the present invention. Figure 1 As shown, including: Step 100: After the aircraft is hovering in the air in a self-stable manner, the altitude of the aircraft is obtained through an altitude sensor; Step 200: collecting a target image of a ground target through a camera, and preprocessing the target image to obtain a preprocessed image; Step 300: performing target recognition on the preprocessed image, and if it is determined that the pending target is recognized, entering the target pre-selection state; Step 400: sending the pending target to the ground terminal, and receiving the selected target returned by the ground terminal; Step 500: using the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target and output target information; Step 600: Calculate the relative three-dimensional position between the selected target and the aircraft according to the target information and the aircraft height, and update the target position parameters based on the relative three-dimensional position; Step 700: Obtain a desired flight attitude from the target position parameters, and control the flight attitude of the aircraft through a flight control system so that the aircraft can complete real-time target locking and tracking.

[0021] Specifically, in an embodiment of the present invention, the aircraft obtains the flight altitude through an altitude sensor; performs image acquisition and preprocessing of ground targets through a camera; performs target recognition on the preprocessed image, and if a possible target is recognized, enters a target preselection state; all possible target images and information are sent to the ground end, and the ground end operator manually selects the tracking target; uses adaptive sorting Adaptive-SORT technology to perform image tracking and recognition on the predetermined target, and provides target information; based on the target information and the flight altitude, the relative three-dimensional position of the target and the aircraft is calculated, and the target position parameters are updated; based on the target position parameters, the desired flight attitude is obtained, and the flight attitude of the aircraft is controlled by the flight control system to enable the aircraft to lock and track the predetermined target.

[0022] The aircraft in the present invention can autonomously track and identify a designated target in a complex dynamic environment, and when encountering obstructions or other interferences, the aircraft can predict the target's motion trajectory to ensure tracking stability.

[0023] In one embodiment, the aircraft in step 100 achieves self-stabilized hovering in the air, and adopts tightly coupled LiDAR-Inertial Odometry (LIO) technology to achieve self-stabilized hovering of the UAV in the absence of Global Navigation Satellite System (GNSS) signals.

[0024] In one embodiment, the preprocessing of the image in step 200 includes an image dedistortion operation, including the following operations: The first step is coordinate transformation, which transforms the coordinates in the image coordinate system Transform to camera coordinate system , achieved through the following formula:

[0025]

[0026] Among them, principal is the main point of the image, is the origin of the coordinates corresponding to the principal point of the image, focal is the focal length of the lens, is the origin of the coordinates corresponding to the focal length of the lens, is any pixel in the image.

[0027] Then adjust the X[i] coordinate according to the coupling coefficient (skew) to remove the non-orthogonality of the x-axis and y-axis of the camera. The adjusted coordinate is:

[0028] The radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2) are further used for distortion correction, which is performed by the following steps: (a) Calculate the distortion radius temp:

[0029] (b) Calculate the radial distortion correction factor tempKr:

[0030] (c) Calculate the tangential distortion and correct the coordinates (deltax, deltay) according to the tangential distortion coefficient:

[0031]

[0032] (d) Perform distortion correction on X[i] and Y[i] to obtain the corrected camera coordinate system coordinates ( , ):

[0033]

[0034] Finally, the rectified camera coordinate system coordinates are converted back to the image coordinate system using the following formula:

[0035]

[0036] After the coordinate conversion is completed, one or more image processing algorithms are used to optimize the clarity or contrast of the corrected image. The algorithm can be an existing mature image optimization algorithm when it is specifically implemented.

[0037] In the embodiment of the present invention, the above-mentioned conversion from the image coordinate system to the camera coordinate system, distortion correction and coordinate inverse conversion operations are all repeated n times, and n is usually taken as 3 to 5 to enhance the distortion correction effect.

[0038] It should be noted that the above-mentioned camera principal point coordinates principal, lens focal length focal, distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2) are achieved through camera calibration and can be obtained by shooting a specific calibration plate, such as Figure 2As shown, Figure 2 The black and white flat plate with a fixed-pitch pattern array in the middle is the calibration plate. By photographing the flat plate with a fixed-pitch pattern array with a camera and calculating with the calibration algorithm, we can derive the camera's geometric model and other related parameters, thereby obtaining high-precision measurement and reconstruction results.

[0039] In one embodiment, step 300 includes: The improved YOLOv11 model is used in combination with the large convolution kernel attention mechanism to build a target recognition model; The preprocessed image is input into the target recognition model, and the target recognition result is output.

[0040] In an embodiment of the present invention, a target recognition model based on deep learning is applied to the preprocessed image data. The target recognition model is an improved YOLOv11 model, and a large convolution kernel attention mechanism is introduced. The large convolution kernel attention mechanism is adopted to enable the model to effectively focus on important features in the image, enhance the model's recognition ability for small targets at a distance and its adaptability to complex backgrounds. Based on the YOLOv11 model, the present invention introduces Single-Head Self-Attention (SHSA) to solve problems such as small targets and occlusions in the target recognition process, thereby improving the target recognition performance of the YOLOv11 model.

[0041] It should be noted that the YOLOv11 model is the latest YOLO version released by the Ultralytics team in September 2024. Compared with other historical YOLO versions, it has further enhanced the feature extraction performance, adopted an improved back-end and neck architecture, and enhanced the feature extraction capability, thereby achieving more accurate object detection and more complex task execution; optimized processing efficiency and speed, introduced improved architecture design and optimized training process, achieved faster processing speed, and maintained the best balance between accuracy and performance; adopted more precise and fewer parameters, had strong cross-environment adaptability, and could be seamlessly deployed in various environments, supporting multiple tasks, whether it is target detection, instance segmentation, image classification, posture estimation or directional object detection, all have excellent performance.

[0042] In one embodiment, in step 400, the identified pending target is sent to the ground terminal, and the communication module used includes a control terminal and a data transmission terminal, both of which support multiple protocols, such as Wi-Fi, 4G, 5G and LoRa, to adapt to different communication application scenarios.

[0043] In one embodiment, step 500 includes: 1. Multi-target detection results of the forward and backward moving video sequence frames corresponding to the selected target First, the video sequence frames are processed using the tracking model to generate a bounding box for each potential dynamic target object detected by the target recognition model in the above embodiment, and the specific bounding box for subsequent tracking is determined by manually selecting the predetermined target. Here, the Hungarian algorithm is introduced to obtain the optimal estimate of the correspondence between multiple bounding boxes of objects in the previous frame and the next frame, and then determine the corresponding first bounding box. The first The prediction box in the frame, such as Figure 3 The steps shown in the series of processing flow branches after the target is identified. Here, Figure 3 The complete control process of the aircraft is described in detail. First, the UAV performs image acquisition. If the target is identified, it enters the subsequent target identification control process. Otherwise, the UAV enters self-stabilized hovering or flies according to the instructions of the ground control terminal. In the target identification control process, the Hungarian algorithm is used to match the current state of the UAV with the corresponding target. The ground terminal manually selects the target to be tracked in the corresponding target, thereby constructing the selected target Kalman filter state model to determine whether the UAV detects the target to be tracked. If the target is detected, the dynamic estimation noise covariance matrix R is established. If the target is not detected, it is determined that the target is blocked or failed to be tracked successfully, so as to predict the target based on the previous state that can be detected, compensate the current target state estimate, and use the dynamic estimation noise covariance matrix R to update the current target state estimate. Then, the relative position of the target and the UAV is calculated through the target information and flight altitude. Finally, the target is locked and tracked, and information is sent to the UAV to control the flight attitude of the UAV. The flight control module on the UAV updates the flight position of the UAV.

[0044] The current frame image has n detection frames , there are m detection boxes in the next frame image ,structure Cost Matrix , where each element Indicates that the i-th detection frame of the previous frame and the jth detection box in the next frame The cost of matching. The cost is measured using the intersection over union (IOU) between the detection boxes, that is:

[0045] At this time, the cost matrix The zero elements in correspond to the optimal solution obtained by the Hungarian matching algorithm as the basis for subsequent Kalman filtering, ensuring that each state variable corresponds to the corresponding target. If no target is detected, the new target state is predicted based on the previous target state through the linear Kalman filter framework and compensated to the current image.

[0046] It is understandable that the cost matrix The zero elements in represent the "minimum cost" match. Through a series of row subtractions, column subtractions, and other matrix operations, we try to create and maintain as many zero elements as possible in the matrix. The more zero elements there are, the more optimal matches there are to choose from. Finally, a non-conflicting match set is selected from the zero elements, that is, there is no pair of zero elements in the same row or column for matching, and the algorithm ends.

[0047] 2. Establish a state model based on Kalman filtering The target state model The description is as follows:

[0048] in , represents the target center position, , Indicates the length and width of the target detection box, , Indicates the moving speed of the target center position, , Represents the time rate of change of the length and width of the target detection box Represents the area of ​​the target detection box. According to the variables in the target state model, the state equation group can be obtained:

[0049] Corresponding to the current moment variable in the target state model, Corresponding to the previous moment variable in the target state model, is the state transition time variable of the target state model, , Indicates the target center position Time variables, , Indicates the length and width of the target detection box Time variables, , Indicates the moving speed of the target center position Time variables, , Indicates the time rate of change of the length and width of the target detection box Time variables, Represents the area of ​​the target detection box Time variables; , Indicates the target center position -1 moment variable, , Indicates the length and width of the target detection box -1 moment variable, , Indicates the moving speed of the target center position -1 moment variable, , Indicates the time rate of change of the length and width of the target detection box -1 moment variable, Represents the area of ​​the target detection box -1 moment variable.

[0050] The state transfer matrix can be constructed based on the linearly constructed state variables :

[0051] The state transfer equation in matrix form is:

[0052] in, is the state variable at time k, is the state variable at time k-1.

[0053] The state transfer matrix will predict the state variables at time k. Since the state update in the target tracking process depends only on the previous state and process noise, the control input matrix in the original Kalman filter framework is set to 0. In order to simulate the random disturbance that may occur during the time growth, we introduce Represents the process noise vector in the state update equation. It is usually assumed to be white noise with Gaussian distribution characteristics, with a mean of zero and a covariance of . is the noise covariance matrix. Assume that the variance of the process noise of each state variable is , ,…… ,but and satisfy:

[0054]

[0055] The above subscripts 1-9 correspond to the 9 variables in the target state model.

[0056] The Kalman filter framework is constructed as follows to predict the specific target state at the current moment without considering the observed value at the current moment: The prior state estimate at time k is recorded as , that is, the result at time k predicted based on the optimal estimate at the previous time (time k-1). The posterior state estimate at time k is recorded as The prior estimated covariance matrix of the state variable at time k-1 for prediction at time k is The covariance matrix of . Represents the posterior estimated covariance matrix of the state variables, that is The covariance matrix of .

[0057] The prediction equations of the Kalman filter for target tracking are:

[0058] is the posterior state estimate at time k-1, that is, the updated state value; The prior estimate vector of the current stage prediction output Each state value is converted into a sensor column vector The mapping relationship of the corresponding measurement values ​​is recorded as the measurement matrix :

[0059] Calculate the Kalman gain K to minimize the mean square error of the system state estimate:

[0060] in is the sensor covariance noise matrix, is the measurement matrix at time k.

[0061] 3. Dynamic Estimation of Measurement Noise Covariance Matrix The embodiment of the present invention adopts an adaptive filtering method to dynamically estimate the measurement noise covariance matrix R through residual analysis to enhance the state estimation of the Kalman filter. The process uses a sliding window to collect residuals and uses the function given below to calculate the residuals :

[0062] in , , , represents the residual vector, represents the measurement matrix.

[0063] Calculate the residual in the N-frame sliding window , which reflects the average error size on the sliding window, is calculated as follows:

[0064] Represents the residual in any frame sliding window The residual vector of .

[0065] The measurement noise covariance matrix R is obtained by Perform adaptive transformation correction. The measurement noise covariance matrix update function is determined as:

[0066] in:

[0067] The parameter λ adjusts the steepness of the function transition, while the parameter β determines the amplitude of the output, such as Figure 4 The measurement noise covariance matrix update function is shown in Figure 1. The larger the value of λ, the The steeper the curve.

[0068] The above error function erf(x) can smoothly scale the (Root Mean Square Error, RMSE) value to prevent This reduces the risk of the filter overreacting to these anomalies and enables the Kalman filter to adapt to changes in the observation noise, thereby improving the robustness and accuracy of the entire filtering process.

[0069] 4. Update target state estimate like Figure 3 The noise covariance matrix is ​​shown in the dynamic estimation. Then update the current target state estimate, adjust and dynamically adjust the error covariance matrix according to the new measurement data, update the state estimate variable, and the update equation is as follows:

[0070] is the unit matrix, with the same dimension as the state vector, and is mainly used for covariance update. is the observation vector, which contains the observation values ​​provided by the measurement system at the current time.

[0071] Through the above equations, nine variables such as the position, speed, and size of the detection box can be predicted in the multi-target detection task. When the target detection is unstable due to changes in environmental conditions or occlusion factors in the image, the prediction results can be directly used to compensate for the target detection.

[0072] The Adaptive-SORT technology adopted in the present invention can effectively prevent sudden and significant changes in the measurement noise covariance matrix in the Kalman filter, reduce the risk of the filter overreacting to these anomalies, and enable the Kalman filter to adapt to changes in observation noise, thereby improving the robustness and accuracy of the entire filtering process.

[0073] In one embodiment, performing three-dimensional spatial position calculation on the obtained target in step 600 includes: Get the pixel coordinates of the target in the image plane ( ), scale the pixel coordinates to get the target image coordinates ( ),like Figure 5 The relative position diagram of the drone image center and the target shown is achieved by the following formula:

[0074] In the formula, Indicates the image length, Indicates the image width.

[0075] Then, based on the knowledge of perspective shadows, the coordinates are converted into the target's camera coordinates ( ), which is achieved by the following formula:

[0076] In the formula, Indicates the focal length of the camera.

[0077] The known position coordinates are converted according to the pitch angle and other data information of the drone-mounted camera. The conversion is achieved through the following formula:

[0078] In the formula, Represents the rotation matrix, including the pitch angle, yaw angle and roll angle of the drone; is an offset vector. In the embodiment of the present invention, the offset of the origins of the two coordinate systems is assumed to be 0.

[0079] Calculate the final coordinates of the target in the world coordinate system with the drone as the origin ( ),like Figure 6 The target position solution principle diagram shown is realized by the following formula:

[0080] In the formula, H represents the flight altitude of the drone, Indicates the camera pitch angle, It represents the angle between the line connecting the target object and the imaging point and the central axis of the camera.

[0081] In one embodiment, step 700 includes: The target position parameters are input into the inner and outer loop controllers of the aircraft position and attitude, and converted into the control quantity of the flight control system. According to the control quantity of the flight control system, accurate and stable real-time target locking and tracking are achieved.

[0082] In addition, when the aircraft fails to identify a valid target, it will enter a self-stabilizing hovering state or fly according to the instructions of the control terminal on the ground.

[0083] In one embodiment, Figure 7 As shown, it also includes an aircraft target locking and tracking device for complex dynamic environments, which specifically includes an aircraft end 201, a communication module 202 and a ground end 203; The aircraft end 201 includes: The image acquisition module 2011 is used to acquire images of ground targets through a camera during the flight of the aircraft; The image preprocessing module 2012 is used to perform operations including but not limited to de-distortion, noise reduction, and definition optimization on the image acquired by the image acquisition module 2011; The target recognition module 2013 is used to perform target recognition operation on the pre-processed image, send all possible target images and information to the communication module, and enter the target pre-selection state; The target locking and tracking module 2014 uses Adaptive-SORT technology to accurately and stably track and identify the predetermined target and provide target information; The position calculation module 2015 is used to calculate the relative three-dimensional position of the target and the aircraft according to the target information and the flight altitude, update the target position parameters, and send the information to the flight attitude control module.

[0084] The flight attitude control module 2016 is used to realize the self-stabilizing flight of the aircraft and control the aircraft to track the target according to the target position parameters.

[0085] The flight attitude control module 2016 specifically includes a self-stabilizing flight module 20161 and a flight tracking module 20162 .

[0086] Furthermore, the communication module 202 is used to send all possible target images and information from the aircraft end to the ground end, and send the ground end selection result to the aircraft end.

[0087] The ground terminal 203 includes a ground terminal display module 2031 and a ground terminal target selection module 2032, which are used to display all possible target images and information, and manually select the tracking target through the operator, and send the selected target information to the communication module.

[0088] The following is a description of the aircraft target locking and tracking system for complex dynamic environments provided by the present invention. The aircraft target locking and tracking system for complex dynamic environments described below and the aircraft target locking and tracking method for complex dynamic environments described above can be referenced to each other.

[0089] Figure 8 FIG. 1 is a schematic diagram of a structure of an aircraft target locking and tracking system for a complex dynamic environment provided by an embodiment of the present invention. Figure 8 As shown, it includes: an acquisition module 81, a preprocessing module 82, a determination module 83, a receiving module 84, a tracking module 85, a solution module 86 and a tracking module 87, wherein: The acquisition module 81 is used to obtain the altitude of the aircraft through the altitude sensor after the aircraft is self-stabilized and hovering in the air; the preprocessing module 82 is used to collect the target image of the ground target through the camera, and preprocess the target image to obtain the preprocessed image; the determination module 83 is used to identify the target after the preprocessed image, and if it is determined that the pending target is identified and acquired, it enters the target pre-selection state; the receiving module 84 is used to send the pending target to the ground end, and receive the selected target returned by the ground end; the tracking module 85 is used to use the Adaptive-SORT algorithm to perform image tracking and identification on the selected target and output target information; the solving module 86 is used to solve the relative three-dimensional position of the selected target and the aircraft according to the target information and the aircraft altitude, and update the target position parameters based on the relative three-dimensional position; the tracking module 87 is used to obtain the desired flight attitude from the target position parameters, and control the flight attitude of the aircraft through the flight control system, so that the aircraft can complete real-time target locking and tracking.

[0090] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor (processor) 910, a communication interface (Communications Interface) 920, a memory (memory) 930 and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call the logic instructions in the memory 930 to execute the aircraft target locking and tracking method for a complex dynamic environment, the method comprising: after the aircraft is self-stabilized and hovering in the air, obtaining the aircraft altitude through an altitude sensor; collecting the target image of the ground target through a camera, and preprocessing the target image to obtain a preprocessed image; performing target recognition on the preprocessed image, and if it is determined that the pending target is recognized and obtained, entering a target pre-selection state; sending the pending target to the ground end, and receiving the selected target returned by the ground end; using the Adaptive-SORT algorithm, performing image tracking and recognition on the selected target, and outputting target information; according to the target information and the aircraft altitude, solving the relative three-dimensional position of the selected target and the aircraft, and updating the target position parameters based on the relative three-dimensional position; obtaining the desired flight attitude from the target position parameters, and controlling the flight attitude of the aircraft through the flight control system, so that the aircraft completes real-time target locking and tracking.

[0091] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for locking and tracking an aircraft target in a complex dynamic environment, characterized in that: include: After the aircraft is hovering in the air, the altitude of the aircraft is obtained through the altitude sensor; Capturing a target image of a ground target by a camera, and preprocessing the target image to obtain a preprocessed image; Performing target recognition on the preprocessed image, and if it is determined that the pending target is recognized, entering a target preselection state; Sending the pending target to a ground terminal, and receiving the selected target returned by the ground terminal; Adopting an adaptive sorting algorithm, performing image tracking and recognition on the selected target, and outputting target information; According to the target information and the height of the aircraft, a relative three-dimensional position between the selected target and the aircraft is calculated, and a target position parameter is updated based on the relative three-dimensional position; The desired flight attitude is obtained from the target position parameters, and the flight attitude of the aircraft is controlled by a flight control system so that the aircraft can complete real-time target locking and tracking.

2. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: Adaptive-SORT algorithm is used to perform image tracking and recognition on the selected target, and after outputting the target information, the method further includes: If it is determined that interference information exists or the target is lost due to object occlusion, the Adaptive-SORT algorithm is used to predict the object's motion trajectory, and the predicted target information is estimated based on the object's motion trajectory.

3. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: The target image of the ground target is collected by a camera, and the target image is preprocessed to obtain a preprocessed image, including: Obtaining the principal point coordinates of the target image and the focal length of the lens of the camera, determining an image coordinate system corresponding to the target image, and converting the image coordinate system into a camera coordinate system corresponding to the camera using the principal point coordinates and the focal length of the lens; Determine a coupling coefficient of the camera coordinate system, and adjust the horizontal coordinate of the camera coordinate system according to the coupling coefficient to obtain an adjusted horizontal coordinate; Acquire a radial distortion coefficient and a tangential distortion coefficient, and perform distortion correction on the camera coordinate system using the radial distortion coefficient, the tangential distortion coefficient, and the adjusted horizontal coordinate to obtain a corrected camera coordinate; Based on the principal point coordinates and the lens focal length, reversely transforming the corrected camera coordinates into the image coordinate system; Repeat the above operations of image coordinate system to camera coordinate system conversion, coordinate distortion correction and coordinate inverse conversion several times to obtain a distortion-corrected image; The clarity or contrast of the distortion-corrected image is optimized to obtain the preprocessed image.

4. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 3, characterized in that: Acquiring a radial distortion coefficient and a tangential distortion coefficient, and performing distortion correction on the camera coordinate system using the radial distortion coefficient, the tangential distortion coefficient, and the adjusted horizontal coordinate to obtain a corrected camera coordinate, including: The distortion radius is calculated by using the adjusted abscissa and ordinate; Calculating a radial distortion correction factor based on the distortion radius and the radial distortion coefficient; Calculating a tangential distortion correction factor according to the tangential distortion coefficient, the distortion radius, the adjusted abscissa and the ordinate; Based on the radial distortion correction factor and the tangential distortion correction factor, the adjusted horizontal coordinate and the vertical coordinate are corrected to obtain the corrected camera coordinates.

5. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: Performing target recognition on the preprocessed image includes: The improved YOLOv11 model is used in combination with the large convolution kernel attention mechanism to build a target recognition model; The preprocessed image is input into the target recognition model, and the target recognition result is output.

6. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: Adaptive-SORT algorithm is used to perform image tracking and recognition on the selected target, and target information is output, including: Determine the moving video sequence frame corresponding to the selected target, and use the Hungarian matching algorithm to obtain a subsequent frame image of the previous frame image in the moving video sequence frame, wherein n detection frames in the previous frame image and m detection frames in the subsequent frame image have an optimal matching estimate; Calculate the intersection and union ratio of any detection box in n detection boxes and any detection box in m detection boxes, and construct it from multiple intersection and union ratios dimensional cost matrix; Constructing a Kalman filter state model of the selected target according to the target center position, target detection frame length and width, target center position moving speed, target detection frame length and width time change rate and target detection frame area of ​​the selected target; Constructing a state transfer equation of the selected target Kalman filter state model, wherein the state transfer equation includes a state transfer matrix, and the state transfer matrix predicts the state variable at time k according to the state variable at time k-1 and the process noise, wherein the process noise is Gaussian distributed white noise, and the noise covariance matrix corresponding to the Gaussian distributed white noise is composed of the variance of the process noise of each state variable; Determine the a priori state estimation value, the a priori state estimation covariance matrix, the a posteriori state estimation value and the a posteriori state estimation covariance matrix of the state variable at the k moment, obtain the Kalman filter prediction equation group for target tracking, predict the output a priori estimation vector by the Kalman filter prediction equation group, convert each state value of the a priori estimation vector into a measurement matrix, calculate the Kalman gain based on the measurement matrix, and minimize the mean square error of the system state estimation; Collect sliding window residuals of each frame through a sliding window, calculate the residual root mean square error value in the N-frame sliding window, and use an error function to transform and adaptively correct the residual root mean square error value to obtain a noise covariance matrix; A new moving video sequence frame is acquired, the new moving video sequence frame is updated based on the noise covariance matrix, and the target information is output.

7. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: The method further comprises: calculating a relative three-dimensional position between the selected target and the aircraft according to the target information and the height of the aircraft, and updating the target position parameters based on the relative three-dimensional position, including: Obtaining pixel coordinates of the target information in the image plane; Scaling the pixel coordinates according to the scaling ratio, the image length and the image width to obtain image coordinates; Converting the image coordinates to camera coordinates based on perspective shading and camera focal length; According to the rotation matrix and offset vector of the camera, the camera coordinates are rotated to obtain the rotated coordinates, wherein the rotation matrix includes the pitch angle, yaw angle and roll angle of the aircraft; The angle between the line connecting the target object and the imaging point and the central axis of the camera is obtained, and the rotated coordinates are converted into the final coordinates of the target in the world coordinate system with the aircraft as the origin, in combination with the aircraft height and the pitch angle.

8. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: The desired flight attitude is obtained from the target position parameters, and the flight attitude of the aircraft is controlled by a flight control system so that the aircraft can complete real-time target locking and tracking, including: Input the target position parameters into the inner and outer loop controllers of the aircraft position and attitude, and convert them into flight control system control quantities; Real-time target locking and tracking is completed based on the flight control system control quantity.

9. The method for locking and tracking an aircraft target in a complex dynamic environment according to claim 1, characterized in that: After performing target recognition on the preprocessed image, the method further includes: If it is determined that no valid target is identified, the aircraft performs self-stabilizing hovering, or flies according to the control instructions of the ground end.

10. An aircraft target locking and tracking system for complex dynamic environments, based on the aircraft target locking and tracking method for complex dynamic environments as claimed in any one of claims 1 to 9, characterized in that: include: The acquisition module is used to obtain the altitude of the aircraft through the altitude sensor after the aircraft is hovering in the air in a self-stable manner; A preprocessing module, used to collect a target image of a ground target through a camera, and preprocess the target image to obtain a preprocessed image; A determination module is used to perform target recognition on the preprocessed image, and if a pending target is determined to be recognized, enter a target preselection state; A receiving module, used for sending the pending target to a ground terminal and receiving a selected target returned by the ground terminal; A tracking module, used to use the Adaptive-SORT algorithm to perform image tracking and recognition on the selected target and output target information; A calculation module, used for calculating the relative three-dimensional position of the selected target and the aircraft according to the target information and the height of the aircraft, and updating the target position parameters based on the relative three-dimensional position; The tracking module is used to obtain the desired flight attitude from the target position parameters, and control the flight attitude of the aircraft through a flight control system so that the aircraft can complete real-time target locking and tracking.

Images