UAV automatic target centering and tracking system, method, device and medium

Abstract

The present application provides a system, method, device and medium for automatic target centering and tracking of unmanned aerial vehicles. The system includes a target detection module, which is used to obtain real-time video streams through a camera, collect image and video data containing targets, and annotate the video data to generate target bounding boxes and category labels; a target tracking module, which is used to continuously track the position of the target in a video frame based on the output of the target detection module according to a Kalman filter and a Hungarian algorithm; a model predictive control module, which is used to establish a state space model, optimization problem formulas and constraints of the unmanned aerial vehicle; a sensor module, which is used to obtain the state information of the unmanned aerial vehicle and provide it to the model predictive control module; and a flight control module, which is used to adjust the flight attitude and position of the unmanned aerial vehicle so that the target always remains in the center of the camera's field of view, and can achieve accurate detection and stable tracking of the target in a complex environment.

Description

Technical Field

[0001] The present application relates to the field of drone inspection technology, and in particular to a drone automatic target centering and tracking system, method, electronic device and storage medium. Background Art

[0002] Among the infrastructure of modern cities, the power system is an important part of ensuring the normal operation of the city and the lives of residents. As the size of cities continues to expand, the power transmission network has become more and more complex. Power inspection is one of the important tasks to ensure the safe and stable operation of the power system. However, traditional power inspection mainly relies on manual inspection and fixed monitoring equipment. This method is not only inefficient, but also has certain safety risks and limitations due to the wide distribution of power lines and complex environments.

[0003] Currently, drones need to process a large amount of video data in real time during flight to detect and track targets. Traditional target detection and tracking algorithms have large computational complexity and are difficult to meet real-time requirements. This may result in high latency and affect the drone's response speed and control accuracy. Summary of the invention

[0004] In view of this, it is necessary to provide a drone automatic target centering and tracking system, method, electronic device and storage medium that can overcome at least one of the above defects.

[0005] In a first aspect, an embodiment of the present application provides an automatic target centering and tracking system for a drone, the system comprising:

[0006] The target detection module is used to obtain real-time video streams through the camera, collect image and video data containing the target, and annotate the video data to generate target bounding boxes and category labels;

[0007] The target tracking module is used to continuously track the position of the target in the video frame based on the output of the target detection module according to the Kalman filter and the Hungarian algorithm, wherein the target position prediction formula of the Kalman filter is:

[0008] x k∣k-1 =Fx k∣k-1 +Bu k

[0009] P k∣k-1 =FP k∣k-1 F T +Q

[0010] Among them, x k∣k-1 is the predicted state vector, P k∣k-1 is the predicted covariance matrix, F is the state transfer matrix, B is the control input matrix, u kis the control input, Q is the process noise covariance matrix;

[0011] The target tracking module is also used to update the position of the target according to the Kalman filter. The update formula of the Kalman filter is:

[0012] K k =P k∣k-1 H T (HP k∣k-1 H T +R) -1

[0013] x k∣k =x k∣k-1 +K k (z k -Hx k∣k-1 )

[0014] P k∣k =(IK k H)P k∣k-1

[0015] Among them, K k is the Kalman gain, H is the observation matrix, R is the measurement noise covariance matrix, z k is the measurement vector;

[0016] A model predictive control module, used to establish a state space model of the UAV, an optimization problem formula and constraints to achieve precise control of the UAV, wherein the optimization problem formula is used to minimize the state error and control input cost of the UAV in the prediction time domain, and the constraints are used to ensure the flight safety of the UAV and that the control input is within the physical capability range;

[0017] Among them, the state space model formula of the drone is:

[0018] x k+1 =Ax k +Bu k

[0019] Among them, x k is the state vector, u k is the control input vector, A is the state matrix, which describes the dynamic model of the state evolution of the UAV over time, and B is the input matrix, which describes how the control input affects the state change of the UAV;

[0020] The optimization problem formula is:

[0021]

[0022] Among them, x k∣tis the state predicted at time t at the kth moment, x r is the reference trajectory or expected state, u k∣t is the control input at the kth moment predicted at time t, Q is the weight matrix of the state error, which is used to adjust the influence of the state error on the objective function, R is the weight matrix of the control input, which is used to adjust the influence of the control input on the objective function, Q f is the weight matrix of the terminal state error, and N is the length of the prediction time domain;

[0023] A sensor module, the sensor module comprising: the camera, an inertial measurement unit and a Beidou positioning module, the sensor module is used to obtain the state information of the UAV and provide it to the model prediction control module;

[0024] The flight control module is used to adjust the flight attitude and position of the UAV so that the target always remains in the center of the camera's field of view.

[0025] In one embodiment, the object detection module applies the YOLO algorithm to output the position, size and confidence score of the bounding box.

[0026] In one embodiment, the constraints include state constraints and input constraints; the state constraints are used to limit the range of predicted states to ensure that the drone does not exceed physical and safety limitations during flight; the input constraints are used to limit the range of control inputs to ensure that control commands are within the physical capabilities of the drone.

[0027] In one embodiment, the state constraint formula is:

[0028] x min ≤x k∣t ≤x max

[0029] Among them, x min is the lower limit vector of the state variable, indicating the minimum allowed value of each state variable, x max is the upper limit vector of the state variables, indicating the maximum allowed value of each state variable;

[0030] The input constraint formula is:

[0031] u min ≤u k∣t ≤u max

[0032] Among them, u min is the lower limit vector of the control input, indicating the minimum allowable value of each control input, u max is the upper limit vector of the control input, indicating the maximum allowed value of each control input.

[0033] In one embodiment, the constraint condition further includes a terminal state constraint, and the terminal state constraint is used to ensure that the drone is in a safe and controllable state at the end of the prediction time domain;

[0034] The formula for the terminal state constraint is:

[0035] x N∣t ∈x f

[0036] Among them, x N∣t is the state vector predicted at time t at the Nth moment, x f is the allowed set of terminal states, indicating the conditions that must be satisfied in the prediction time domain terminal state.

[0037] In one embodiment, the UAV automatic target centering and tracking system further includes an error calculation unit, and the error calculation unit is used to calculate the error between the center of the target and the center of the image;

[0038] The formula of the error calculation unit is:

[0039] e x =x target -x center

[0040] e y =y target -y center

[0041] Among them, x target ,y target is the coordinate of the target center, x center ,y center , is the coordinate of the center of the image, e x , e y are the lateral and longitudinal errors.

[0042] In one embodiment, the flight control module is further used to adjust the thrust, pitch angle and yaw angle of the UAV according to the optimal control input output by the model prediction control module to adjust the flight attitude and position of the UAV.

[0043] In a second aspect, an embodiment of the present application provides a method for automatic target centering and tracking of a drone, which is applied to the automatic target centering and tracking system of a drone as described in the first aspect, comprising:

[0044] Acquire real-time video streams through cameras, collect image and video data containing targets, annotate the video data, and generate target bounding boxes and category labels;

[0045] The position of the target is continuously tracked in the video frame based on the image containing the target according to the Kalman filter and the Hungarian algorithm, wherein the target position prediction formula of the Kalman filter is:

[0046] x k∣k-1 =Fx k∣k-1 +Bu k

[0047] P k∣k-1 =FP k∣k-1 F T +Q

[0048] Among them, x k∣k-1 is the predicted state vector, P k∣k-1 is the predicted covariance matrix, F is the state transfer matrix, B is the control input matrix, u k is the control input, Q is the process noise covariance matrix;

[0049] The position of the target is updated according to the Kalman filter, and the updating formula of the Kalman filter is:

[0050] K k =P k∣k-1 H T (HP k∣k-1 H T +R) -1

[0051] x k∣k =x k∣k-1 +K k (z k -Hx k∣k-1 )

[0052] P k∣k =(IK k H)P k∣k-1

[0053] Among them, K k is the Kalman gain, H is the observation matrix, R is the measurement noise covariance matrix, z k is the measurement vector;

[0054] Establishing a state space model, an optimization problem formula and constraints of the UAV to achieve precise control of the UAV, wherein the optimization problem formula is used to minimize the state error and control input cost of the UAV in the prediction time domain, and the constraints are used to ensure the flight safety of the UAV and that the control input is within the physical capability range;

[0055] Among them, the state space model formula of the drone is:

[0056] x k+1=Ax k +Bu k

[0057] Among them, x k is the state vector, u k is the control input vector, A is the state matrix, which describes the dynamic model of the state evolution of the UAV over time, and B is the input matrix, which describes how the control input affects the state change of the UAV;

[0058] The optimization problem formula is:

[0059]

[0060] Among them, x k∣t is the state predicted at time t at the kth moment, x r is the reference trajectory or expected state, u k∣t is the control input at the kth moment predicted at time t, Q is the weight matrix of the state error, which is used to adjust the influence of the state error on the objective function, R is the weight matrix of the control input, which is used to adjust the influence of the control input on the objective function, Q f is the weight matrix of the terminal state error, and N is the length of the prediction time domain;

[0061] The flight attitude and position of the UAV are adjusted so that the target always remains in the center of the camera's field of view.

[0062] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the second aspect when executing the instructions.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, wherein the instructions instruct a device to execute as described in the second aspect.

[0064] The automatic target centering and tracking system, method, electronic device and storage medium provided by the implementation method of the present application can realize accurate detection and stable tracking of targets in complex environments. By integrating advanced target detection algorithms, target tracking modules of Kalman filter and Hungarian algorithm, and flight control modules based on model predictive control (MPC), the system can acquire and process data from cameras, inertial measurement units and Beidou positioning modules in real time to form a closed-loop control mechanism, thereby ensuring that the drone always keeps the target in the center of the camera's field of view during flight. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a module of an automatic target centering and tracking system for a drone provided in one embodiment of the present application.

[0066] Figure 2 A schematic flow chart of a method for automatic target centering and tracking of a drone provided in one embodiment of the present application.

[0067] Figure 3 A schematic diagram of an electronic device provided in one embodiment of the present application.

[0068] Main component symbols

[0069] UAV automatic target centering and tracking system10

[0070] Object detection module 110

[0071] Target tracking module 120

[0072] Model Predictive Control Module 130

[0073] Sensor module 140

[0074] Flight Control Module 150

[0075] Electronic devices 20

[0076] Processor 21

[0077] Memory 22

[0078] Method steps S100-S500 DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0080] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0081] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. Features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0082] Based on the implementations in this application, all other implementations obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] Among the infrastructure of modern cities, the power system is an important part of ensuring the normal operation of the city and the lives of residents. As the size of cities continues to expand, the power transmission network has become more and more complex. Power inspection is one of the important tasks to ensure the safe and stable operation of the power system. However, traditional power inspection mainly relies on manual inspection and fixed monitoring equipment. This method is not only inefficient, but also has certain safety risks and limitations due to the wide distribution of power lines and complex environments.

[0084] Currently, drones need to process a large amount of video data in real time during flight to detect and track targets. Traditional target detection and tracking algorithms have large computational complexity and are difficult to meet real-time requirements. This may result in high latency and affect the drone's response speed and control accuracy.

[0085] Therefore, the present application provides a drone automatic target centering and tracking system, a drone automatic target centering and tracking method, an electronic device and a storage medium that can accurately detect and stably track targets in complex environments. By integrating advanced target detection algorithms, a target tracking module with a Kalman filter and a Hungarian algorithm, and a flight control module based on model predictive control (MPC), the system can acquire and process data from a camera, an inertial measurement unit and a Beidou positioning module in real time, forming a closed-loop control mechanism, thereby ensuring that the drone always keeps the target in the center of the camera's field of view during flight.

[0086] Figure 1 1 is a schematic diagram of the system modules of the automatic target centering and tracking system 10 for unmanned aerial vehicles provided in one embodiment of the present application. Figure 1As shown, the UAV automatic target centering and tracking system 10 includes at least the following parts: a target detection module 110 , a target tracking module 120 , a model prediction control module 130 , a sensor module 140 and a flight control module 150 .

[0087] In the embodiment of the present application, the target detection module 110 is used to obtain a real-time video stream through a camera, collect image and video data containing the target, and annotate the video data to generate a target bounding box and category label.

[0088] In the embodiment of the present application, the target tracking module 120 is used to continuously track the position of the target in the video frame based on the output of the target detection module according to the Kalman filter and the Hungarian algorithm, wherein the target position prediction formula of the Kalman filter is:

[0089] x k∣k-1 =Fx k∣k-1 +Bu k

[0090] P k∣k-1 =FP k∣k-1 F T +Q

[0091] Among them, x k∣k-1 is the predicted state vector, P k∣k-1 is the predicted covariance matrix, F is the state transition matrix, which describes the dynamic model of the UAV's state evolution over time. B is the control input matrix, which describes how the control input affects the state change of the UAV. k is the control input and Q is the process noise covariance matrix.

[0092] In the embodiment of the present application, the target tracking module 120 is also used to update the position of the target according to the Kalman filter. The update formula of the Kalman filter is:

[0093] K k =P k∣k-1 H T (HP k∣k-1 H T +R) -1

[0094] x k∣k =x k∣k-1 +K k (z k -Hx k∣k-1 )

[0095] P k∣k =(IK k H)P k∣k-1

[0096] Among them, Kk is the Kalman gain, H is the observation matrix, R is the measurement noise covariance matrix, z k

[0097] is the measurement vector.

[0098] In an embodiment of the present application, the model predictive control module 130 is used to establish a state space model, optimization problem formula and constraints of the UAV to achieve precise control of the UAV. The optimization problem formula is used to minimize the state error and control input cost of the UAV in the prediction time domain, and the constraints are used to ensure the flight safety of the UAV and the control input is within the physical capability.

[0099] Among them, the state space model formula of the drone is:

[0100] x k+1 =Ax k +Bu k

[0101] Among them, x k is the state vector, u k is the control input vector, A is the state matrix, which describes the dynamic model of the UAV's state evolution over time, and B is the input matrix, which describes how the control input affects the state change of the UAV.

[0102] The optimization problem formula is:

[0103]

[0104] Among them, x k∣t is the state predicted at time t at the kth moment, indicating the target trajectory or state that the drone needs to track. r is the reference trajectory or expected state, u k∣t is the control input at the kth moment predicted at time t, Q is the weight matrix of the state error, which is used to adjust the impact of the state error on the objective function. A larger weight indicates that more emphasis is placed on minimizing the state error. R is the weight matrix of the control input, which is used to adjust the impact of the control input on the objective function. A larger weight indicates that more emphasis is placed on smoothing and energy saving of the control input. Q f is the weight matrix of the terminal state error. A larger weight indicates that the accuracy of the terminal state in the prediction time domain is more important. N is the length of the prediction time domain.

[0105] It is understandable that the choice of the length N of the prediction time domain depends on the requirements of the control task and the computing resources. A shorter prediction time domain has a faster calculation speed and is suitable for applications with high real-time requirements, but the prediction accuracy may be lower. A longer prediction time domain has a higher prediction accuracy, but the calculation amount is large, which is suitable for applications with abundant computing resources. In actual applications, the length of the prediction time domain can be adjusted according to the computing power of the drone chip.

[0106] In the embodiment of the present application, the sensor module 140 includes: a camera, an inertial measurement unit and a Beidou positioning module. The sensor module 140 obtains the status information of the drone and provides it to the model prediction control module.

[0107] In the embodiment of the present application, the flight control module 150 is used to adjust the flight attitude and position of the UAV so that the target always remains in the center of the camera's field of view.

[0108] In the embodiment of the present application, the object detection module 110 applies the YOLO algorithm to output the position, size and confidence score of the bounding box.

[0109] In the embodiment of the present application, the constraints include state constraints and input constraints; the state constraints are used to limit the range of the predicted state to ensure that the UAV does not exceed physical and safety limitations during flight; the input constraints are used to limit the range of control inputs to ensure that the control commands are within the physical capabilities of the UAV.

[0110] In the embodiment of the present application, the state constraint formula is:

[0111] x min ≤x k∣t ≤x max

[0112] Among them, x min is the lower limit vector of the state variables, indicating the minimum allowed value of each state variable, such as minimum height, minimum speed, etc. max is the upper limit vector of the state variables, indicating the maximum allowed value of each state variable, such as maximum height, maximum speed, etc.

[0113] It can be understood that state constraints are used to limit the range of predicted states, ensuring that the drone does not exceed physical and safety limits during flight. For example, state variables such as altitude, speed, and position must all be within a safe range.

[0114] In the embodiment of the present application, the input constraint formula is:

[0115] u min ≤u k∣t ≤u max

[0116] Among them, u min is the lower limit vector of the control input, indicating the minimum allowable value of each control input, such as minimum thrust, minimum speed, etc. max It is the upper limit vector of the control input, indicating the maximum allowable value of each control input, such as maximum thrust, maximum speed, etc.

[0117] It can be understood that input constraints are used to limit the range of control inputs and ensure that control commands are within the physical capabilities of the drone. For example, control variables such as motor speed and thrust must be within the hardware capabilities of the drone.

[0118] In an embodiment of the present application, the constraint conditions also include a terminal state constraint, which is used to ensure that the drone is in a safe and controllable state at the end of the prediction time domain.

[0119] The formula for the terminal state constraint is:

[0120] x N∣t ∈x f

[0121] Among them, x N∣t is the state vector predicted at time t at the Nth moment, x f is the allowed set of terminal states, indicating the conditions that must be satisfied in the prediction time domain terminal state.

[0122] It can be understood that the terminal state constraint is used to ensure that the state at the end of the prediction horizon satisfies certain conditions, such as a desired position or stable state. This helps ensure that the drone is in a safe and controllable state at the end of the prediction horizon.

[0123] It can be understood that by optimizing the problem formula and the constraints, the model predictive control module 130 can optimize the control input sequence according to the current state and the reference trajectory in each control cycle, so as to minimize the state error and control input cost of the UAV in the prediction time domain. At the same time, the constraints ensure the flight safety of the UAV and the control input is within the physical capability range, thereby achieving precise flight control of the UAV.

[0124] In the embodiment of the present application, the UAV automatic target centering and tracking system 10 also includes an error calculation unit, which is used to calculate the error between the center of the target and the center of the image.

[0125] The formula of the error calculation unit is:

[0126] e x =x target -x center

[0127] e y =y targrt -y center

[0128] Among them, x target ,y target is the coordinate of the target center, x center ,y center , is the coordinate of the center of the image, e x , ey are the lateral and longitudinal errors.

[0129] It can be understood that the error calculation unit obtains the bounding box of the target by using the YOLO algorithm through the target detection module 110. The center position of the bounding box is the target center coordinate x target ,y target . The camera image center coordinate x center ,y center is a fixed value, usually half of the image resolution. For example, if the image resolution is 1920x1080, the image center coordinates are (960,540). x Affects the left and right movement of the drone. Generates lateral movement instructions based on the lateral error to adjust the left and right flight of the drone. y Affects the up and down movement of the drone. Generates longitudinal movement instructions based on the longitudinal error to adjust the up and down flight of the drone. Adjusts the yaw angle of the drone based on the lateral and longitudinal errors to keep the target in the center of the field of view.

[0130] In the embodiment of the present application, the flight control module 150 is also used to adjust the thrust, pitch angle and yaw angle of the UAV according to the optimal control input output by the model prediction control module to adjust the flight attitude and position of the UAV.

[0131] It is understandable that the automatic target centering and tracking system 10 for unmanned aerial vehicles provided in an embodiment of the present application can effectively realize automatic detection, continuous tracking and precise control of targets. By using a camera to obtain real-time video streams and using the YOLO algorithm for target detection and labeling, the system can generate accurate target bounding boxes and category labels. Combined with the Kalman filter and the Hungarian algorithm, the system can continuously track the position of the target in the video frame and make real-time corrections based on the prediction and update formulas. The model predictive control module ensures that the state error and control input cost of the unmanned aerial vehicle are minimized by establishing a state space model and solving optimization problems, while satisfying safety and physical constraints. The multi-source data provided by the sensor module further improves the accuracy of state estimation, and the flight control module adjusts the flight attitude and position of the unmanned aerial vehicle according to the error calculation results, so that the target is always kept in the center of the camera's field of view. This system is particularly suitable for scenes such as power inspections. While ensuring the safety of the unmanned aerial vehicle flight, it can efficiently and accurately complete the inspection tasks, improving the automation and intelligence level of the work.

[0132] Figure 2 An automatic target centering and tracking method for a drone is provided in an embodiment of the present application and is applied to Figure 1The automatic target centering and tracking system 10 of the unmanned aerial vehicle is shown. The automatic target centering and tracking method of the unmanned aerial vehicle comprises: S100: acquiring a real-time video stream through a camera, collecting images and video data containing the target, annotating the video data, and generating a target bounding box and a category label; S200: continuously tracking the position of the target in the video frame based on the image containing the target according to the Kalman filter and the Hungarian algorithm; S300: updating the position of the target according to the Kalman filter; S400: establishing a state space model of the unmanned aerial vehicle, an optimization problem formula and constraint conditions to achieve precise control of the unmanned aerial vehicle; S500: adjusting the flight attitude and position of the unmanned aerial vehicle so that the target always remains in the center of the camera field of view.

[0133] S100: Obtain real-time video stream through the camera, collect image and video data containing the target, annotate the video data, and generate target bounding box and category label.

[0134] In the embodiment of the present application, in step S100, the method for automatic target centering and tracking of a drone includes acquiring a real-time video stream through a camera, collecting images and video data containing the target, annotating the video data, and generating a target bounding box and a category label. It is understandable that the use of a pre-trained model can quickly and accurately identify the target in the video frame, and generate a bounding box and a category label of the target, thereby providing an accurate initial position for subsequent target tracking. Step S100 is used to ensure that the drone can continuously monitor its surrounding environment and capture the target object in a timely manner.

[0135] S200: Continuously track the position of the target in the video frame based on the image containing the target according to the Kalman filter and the Hungarian algorithm.

[0136] In an embodiment of the present application, in step S200, the automatic target centering and tracking method of the drone includes continuously tracking the position of the target in the video frame based on an image containing the target according to a Kalman filter and a Hungarian algorithm.

[0137] Among them, the target position prediction formula of the Kalman filter is:

[0138] x k∣k-1 =Fx k∣k-1 +Bu k

[0139] P k∣k-1 =FP k∣k-1 F T +Q

[0140] Among them, x k∣k-1 is the predicted state vector, P k∣k-1 is the predicted covariance matrix, F is the state transfer matrix, B is the control input matrix, u kis the control input and Q is the process noise covariance matrix.

[0141] It can be understood that through the state transfer matrix and the control input matrix, the Kalman filter can predict the position of the target at the current moment based on the state at the previous moment and the current control input. The prediction in step S200 provides a preliminary estimate for target tracking, taking into account the dynamic characteristics of the system and process noise.

[0142] S300: Update the position of the target according to the Kalman filter.

[0143] In an embodiment of the present application, in step S300, the automatic target centering and tracking method of the drone includes updating the position of the target according to the Kalman filter.

[0144] The update formula of the Kalman filter is:

[0145] K k =P k∣k-1 H T (HP k∣k-1 H T +R) -1

[0146] x k∣k =x k∣k-1 +K k (z k -Hx k∣k-1 )

[0147] P k∣k =(IK k H)P k∣k-1

[0148] Among them, K k is the Kalman gain, H is the observation matrix, R is the measurement noise covariance matrix, z k is the measurement vector.

[0149] It can be understood that through the update step of the Kalman filter, the system is able to correct the predicted target position, combining the actual measured value and the predicted value to calculate the Kalman gain. This correction mechanism ensures that the estimate of the target position is more accurate, and also updates the covariance matrix to reflect the current level of uncertainty.

[0150] S400: Establish the state space model, optimization problem formulation and constraints of the UAV to achieve precise control of the UAV.

[0151] In an embodiment of the present application, in step S400, the automatic target centering and tracking method for a UAV includes establishing a state space model of the UAV, an optimization problem formula, and constraint conditions to achieve precise control of the UAV.

[0152] It can be understood that the optimization problem formulation is used to minimize the state error and control input cost of the UAV in the prediction time domain, and the constraints are used to ensure the flight safety of the UAV and the control input is within the physical capabilities.

[0153] Among them, the state space model formula of the drone is:

[0154] x k+1 =Ax k +Bu k

[0155] Among them, x k is the state vector, u k is the control input vector, A is the state matrix, which describes the dynamic model of the UAV's state evolution over time, and B is the input matrix, which describes how the control input affects the state change of the UAV.

[0156] The optimization problem formula is:

[0157]

[0158] Among them, x k∣t is the state predicted at time t at the kth moment, x r is the reference trajectory or expected state, u k∣t is the control input at the kth moment predicted at time t, Q is the weight matrix of the state error, which is used to adjust the influence of the state error on the objective function, R is the weight matrix of the control input, which is used to adjust the influence of the control input on the objective function, Q f is the weight matrix of the terminal state error, and N is the length of the prediction time domain.

[0159] It can be understood that the optimization problem is set to ensure that the UAV can get as close to the desired trajectory or state as possible by minimizing the state error and control input cost. This optimization mechanism can provide the best control input sequence in the prediction time domain, thereby achieving precise control of the UAV.

[0160] S500: Adjust the flight attitude and position of the drone so that the target always remains in the center of the camera's field of view.

[0161] In an embodiment of the present application, in step S500, the automatic target centering and tracking method of the drone includes adjusting the flight attitude and position of the drone so that the target always remains in the center of the camera's field of view.

[0162] It can be understood that the automatic target centering and tracking method of the drone can ensure that the target is always in the center of the camera's field of view by adjusting the drone's flight attitude (such as pitch, yaw and roll angles) and position in real time. This adjustment mechanism combines sensor data, prediction models and optimization control results to achieve efficient and accurate target tracking.

[0163] Figure 3 This is an electronic device 20 provided in an embodiment of the present application. Figure 3 As shown, the electronic device 20 includes at least the following parts: a processor 21 and a memory 22 .

[0164] In the embodiment of the present application, the memory 22 is used to store instructions executable by the processor 21, and the processor 21 is configured to implement the following when executing the instructions: Figure 1 The method shown.

[0165] In an embodiment of the present application, a computer-readable storage medium includes instructions, and the instructions instruct a device to execute the method of the first aspect. For example, the instructions instruct the device to execute Figure 1 The method shown in steps S100 to S900.

[0166] The program operating in the electronic device 20 involved in one embodiment of the present application may be a program (a program that enables a computer to function) that controls a central processing unit (CPU) and the like to realize the functions of the above-mentioned embodiment involved in one scheme of the present invention. Then, the information processed by these devices is temporarily stored in a random access memory (RAM) during its processing, and then stored in various ROMs such as a read-only memory (Flash ROM) and a hard disk drive (HDD), and is read, modified, and written by the CPU as needed.

[0167] It should be noted that part of the electronic device 20 of the above embodiment may be implemented by a computer. In this case, a program for implementing the control function may be recorded in a computer-readable recording medium and implemented by reading the program recorded in the recording medium into a computer system and executing it.

[0168] It should be noted that the "computer system" mentioned here refers to a computer system built into the electronic device 20, and adopts a computer system including hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to removable media such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into the computer system.

[0169] Furthermore, "computer-readable recording media" may include: media that dynamically store programs for a short period of time, such as communication lines when sending programs via networks such as the Internet or communication lines such as telephone lines; and media that store programs for a fixed period of time, such as volatile memories inside computer systems that serve as servers or clients in this case. In addition, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, or a program that can realize the above-mentioned functions by combining with a program already recorded in a computer system.

[0170] In addition, the electronic device 20 in the above-mentioned embodiment can also be implemented as a collection (device group) composed of multiple devices. Each device constituting the device group can have a part or all of the functions or functional blocks of the electronic device 20 in the above-mentioned embodiment. As a device group, it is sufficient to have all the functions or functional blocks of the electronic device 20.

[0171] It is understandable that the drone automatic target centering and tracking system 10, drone automatic target centering and tracking method, electronic device 20 and storage medium provided in the embodiment of the present application can realize accurate detection and stable tracking of targets in complex environments. By integrating advanced target detection algorithms, target tracking modules of Kalman filter and Hungarian algorithm, and flight control modules based on model predictive control (MPC), the system can acquire and process data from cameras, inertial measurement units and Beidou positioning modules in real time to form a closed-loop control mechanism, thereby ensuring that the drone always keeps the target in the center of the camera's field of view during flight.

[0172] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present application and are not intended to be limiting of the present application. As long as they are within the spirit and scope of the present application, appropriate changes and modifications to the above embodiments are within the scope of protection claimed in the present application.

Claims

1. An automatic target centering and tracking system for unmanned aerial vehicles, characterized in that: The system comprises: The target detection module is used to obtain real-time video streams through the camera, collect image and video data containing the target, and annotate the video data to generate target bounding boxes and category labels; The target tracking module is used to continuously track the position of the target in the video frame based on the output of the target detection module according to the Kalman filter and the Hungarian algorithm, wherein the target position prediction formula of the Kalman filter is: ； ； in, is the predicted state vector, is the predicted covariance matrix, is the state transition matrix, is the control input matrix, is the control input, is the process noise covariance matrix; The target tracking module is also used to update the position of the target according to the Kalman filter. The update formula of the Kalman filter is: ； ； ； in, is the Kalman gain, is the observation matrix, is the measurement noise covariance matrix, is the measurement vector; A model predictive control module, used to establish a state space model of the UAV, an optimization problem formula and constraints to achieve precise control of the UAV, wherein the optimization problem formula is used to minimize the state error and control input cost of the UAV in the prediction time domain, and the constraints are used to ensure the flight safety of the UAV and that the control input is within the physical capability range; Among them, the state space model formula of the drone is: ； in, is the state vector, is the control input vector, is the state matrix, describing the dynamic model of the UAV’s state evolution over time, is the input matrix, describing how the control input affects the state change of the UAV; The optimization problem formula is: ； in, is the state at the kth moment predicted at time t, is the reference trajectory or desired state, is the control input at the kth moment predicted at time t, is the weight matrix of the state error, which is used to adjust the influence of the state error on the objective function. is the weight matrix of the control input, which is used to adjust the influence of the control input on the objective function. is the weight matrix of the terminal state error, is the length of the prediction time domain, is the state vector at the Nth moment predicted at time t; A sensor module, the sensor module comprising: the camera, an inertial measurement unit and a Beidou positioning module, the sensor module is used to obtain the state information of the UAV and provide it to the model prediction control module; The flight control module is used to adjust the flight attitude and position of the UAV so that the target always remains in the center of the camera's field of view.

2. The automatic target centering and tracking system for unmanned aerial vehicles according to claim 1, characterized in that: The object detection module uses the YOLO algorithm to output the location, size and confidence score of the bounding box.

3. The automatic target centering and tracking system for unmanned aerial vehicles according to claim 1, characterized in that: The constraints include state constraints and input constraints; the state constraints are used to limit the range of predicted states to ensure that the UAV does not exceed physical and safety limitations during flight; the input constraints are used to limit the range of control inputs to ensure that control commands are within the physical capabilities of the UAV.

4. The automatic target centering and tracking system for unmanned aerial vehicles according to claim 3, characterized in that: The state constraint formula is: ； in, is the lower limit vector of the state variables, indicating the minimum allowed value of each state variable, is the upper limit vector of the state variables, indicating the maximum allowed value of each state variable; The input constraint formula is: ； in, is the lower limit vector of the control input, indicating the minimum allowable value of each control input, is the upper limit vector of the control input, indicating the maximum allowed value of each control input.

5. The automatic target centering and tracking system for unmanned aerial vehicles according to claim 3 or 4, characterized in that: The constraints also include terminal state constraints, which are used to ensure that the UAV is in a safe and controllable state at the end of the prediction time domain; The formula for the terminal state constraint is: ； in, is the allowed set of terminal states, indicating the conditions that must be satisfied in the prediction time domain terminal state.

6. The automatic target centering and tracking system for unmanned aerial vehicles according to claim 1, characterized in that: The automatic target centering and tracking system for unmanned aerial vehicles also includes an error calculation unit, which is used to calculate the error between the center of the target and the center of the image; The formula of the error calculation unit is: ； ； in, , are the coordinates of the target center, , , are the coordinates of the center of the image, , are the lateral and longitudinal errors.

7. The automatic target centering and tracking system for unmanned aerial vehicles according to claim 1, characterized in that: The flight control module is also used to adjust the thrust, pitch angle and yaw angle of the UAV according to the optimal control input output by the model prediction control module, so as to adjust the flight attitude and position of the UAV.

8. A method for automatic target centering and tracking of a UAV, applied to the automatic target centering and tracking system of a UAV as claimed in any one of claims 1 to 7, characterized in that: include: Acquire real-time video streams through cameras, collect image and video data containing targets, annotate the video data, and generate target bounding boxes and category labels; The position of the target is continuously tracked in the video frame based on the image containing the target according to the Kalman filter and the Hungarian algorithm, wherein the target position prediction formula of the Kalman filter is: ； ； in, is the predicted state vector, is the predicted covariance matrix, is the state transition matrix, is the control input matrix, is the control input, is the process noise covariance matrix; The position of the target is updated according to the Kalman filter, and the updating formula of the Kalman filter is: ； ； ； in, is the Kalman gain, is the observation matrix, is the measurement noise covariance matrix, is the measurement vector; Establishing a state space model, an optimization problem formula and constraints of the UAV to achieve precise control of the UAV, wherein the optimization problem formula is used to minimize the state error and control input cost of the UAV in the prediction time domain, and the constraints are used to ensure the flight safety of the UAV and that the control input is within the physical capability range; Among them, the state space model formula of the drone is: ； in, is the state vector, is the control input vector, is the state matrix, describing the dynamic model of the UAV’s state evolution over time, is the input matrix, describing how the control input affects the state change of the UAV; The optimization problem formula is: ； in, is the state at the kth moment predicted at time t, is the reference trajectory or desired state, is the control input at the kth moment predicted at time t, is the weight matrix of the state error, which is used to adjust the influence of the state error on the objective function. is the weight matrix of the control input, which is used to adjust the influence of the control input on the objective function. is the weight matrix of the terminal state error, is the length of the prediction time domain; The flight attitude and position of the UAV are adjusted so that the target always remains in the center of the camera's field of view.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the automatic target centering and tracking method for the unmanned aerial vehicle described in claim 8 when executing the instructions.

10. A computer-readable storage medium, characterized in that: The method comprises instructions for instructing a device to execute the automatic target centering and tracking method for a UAV as described in claim 8.

Images