An Autonomous Detection and Tracking Method for Indo-Pacific Hippodolites Using UAV-borne Vision

Abstract

This invention relates to the interdisciplinary field of intelligent unmanned systems and computer vision, specifically disclosing a method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision. The method includes the following steps: S1, constructing an image database of Indo-Pacific humpback dolphins in a real-world environment; S2, training a high-precision Indo-Pacific humpback dolphin detection model, and lightweighting the model using structured pruning techniques; S3, inputting real-time airborne images into the detection model to obtain target detection box information, constructing visual servo feature vectors, and using a Kalman filter algorithm to optimally estimate the features and calculate the real-time deviation of the visual servo; S4, using a PID control algorithm to calculate the UAV's three-dimensional speed control commands and drive the flight controller to execute corresponding maneuvers. This invention's method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision enables real-time autonomous tracking and monitoring of Indo-Pacific humpback dolphins by UAVs.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent unmanned systems and computer vision, and particularly relates to a method for autonomous detection and tracking of Sousa chinensis by onboard vision of a drone. BACKGROUND

[0002] Sousa chinensis is a rare marine mammal that inhabits nearshore shallow waters and is known as the "water panda". The health of its population is a "barometer" of the health of the coastal ecosystem. However, with high-intensity nearshore human activities such as shipping, fishing, and marine engineering, Sousa chinensis is facing habitat compression and survival threats, and has been assessed as "vulnerable" by the International Union for Conservation of Nature. Therefore, the continuous and effective monitoring of the Sousa chinensis population is crucial for developing protection strategies.

[0003] Existing monitoring of Sousa chinensis mainly relies on ship line surveys and traditional photo identification techniques. With the development of artificial intelligence technology, individual identification of Sousa chinensis has shifted to deep learning-based methods in recent years. However, in actual drone monitoring tasks, there are still the following main technical problems:

[0004] (1) Conflict between model computing power and real-time performance: Existing high-precision target detection models usually have large parameter quantities and high computational complexity. Small and medium-sized scientific research monitoring drones are limited by payload and battery endurance, and the computing power of their onboard edge computing devices is limited, making it difficult to directly run large deep learning models, resulting in high detection delay and failing to meet the demand for "real-time discovery and real-time tracking".

[0005] (2) Lack of autonomous tracking capability: Current drone monitoring mostly uses the mode of "manual remote control flight + gimbal hand control camera". This mode is highly dependent on the operating experience of the pilot, and under the conditions of complex sea surface light changes and continuous movement of Sousa chinensis, the target is easily lost. In addition, long-term manual control has limitations in terms of labor cost and operation efficiency, making it difficult to achieve long-term continuous monitoring.

[0006] (3) Decoupling of visual features and control: Existing vision systems usually only take care of "seeing" (i.e., detection and identification), and the identification results (the position of the Sousa chinensis) are usually only recorded as data or displayed to the operator. The geometric features (size changes of the target in the image) output by the vision detection are not directly converted into motion control signals for the drone, resulting in a separation between "perception" and "action", and making it impossible to achieve closed-loop autonomous servo tracking. SUMMARY

[0007] The present invention aims to solve at least one of the technical problems mentioned above, and provides an autonomous detection and tracking method for Chinese white dolphins using UAV airborne vision. This method can solve the problems of the difficulty in deploying target detection models on the airborne end and the poor real-time performance, as well as the difficulty in long-term tracking of white dolphins by UAVs relying on manual remote control. This method enables UAVs to perform real-time autonomous tracking and monitoring of Chinese white dolphins.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: an autonomous detection and tracking method for Indo-Pacific humpback dolphins using UAV-borne vision, characterized by comprising the following steps:

[0009] S1. Construct a database of images of Chinese white dolphins in a real-world environment. Use the Unity 3D engine to build a virtual simulation environment to expand the image data and preprocess and enhance the images in the database.

[0010] S2. Use the image data from step S1 to train a high-precision Chinese white dolphin detection model, and use structured pruning technology to lightweight the model to address the computing power limitations of the UAV-borne edge computing device.

[0011] S3. Input the real-time images collected by the UAV into the Chinese white dolphin detection model in S2 to obtain the Chinese white dolphin detection box information. Convert the above detection box information into a visual servo feature vector that can be used for UAV closed-loop control. By extracting the geometric information of the Chinese white dolphin bounding box, construct image features describing the relative position and distance of the target, and calculate the real-time servo deviation.

[0012] S4. The PID control algorithm is used to calculate the three-dimensional speed control command of the UAV and drive the flight controller to perform the corresponding maneuver, thereby realizing the UAV's closed-loop autonomous tracking of the Chinese white dolphin.

[0013] Preferably, the construction of the Indo-Pacific humpback dolphin image database includes the following steps:

[0014] S1.1.1 Utilize drones to conduct aerial photography at multiple altitudes and angles in the target sea area, collecting images covering all age groups of Indo-Pacific humpback dolphins, including gray-black calves, subadults with increased spots, and white or pink adults, as well as clear images including dorsal fin morphology and body spot patterns.

[0015] S1.1.2. Frame extraction processing is performed on the acquired image data to remove images with motion blur and out of focus. The LabelImg annotation tool is used to annotate the white dolphins in the images with rectangular boxes, recording the target's category label and coordinate information to form a real source domain dataset. .

[0016] Preferably, the image data augmentation process for building a virtual simulation environment using the Unity 3D engine includes the following steps:

[0017] S1.2.1. Create a high-fidelity 3D parametric model of the Chinese white dolphin in Unity, and implement its skeletal animation through script control to simulate the dolphin's swimming, emerging from the water, diving and rolling postures.

[0018] S1.2.2. Set the Euler angle parameters of the directional light source in the virtual environment. The X-axis rotation angle is used to simulate the solar altitude angle, with a random sampling range of 15° to 90°; the Y-axis rotation angle is used to simulate the solar azimuth angle, with a random sampling range of 0° to 360°; the light intensity has a random range of 0.8 to 2.0; simultaneously, the water material shader parameters are adjusted to simulate sea conditions, with the wave height set to a random range of 0.1 m to 1.0 m, the water transparency set to a random range of 0.4 to 0.9, and the RGB values ​​of the water color randomly interpolated within the blue-green and turbid yellow-brown range; randomize the extrinsic and intrinsic parameters of the virtual camera, setting the shooting height range to 15 m to 40 m, the camera pitch angle range to 45° to 90°, and the field of view range to 60° to 90°; superimpose Gaussian noise with a mean of 0 and a variance range of 0.001 to 0.02 into the image; and randomly generate reflective areas and floating objects on the sea surface to enhance the model's robustness to environmental disturbances;

[0019] S1.2.3. Utilizing Unity's scripting interface, while rendering the composite image, automatically obtain the projection coordinates of the 3D model in screen space, generate pixel-level bounding box annotation information, and batch export the composite image and its annotation files to form a virtual source domain dataset. .

[0020] Preferably, the image preprocessing and enhancement includes... and Mix and then process as follows:

[0021] S1.3.1 Perform random rotation, random cropping, and horizontal flipping on the image to simulate different positions during drone shooting. Perform random perturbation on brightness, contrast, and saturation. At the same time, introduce Gaussian noise to simulate noise of the image sensor or interference of the image transmission signal under high sensitivity. Introduce motion blur to simulate the body shaking of the drone during flight or the trailing shadow produced by the high-speed swimming of the Chinese white dolphin.

[0022] S1.3.3 Divide the augmented mixed dataset into training, validation and test sets in an 8:1:1 ratio, and convert them into COCO format.

[0023] Preferably, step S2 includes:

[0024] S2.1 Construct a deep neural network model with residual structure, using RTMDetection as the basic detection framework, which includes:

[0025] S2.1.1. The CSPNEXt backbone network is used as the feature extraction network;

[0026] S2.1.2 Design PAFPN as a feature fusion layer, which fuses multi-scale feature maps through a bidirectional path from top to bottom and from bottom to top;

[0027] S2.1.3. A decoupled head structure is adopted to separate the task of classifying white dolphins from the task of regressing the bounding box position of dolphins.

[0028] S2.2 Network model training and optimization: The network is trained under full supervision using the hybrid dataset generated in step S1. This includes:

[0029] S2.2.1, Define the total loss function :

[0030] ,

[0031] in, This is the quality focus loss, used to optimize the joint distribution of classification scores and location confidence. For generalized intersection-union loss, it is used to regress the geometric overlap between the predicted bounding box and the ground truth bounding box; and These are the weighting coefficients;

[0032] S2.2.2 Use the AdamW optimizer to update parameters, setting the initial learning rate to 0.001. Employ cosine annealing to adjust the learning rate until the model's mean accuracy on the validation set converges, thus obtaining the baseline model. ;

[0033] S2.3. Sparse training and structured pruning of the model based on the BN layer scaling factor, including:

[0034] S2.3.1, in the total loss function The L1 regularization term is introduced to scale all batch-normalized BN layers in the model. By applying sparsity constraints, the corrected loss function is obtained as follows:

[0035] ,

[0036] in, Let be the sparsity penalty coefficient, and take . =0.001, Given the set of scaling factors for all BN layers, sparse training forces the channels that contribute less to the network output to have different scaling factors. The value tends to 0.

[0037] S2.3.2 Obtaining the layers after sparse training Distribution, setting a global pruning threshold For all The top 30% quantiles after sorting are traversed through the network structure to remove [the remaining quantiles]. By analyzing the channels and their convolutional kernel weights, the pruned model structure can be obtained. ;

[0038] S2.4: Fine-tuning and solidification of the model, Retraining is performed on the image database built by S1, using a small learning rate for iteration. When the accuracy of the fine-tuned model recovers to more than 95% of the baseline model and the model size meets the requirements for airborne deployment, training is stopped and the final lightweight model weight file is exported.

[0039] Preferably, step S3 includes the following steps:

[0040] S3.1 Input the real-time images captured by the UAV into the Indo-Pacific humpback dolphin detection model in S2 to obtain the Indo-Pacific humpback dolphin detection box information, and extract the pixel coordinate information of the four vertices of the detection box: top left corner... Top right corner bottom right corner and bottom left corner Before feature extraction, a geometric rationality check is performed. If the aspect ratio of the detection box exceeds the preset range of biological features of the white dolphin, or if the edge of the detection box touches the image boundary, the current frame feature is marked as invalid and the previous frame state is maintained.

[0041] S3.2 Select the centroid coordinates and bounding box area of ​​the target as the core visual servo features, which are used to control the horizontal position and flight altitude of the UAV, respectively. These features include:

[0042] S3.2.1 Calculate the geometric center of the target on the image plane using the coordinates of its four corner points. The formula used to characterize the horizontal relative position of the drone and the dolphin is as follows:

[0043] ,

[0044] S3.2.2 Calculate the bounding box area using information from the four corner points. The value is used to characterize the relative distance between the drone and the dolphin, and the calculation formula is:

[0045]

[0046] S3.2.3 Constructing the current moment Image feature vectors ;

[0047] S3.3, Use a Kalman filter to process the eigenvectors Perform optimal estimation and establish state equations. ,in The rate of change of the eigenvectors; establish the state equation. and observation equations , here For the observation matrix, take This indicates that only location information was observed; the prediction and update steps include:

[0048] S3.3.1 Prediction Steps: Based on the posterior state estimate from the previous time step Using the state transition matrix Estimate the prior state value at the current time. The calculation formula is: Simultaneously, the prior error covariance matrix is ​​derived. The calculation formula is: ,in The process noise covariance matrix;

[0049] S3.3.2 Update Steps: Calculate Kalman Gain This is used to balance the covariance of prediction error and the covariance of observation noise; the calculation formula is as follows. ,in To observe the noise covariance matrix;

[0050] S3.3.3, State Correction and Output: Combining the observation vector at the current time step The Kalman gain is used to correct the prior state estimate to obtain the optimal posterior state estimate at the current time. The calculation formula is: Then, by selecting the matrix... Extracting smoothed feature vectors The calculation formula is: ,in This yields the smoothed feature vector after removing Gaussian white noise. .

[0051] S3.4 Calculate the real-time servo deviation, which includes:

[0052] S3.4.1 Define the ideal image state for drone tracking of Indo-Pacific humpback dolphins. .in, and Represented as the physical center coordinates of the image plane. This represents the expected pixel area of ​​an adult white dolphin at a reference height.

[0053] S3.4.2 Calculate the current smoothing feature With expected features The difference between them is used to generate a visual servoing error vector. The calculation formula is:

[0054] .

[0055] Preferably, step S4 includes:

[0056] S4.1. A decoupled image-based visual servo control strategy is adopted to establish a mapping relationship between image feature errors and the velocity of the UAV body coordinate system.

[0057] S4.2 Construct an incremental PID controller;

[0058] S4.3 Limit the amplitude of the PID output and set the maximum allowable speed threshold. If the calculated control quantity exceeds the threshold, it is truncated to the maximum value.

[0059]

[0060] In addition, the output speed command is low-pass filtered;

[0061] S4.4, Target loss relocation and exception handling strategy;

[0062] S4.5. Using the onboard SDK, the calculated and smoothed final speed command is... The package is sent to the drone's underlying flight controller, which drives the motors to perform the corresponding actions, completing the closed-loop tracking task.

[0063] Preferably, in step S4.1, the mapping relationship is established by relating the four control degrees of freedom of the UAV (throttle, pitch, roll, and yaw) to the current moment. Visual feature error Perform the following mapping:

[0064] Yaw angular velocity: determined by horizontal centroid error Control and eliminate the target's horizontal deviation in the image, ensuring that the drone's nose is always pointed at the white dolphin;

[0065] Forward speed: determined by vertical centroid error Control and adjust the horizontal distance between the drone and the target to keep the target centered in the vertical direction of the image;

[0066] Vertical velocity: determined by area error Control, utilizing the bounding box area With depth The inverse relationship, when the detected dolphin area Smaller than expected area When the drone descends, it is controlled to descend; conversely, it ascends.

[0067] Preferably, the construction of the incremental PID controller includes:

[0068] An incremental PID algorithm is used to calculate the speed increment of each control channel and the final output value in parallel:

[0069] (1) Forward speed control: Calculate the control increment Final output The formula for calculating the increment is:

[0070] ;

[0071] (2) Yaw rate control: utilizing horizontal centroid error Calculate control increment Final output The formula for calculating the increment is:

[0072] ;

[0073] (3) Vertical speed control: utilizing area error Calculate control increment Final output The formula for calculating the increment is:

[0074] ;

[0075] in, and These are the error values ​​for the previous time step and the time step before that, respectively. These are the independent proportional, integral, and differential gain coefficients for each channel, which need to be tuned through flight experiments.

[0076] Preferably, the specified target loss relocation and anomaly handling strategy includes:

[0077] Tracking status: When targets are continuously detected, repeat steps S3-S4;

[0078] Predicted state retention: If the target is lost for a certain period of time Assuming that the dolphin's movement has inertia, the flight velocity vector of the previous moment remains unchanged, and the drone's own movement continues to cover the predicted area;

[0079] Hover search status: If the target is lost for a period of time If the target is determined to be lost, the drone immediately switches to hover mode and performs a slow, rotating scan in place until the target is detected again or a return-to-home command is received.

[0080] The beneficial effects are as follows: Compared with the existing technology, the UAV-borne vision-based autonomous detection and tracking method for Indo-Pacific humpback dolphins of the present invention solves the problems of difficult deployment and slow inference of complex deep learning models at the edge of the UAV by using deep learning target detection algorithms and model pruning technology, thereby realizing real-time detection of Indo-Pacific humpback dolphins on the sea surface and in shallow waters. At the same time, by constructing a visual servo control mechanism based on the detection box area, a speed mapping relationship between image features and UAV movement is established, enabling the UAV to autonomously adjust its flight altitude and horizontal position according to the changes in the position and size of the dolphin in the field of vision, thereby achieving autonomous and stable tracking of Indo-Pacific humpback dolphins and improving the intelligence level and work efficiency of dolphin monitoring. Attached Figure Description

[0081] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0082] Figure 1 This is a flowchart illustrating the workflow of an unmanned aerial vehicle (UAV)-based airborne vision method for autonomous detection and tracking of Indo-Pacific humpback dolphins according to the present invention.

[0083] Figure 2 A diagram showing the network recognition effect for the Indo-Pacific humpback dolphin.

[0084] Figure 3 3D trajectory curves during the autonomous tracking process of the drone;

[0085] Figure 4 This is the image error convergence curve during the autonomous tracking process of the UAV. Detailed Implementation

[0086] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0087] It should be noted that when a component is described as "fixed to" another component, it can be directly on the other component or may have a central component. When a component is described as "connected to" another component, it can be directly connected to the other component or may have a central component. When a component is described as "set on" another component, it can be directly set on the other component or may have a central component. When a component is described as "set in the middle," it is not simply set in the exact center, as long as it is not set within the area defined by both ends being in the middle. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0088] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0089] like Figures 1 to 4 As shown, this invention discloses an autonomous detection and tracking method for Indo-Pacific humpback dolphins using UAV-borne vision, specifically including the following steps:

[0090] S1: Construction of the Indo-Pacific humpback dolphin image dataset

[0091] To address the challenges of collecting wild data on Indo-Pacific humpback dolphins, imbalanced samples, and complex lighting conditions at sea, this module combines real aerial photography data with virtual simulation data to construct a high-quality, diverse training dataset. The specific steps are as follows:

[0092] S1.1 Construct a sample library of Chinese white dolphins in a real environment.

[0093] S1.1.1 Data Acquisition: Utilize drones to conduct multi-altitude, multi-angle aerial photography in the target sea area. The acquisition altitude range covers 15 to 40 meters, and the acquisition angle covers a gimbal tilt angle of 90 degrees vertically downward to 45 degrees tilt. The acquisition subjects cover all age groups of Indo-Pacific humpback dolphins, including gray-black calves, subadults with increased spots, and white or pink adults, as well as clear images including dorsal fin morphology and body spot patterns.

[0094] S1.1.2 Data Cleaning and Labeling: Frame extraction was performed on the acquired video stream to remove images with severe motion blur and out of focus. The LabelImg tool was used to create bounding boxes for the white dolphins in the images, recording the target's category label and coordinate information to form a real source domain dataset. .

[0095] S1.2 Synthetic Sample Generation Based on Unity 3D Domain Randomization Technology. To address the issues of missing rare pose samples and data imbalance in field data, a virtual simulation environment was constructed using the Unity 3D engine to augment the data.

[0096] S1.2.1 3D Model Construction: A high-fidelity 3D parametric model of the Indo-Pacific humpback dolphin is built in Unity, and its skeletal animation is implemented through script control to simulate the dolphin's swimming, surfacing, diving and rolling postures.

[0097] S1.2.2 Domain Randomization Scene Configuration: To reduce the information distribution between virtual and real data, the simulation environment is randomized: the position and intensity of light sources are randomly adjusted to simulate sunlight at different times, sea wave height and texture, water color and transparency; the viewpoint, height, focal length and noise parameters of the virtual camera are randomized to simulate the shooting effect of the drone under different flight conditions; reflective areas and floating objects on the sea surface are randomly generated to enhance the robustness of the model to environmental disturbances. Specifically, the Euler angle parameters of the directional light source in the virtual environment can be set, where the X-axis rotation angle is used to simulate the solar altitude angle, with a random sampling range of 15° to 90°; the Y-axis rotation angle is used to simulate the solar azimuth angle, with a random sampling range of 0° to 360°; illumination... The intensity ranges randomly from 0.8 to 2.0. Simultaneously, sea conditions are simulated by adjusting the water material shader parameters, setting the wave height to a random range of 0.1 to 1.0 meters, the water transparency (Alpha channel) to a random range of 0.4 to 0.9, and the RGB values ​​of the water color to be randomly interpolated within the blue-green and turbid yellow-brown range. The extrinsic and intrinsic parameters of the virtual camera are randomized, setting the shooting height to a range of 15 to 40 meters, the camera pitch angle to a range of 45° to 90°, and the field of view (FOV) to a range of 60° to 90°. Gaussian noise with a mean of 0 and a variance range of 0.001 to 0.02 is superimposed on the image. Reflective areas and floating objects are randomly generated on the sea surface to enhance the model's robustness to environmental disturbances.

[0098] S1.2.3 Automatic Annotation and Generation: Utilizing Unity's scripting interface, the projected coordinates of the 3D model in screen space are automatically obtained while rendering the composite image, generating pixel-level bounding box annotations. Composite images and their annotation files are exported in batches to form a virtual source domain dataset. .

[0099] S1.3 Image Preprocessing and Enhancement. To improve the model's generalization ability in complex sea environments, image preprocessing and enhancement will be performed. and Mix and then process as follows:

[0100] S1.3.1 Basic Data Enhancement: Random rotation, random cropping, and horizontal flipping are applied to the images to simulate different orientations during drone shooting. Random perturbations are applied to brightness, contrast, and saturation to cope with varying lighting conditions in the field. Gaussian noise is introduced to simulate noise from the image sensor or interference with the image transmission signal under high sensitivity. Motion blur is introduced to simulate the body shake of the drone during flight or the trailing shadows caused by the high-speed swimming of the Indo-Pacific humpback dolphin, improving the model's stability in detecting low-quality and blurred images.

[0101] S1.3.3 Dataset partitioning: The augmented mixed dataset is divided into training, validation and test sets in an 8:1:1 ratio and uniformly converted to COCO format.

[0102] S2: Training and Pruning Module for the Indo-Pacific Hippodolite Detection Network

[0103] This module aims to train a high-precision Chinese white dolphin detection model using the augmented dataset built with S1. Addressing the computing power limitations of UAV-borne edge computing devices, it employs structured pruning techniques to lightweight the model, achieving an optimal balance between detection accuracy and inference speed. The specific steps are as follows:

[0104] S2.1: Construct a deep neural network architecture with residual structures. RTMDetection (Real-time Models for Object Detection) is used as the basic detection framework.

[0105] S2.1.1 (Backbone) Construction: The CSPNEXt backbone network is used as the feature extraction network. This structure enhances gradient propagation by introducing cross-layer connections in the residual blocks, which can effectively extract semantic features such as dorsal fin morphology and body color texture of the Indo-Pacific humpback dolphin, while maintaining low computational latency.

[0106] S2.1.2 Neck Network Construction: Considering that changes in flight altitude from the drone's aerial perspective can lead to large-scale changes in the appearance of dolphins in images, a PAFPN (Path Aggregation Feature Pyramid Network) is designed as a feature fusion layer. Through bidirectional paths from top to bottom and bottom to top, multi-scale feature maps are fused to improve the model's ability to detect small targets (distant Indo-Pacific humpback dolphins) and large targets (close-up Indo-Pacific humpback dolphins) on the sea surface.

[0107] S2.1.3 Head Construction: A decoupled head structure is adopted to separate the task of classifying white dolphins and the task of regressing the bounding box position of dolphins, avoiding the conflict between the two tasks in the feature space and accelerating the convergence of the model.

[0108] S2.2: Network Model Training and Optimization. The network is trained under full supervision using the hybrid dataset generated in module S1.

[0109] S2.2.1 Defining the Loss Function: To address the issues of imbalanced positive and negative samples (more sea background than dolphin targets) and uneven distribution of easy and difficult samples, a dynamic soft-label allocation strategy is adopted, and a total loss function is defined. :

[0110]

[0111] in, This is the quality focus loss, used to optimize the joint distribution of classification scores and location confidence. For generalized intersection-union loss, it is used to regress the geometric overlap between the predicted bounding box and the ground truth bounding box; and These are the weighting coefficients.

[0112] S2.2.2 Iterative Optimization: The AdamW optimizer is used to update parameters, with an initial learning rate of 0.001. Cosine annealing is used to adjust the learning rate until the mean average precision (mAP) of the model on the validation set converges, thus obtaining the baseline model. .

[0113] S2.3: Sparsification training and model pruning based on BN layer scaling factor.

[0114] To address the issue of insufficient computing power in drone onboard equipment, Perform channel-level pruning.

[0115] S2.3.1 Sparsity Training: An L1 regularization term is introduced into the training loss function of S2.2, which is a scaling factor for all batch normalization (BN) layers in the network. Apply sparsity constraints. The corrected loss function is:

[0116]

[0117] in, The sparsity penalty coefficient is taken here. =0.001, Let be the set of scaling factors for all Batch Normalization (BN) layers. Through sparse training, the scaling factors for channels that contribute less to the network output are forced to scalp. The value tends to 0.

[0118] S2.3.2 Structured Pruning: Obtaining the Structure of Each Layer After Sparse Training Distribution, setting a global pruning threshold For all The top 30% quantiles after sorting. Traverse the network structure and remove... The channels and their connected convolutional kernel weights. Pruning not only reduces the number of model parameters but also significantly reduces floating-point operations (FLOPs), resulting in the pruned model structure. .

[0119] S2.4: Fine-tuning and Model Consolidation. Because pruning disrupts the original weight distribution, it may cause a temporary decrease in accuracy, necessitating fine-tuning. Retraining is performed on the image database built in S1, using a small learning rate (e.g., 1 / 10 of the original learning rate) and a small number of iterations to restore the model's ability to extract features of the Indo-Pacific humpback dolphin. When the accuracy of the fine-tuned model recovers to more than 95% of the baseline model and the model size meets the requirements for airborne deployment, training is stopped and the final lightweight model weight file is exported.

[0120] S3: Visual Servo Feature Extraction Module

[0121] This module aims to connect front-end perception and back-end control, transforming the raw target detection data output by the S3 module into visual servo feature vectors that can be used for UAV closed-loop control. By extracting the geometric information of the bounding box of the Indo-Pacific humpback dolphin, image features describing the target's relative position and distance are constructed, and real-time servo deviation is calculated. The specific steps are as follows:

[0122] S3.1: Bounding box corner coordinate extraction and validity verification. Receives the Indo-Pacific humpback dolphin detection box information output from module S3, such as... Figure 2 As shown. The detection box is defined as a rectangle, and the pixel coordinates of its four vertices are extracted: top left corner... Top right corner bottom right corner and bottom left corner Geometric rationality is verified before feature extraction. If the aspect ratio of the detection box exceeds the preset range of biological characteristics of white dolphins (e.g., the aspect ratio is too large or too small), or the edge of the detection box touches the image boundary (resulting in incomplete features), the features of the current frame are marked as invalid, and the state of the previous frame is maintained to prevent abnormal data from interfering with the control system.

[0123] S3.2: Construction of Visual Servo Feature Vectors for Images. The centroid coordinates and bounding box area of ​​the target are selected as the core visual servo features, which are used to control the horizontal position and flight altitude of the UAV, respectively.

[0124] S3.2.1 Centroid Feature Extraction: Calculate the geometric center of the target on the image plane using the coordinates of its four corner points. This is used to characterize the horizontal relative position of the drone and the dolphin. The calculation formula is as follows:

[0125]

[0126] S3.2.2 Area Feature Extraction: Calculating the bounding box area using information from the four corner points. This variable, denoted as , represents the relative distance between the drone and the dolphin (i.e., a proxy variable for depth information). Based on the imaging principle of a pinhole camera, the area of ​​an object in an image is inversely proportional to the square of its distance; therefore, changes in area directly reflect changes in altitude. The calculation formula is as follows:

[0127]

[0128] S3.2.3 Feature Vector Generation: Constructing the Image Feature Vector at the Current Time Step .

[0129] S3.3: Feature Smoothing Filtering. To address the issue of detection frame jitter caused by sea surface fluctuations and high-frequency vibrations from UAVs, a **Kalman Filter** is used to smooth the signal input to the control system to ensure smoothness of the feature vector. Perform the optimal estimate.

[0130] Establish state equations and observation equations , where the state variables are the eigenvectors and their rates of change.

[0131] Through prediction and update steps, the current observation value is fused with the predicted value from the previous time step, Gaussian white noise is filtered out, and a smoothed feature vector is output. .

[0132] S3.4: Servo bias calculation.

[0133] S3.4.1 Define desired features: Define the ideal image state for drone tracking of Indo-Pacific humpback dolphins. .in, and Represented as the physical center coordinates of the image plane, designed to keep the white dolphin always in the center of the field of view. This represents the expected pixel area of ​​an adult white dolphin at a reference height, used to maintain a safe observation distance. Here, the reference height is set to 30 meters.

[0134] S3.4.2 Deviation Calculation: Calculate the current smoothing feature. With expected features The difference between them is used to generate a visual servoing error vector. As input for the S4 module control law design:

[0135]

[0136] S4: Unmanned Aerial Vehicle (UAV) Autonomous Flight Control Module

[0137] This module aims to use the visual servo error vector output by the S3 module to calculate the UAV's three-dimensional velocity control commands through a PID control algorithm, and drive the flight controller to execute corresponding maneuvers, thereby achieving closed-loop autonomous tracking of the Indo-Pacific humpback dolphin by the UAV. Figure 3 As shown. The specific steps are as follows:

[0138] S4.1: Design of visual servo control law based on image moments.

[0139] To achieve stable tracking of dynamic targets, this invention employs a decoupled image-based visual servoing (IBVS) control strategy to establish a mapping relationship between image feature errors and the velocity of the UAV's body coordinate system.

[0140] S4.1.1 Degrees of Freedom Decoupling Control: Decoupling the four control degrees of freedom (throttle, pitch, roll, yaw) of the UAV from visual feature errors. Perform the following mapping:

[0141] Yaw rate ): Due to horizontal centroid error Control. The goal is to eliminate the target's horizontal offset in the image, ensuring that the drone's nose is always pointed at the white dolphin.

[0142] Pitch / Forward Velocity ): Due to vertical centroid error Control. The goal is to adjust the horizontal distance between the drone and the target, keeping the target centered vertically in the image.

[0143] Vertical velocity (Throttle / Vertical Velocity) ): Due to area error Control. Utilizing the bounding box area. With depth inverse relationship ( When the detected dolphin area Smaller than expected area When the target moves further away or deeper, control the drone to descend. Conversely, it rises, thus maintaining a constant observation distance.

[0144] S4.2: Construction of an incremental PID controller.

[0145] To address the challenges of strong wind and wave interference in the marine environment and the nonlinearity of the UAV's dynamic model, an incremental PID (proportional-integral-derivative) algorithm is employed to calculate the control variables for each axis, thereby enhancing the system's anti-interference capability. (Based on forward speed...) Taking the calculation as an example, the first Control output increment at any time The calculation is as follows:

[0146]

[0147] The final output speed is: .

[0148] Similarly, using Calculate yaw rate ,use Calculate vertical velocity .in, These are the independent gain coefficients for each channel, which need to be tuned through flight experiments.

[0149] S4.3: Speed ​​command saturation limit and safety smoothing.

[0150] To prevent the drone from making violent movements that could disturb the white dolphins or cause instability due to jumps in the detection frame or excessive errors, the amplitude of the PID output needs to be limited.

[0151] Set the maximum permissible speed threshold If the calculated control value exceeds the threshold, it is truncated to the maximum value.

[0152]

[0153] In addition, the output speed command is low-pass filtered to ensure the smoothness of the control signal.

[0154] S4.4: Target loss relocation and anomaly handling strategies.

[0155] In real-world sea conditions, the detection of Indo-Pacific humpback dolphins may be lost due to deep dives or wave obstruction (i.e., the S3 module does not output a valid bounding box). This module is designed with the following state machine logic:

[0156] Tracking: When a target is continuously detected, normal servo control S3.1-S4.5 is executed.

[0157] Predicting the state: If the target is lost for a certain period of time. (Take 2 seconds here) Assuming that the dolphin's movement has inertia, keep the flight velocity vector of the previous moment unchanged, and continue to cover the prediction area by utilizing the drone's own movement.

[0158] Hover search status (Searching): If the target is lost for a period of time If the target is detected as lost, the drone immediately switches to hover mode and performs a slow, in-place yaw spin scan until the target is detected again or a return-to-home command is received.

[0159] S4.5: Command sending and execution.

[0160] Using the onboard SDK (such as DJI Onboard SDK or MAVLink protocol), the calculated and smoothed final speed command is... The package is sent to the drone's underlying flight controller, which drives the motors to perform the corresponding actions, completing the closed-loop tracking task.

[0161] The present invention provides an autonomous detection and tracking method for Indo-Pacific humpback dolphins using UAV-borne vision, comprising the following steps: First, real-time video streams are acquired using an UAV-borne camera. A lightweight deep learning model is used to quickly detect Indo-Pacific humpback dolphins at the edges and output target detection boxes. Then, the centroid and area of ​​the detection boxes are extracted as visual servo features. After optimization using Kalman filtering, these features are compared with the expected values ​​to generate a feature error vector. Based on this error, an incremental PID controller is used to calculate three-dimensional velocity control commands in the forward, yaw, and vertical directions, driving the UAV to adjust its attitude to reduce visual errors. Figure 4 As shown, the system forms a closed-loop feedback control of "perception-decision-execution". When the image feature error norm converges to within the preset threshold, it is determined to be a stable track and high-value observation data is automatically recorded. Otherwise, it continues to perform dynamic correction.

[0162] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of the technical solutions of the present invention.

Claims

1. A method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision, characterized in that, Includes the following steps: S1. Construct a database of images of Chinese white dolphins in a real-world environment. Use the Unity 3D engine to build a virtual simulation environment to expand the image data and preprocess and enhance the images in the database. S2. Use the image data from step S1 to train a high-precision Chinese white dolphin detection model, and use structured pruning technology to lightweight the model to address the computing power limitations of the UAV-borne edge computing device. S3. Input the real-time images collected by the UAV into the Chinese white dolphin detection model in S2 to obtain the Chinese white dolphin detection box information. Convert the above detection box information into a visual servo feature vector that can be used for UAV closed-loop control. By extracting the geometric information of the Chinese white dolphin bounding box, construct image features describing the relative position and distance of the target, and calculate the real-time servo deviation. S4. The PID control algorithm is used to calculate the three-dimensional speed control command of the UAV and drive the flight controller to perform the corresponding maneuver, thereby realizing the UAV's closed-loop autonomous tracking of the Chinese white dolphin.

2. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 1, characterized in that, The construction of the Indo-Pacific humpback dolphin image database includes the following steps: S1.1.1 Utilize drones to conduct aerial photography at multiple altitudes and angles in the target sea area, with the altitude range covering 15 to 40 meters, the viewing angle covering the gimbal pitch angle from 90 degrees vertically downward to 45 degrees tilted, and the subjects covered by the Chinese white dolphin at all ages, including gray-black calves, sub-adults with increased spots, and white or pink adults, as well as clear images including the dorsal fin shape and body spot patterns. S1.1.

2. Frame extraction processing is performed on the acquired image data to remove images with motion blur and out of focus. The LabelImg annotation tool is used to annotate the white dolphins in the images with rectangular boxes, recording the target's category label and coordinate information to form a real source domain dataset. .

3. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 2, characterized in that, The Unity 3D engine constructs a virtual simulation environment for image data augmentation, which includes the following steps: S1.2.

1. Create a high-fidelity 3D parametric model of the Chinese white dolphin in Unity, and implement its skeletal animation through script control to simulate the dolphin's swimming, emerging from the water, diving and rolling postures. S1.2.

2. Set the Euler angle parameters of the directional light source in the virtual environment. The X-axis rotation angle is used to simulate the solar altitude angle, with a random sampling range of 15° to 90°; the Y-axis rotation angle is used to simulate the solar azimuth angle, with a random sampling range of 0° to 360°; the light intensity has a random range of 0.8 to 2.0; simultaneously, the water material shader parameters are adjusted to simulate sea conditions, with the wave height set to a random range of 0.1 m to 1.0 m, the water transparency set to a random range of 0.4 to 0.9, and the RGB values ​​of the water color randomly interpolated within the blue-green and turbid yellow-brown range; randomize the extrinsic and intrinsic parameters of the virtual camera, setting the shooting height range to 15 m to 40 m, the camera pitch angle range to 45° to 90°, and the field of view range to 60° to 90°; superimpose Gaussian noise with a mean of 0 and a variance range of 0.001 to 0.02 into the image; and randomly generate reflective areas and floating objects on the sea surface to enhance the model's robustness to environmental disturbances; S1.2.

3. Utilizing Unity's scripting interface, while rendering the composite image, automatically obtain the projection coordinates of the 3D model in screen space, generate pixel-level bounding box annotation information, and batch export the composite image and its annotation files to form a virtual source domain dataset. .

4. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 3, characterized in that, The image preprocessing and enhancement includes: and Mix and then process as follows: S1.3.1 Enhance the image by performing random rotation, random cropping, and horizontal flipping to simulate different positions during drone shooting; perform random perturbation of brightness, contrast, and saturation to adapt to changing lighting environments; introduce Gaussian noise to simulate noise of the image sensor or interference of image transmission signals under high sensitivity; and introduce motion blur to simulate the body shaking of the drone during flight or the trailing shadows produced by the high-speed swimming of the Chinese white dolphin. S1.3.2 Divide the hybrid dataset after enhancement processing in S1.3.1 into training set, validation set and test set in a ratio of 8:1:1, and convert them into COCO format.

5. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 1, characterized in that, Step S2 includes: S2.1 Construct a deep neural network model with residual structure, using RTMDetection as the basic detection framework, which includes: S2.1.

1. The CSPNEXt backbone network is used as the feature extraction network; S2.1.2 Design PAFPN as a feature fusion layer, which fuses multi-scale feature maps through a bidirectional path from top to bottom and from bottom to top; S2.1.

3. A decoupled head structure is adopted to separate the task of classifying white dolphins from the task of regressing the bounding box position of dolphins. S2.2 Network model training and optimization: The network is trained under full supervision using the hybrid dataset generated in step S1. This includes: S2.2.1, Define the total loss function : ， in, This is the quality focus loss, used to optimize the joint distribution of classification scores and location confidence. For generalized intersection-union loss, it is used to regress the geometric overlap between the predicted bounding box and the ground truth bounding box; and These are the weighting coefficients; S2.2.2 Use the AdamW optimizer to update parameters, setting the initial learning rate to 0.

001. Employ cosine annealing to adjust the learning rate until the model's mean accuracy on the validation set converges, thus obtaining the baseline model. ; S2.

3. Sparse training and structured pruning of the model based on the BN layer scaling factor, including: S2.3.1, in the total loss function The L1 regularization term is introduced to scale all batch-normalized BN layers in the model. By applying sparsity constraints, the corrected loss function is obtained as follows: ， in, Let be the sparsity penalty coefficient, and take . =0.001, Given the set of scaling factors for all BN layers, sparse training forces the channels that contribute less to the network output to have different scaling factors. The value tends to 0. S2.3.2 Obtaining the layers after sparse training Distribution, setting a global pruning threshold For all The top 30% quantiles after sorting are traversed through the network structure to remove [the remaining quantiles]. By analyzing the channels and their convolutional kernel weights, the pruned model structure can be obtained. ; S2.4: Fine-tuning and solidification of the model, Retraining is performed on the image database built by S1, using a small learning rate for iteration. When the accuracy of the fine-tuned model recovers to more than 95% of the baseline model and the model size meets the requirements for airborne deployment, training is stopped and the final lightweight model weight file is exported.

6. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 1, characterized in that, Step S3 includes the following steps: S3.1 Input the real-time images captured by the UAV into the Indo-Pacific humpback dolphin detection model in S2 to obtain the Indo-Pacific humpback dolphin detection box information, and extract the pixel coordinate information of the four vertices of the detection box: top left corner... Top right corner bottom right corner and bottom left corner Before feature extraction, a geometric rationality check is performed. If the aspect ratio of the detection box exceeds the preset range of biological features of the white dolphin, or if the edge of the detection box touches the image boundary, the current frame feature is marked as invalid and the previous frame state is maintained. S3.2 Select the centroid coordinates and bounding box area of ​​the target as the core visual servo features, which are used to control the horizontal position and flight altitude of the UAV, respectively. These features include: S3.2.1 Calculate the geometric center of the target on the image plane using the coordinates of its four corner points. The formula used to characterize the horizontal relative position of the drone and the dolphin is as follows: ， S3.2.2 Calculate the bounding box area using information from the four corner points. The value is used to characterize the relative distance between the drone and the dolphin, and the calculation formula is: ； S3.2.3 Constructing the current moment Image feature vectors ; S3.3, Use a Kalman filter to process the eigenvectors Perform optimal estimation and establish state equations. ,in The rate of change of the eigenvectors; establish the state equation. and observation equations , here For the observation matrix, take This indicates that only location information was observed; the prediction and update steps include: S3.3.1 Prediction Steps: Based on the posterior state estimate from the previous time step Using the state transition matrix Estimate the prior state value at the current time. The calculation formula is: Simultaneously, the prior error covariance matrix is ​​derived. The calculation formula is: ,in The process noise covariance matrix; S3.3.2 Update Steps: Calculate Kalman Gain This is used to balance the covariance of prediction error and the covariance of observation noise; the calculation formula is as follows. ,in To observe the noise covariance matrix; S3.3.3, State Correction and Output: Combining the observation vector at the current time step The Kalman gain is used to correct the prior state estimate to obtain the optimal posterior state estimate at the current time. The calculation formula is: Then, by selecting the matrix... Extracting smoothed feature vectors The calculation formula is: ,in This yields the smoothed feature vector after removing Gaussian white noise. . S3.4 Calculate the real-time servo deviation, which includes: S3.4.1 Define the ideal image state for drone tracking of Indo-Pacific humpback dolphins. .in, and Represented as the physical center coordinates of the image plane. This represents the expected pixel area of ​​an adult white dolphin at a reference height. S3.4.2 Calculate the current smoothing feature With expected features The difference between them is used to generate a visual servoing error vector. The calculation formula is: 。 7. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 1, characterized in that, Step S4 includes: S4.

1. A decoupled image-based visual servo control strategy is adopted to establish a mapping relationship between image feature errors and the velocity of the UAV body coordinate system. S4.2 Construct an incremental PID controller; S4.3 Limit the amplitude of the PID output and set the maximum allowable speed threshold. If the calculated control quantity exceeds the threshold, it is truncated to the maximum value. ， In addition, the output speed command is low-pass filtered; S4.4, Target loss relocation and exception handling strategy; S4.

5. Using the onboard SDK, the calculated and smoothed final speed command is... The package is sent to the drone's underlying flight controller, which drives the motors to perform the corresponding actions, completing the closed-loop tracking task.

8. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 1, characterized in that, When establishing the mapping relationship in step S4.1, the four control degrees of freedom of the UAV—throttle, pitch, roll, and yaw—are linked to the current moment. Visual feature error Perform the following mapping: Yaw angular velocity: determined by horizontal centroid error Control and eliminate the target's horizontal deviation in the image, ensuring that the drone's nose is always pointed at the white dolphin; Forward speed: determined by vertical centroid error Control and adjust the horizontal distance between the drone and the target to keep the target centered in the vertical direction of the image; Vertical velocity: determined by area error Control, utilizing the bounding box area With depth The inverse relationship, when the detected dolphin area Smaller than expected area When the drone descends, it is controlled to descend; conversely, it ascends.

9. The method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision as described in claim 1, characterized in that, The construction of the incremental PID controller includes: An incremental PID algorithm is used to calculate the speed increment of each control channel and the final output value in parallel: (1) Forward speed control: Calculate the control increment Final output The formula for calculating the increment is: ； (2) Yaw rate control: utilizing horizontal centroid error Calculate control increment Final output The formula for calculating the increment is: ； (3) Vertical speed control: utilizing area error Calculate control increment Final output The formula for calculating the increment is: ； in, and These are the error values ​​for the previous time step and the time step before that, respectively. These are the independent proportional, integral, and differential gain coefficients for each channel, which need to be tuned through flight experiments.

10. A method for autonomous detection and tracking of Indo-Pacific humpback dolphins using UAV-borne vision according to claim 1, characterized in that, The specified target loss relocation and anomaly handling strategy includes: Tracking status: When targets are continuously detected, repeat steps S3-S4; Predicted state retention: If the target is lost for a certain period of time Assuming that the dolphin's movement has inertia, the flight velocity vector of the previous moment remains unchanged, and the drone's own movement continues to cover the predicted area; Hover search status: If the target is lost for a period of time If the target is determined to be lost, the drone immediately switches to hover mode and performs a slow, rotating scan in place until the target is detected again or a return-to-home command is received.

Images