Multi-target detection and tracking method

Abstract

The invention discloses a multi-target detection and tracking method, and relates to the technical field of computer vision and intelligent monitoring. The method comprises the following steps: acquiring multi-source video data of an unmanned aerial vehicle and a middle-high point fixed camera, and after scene adaptation preprocessing, outputting a target detection frame by using a multi-scale detection model fused with scene context; block enhanced appearance features and geometrical relationship features of the target are extracted to construct a dynamic feature library, and an initial track is generated based on a multi-stage adaptive association mechanism; through a child-mother type multi-machine collaborative optimization track, linkage control is triggered in combination with abnormal behavior analysis, and close-range evidence obtaining of the unmanned aerial vehicle and linkage of fixed equipment recording are controlled. According to the method, the multi-source data fusion capability, the multi-scale target detection precision and the trajectory association robustness in a complex scene are improved, and intelligent management and control requirements in the fields of traffic, forestry and the like can be efficiently supported.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and intelligent monitoring technology, and in particular to a multi-target detection and tracking method. Background Art

[0002] Continuous and accurate detection and tracking of multiple targets (pedestrians, vehicles, etc.) in wide-area scenarios (such as transportation hubs, city squares, and large-scale event venues) is crucial in areas such as smart city management, public safety monitoring, and large-scale event security. Existing technologies typically rely on single-viewpoint surveillance systems, such as widely deployed fixed high-point cameras or single drone patrols. However, these traditional approaches have significant limitations when dealing with complex, dynamic, and large-scale scenarios: Cross-viewpoint trajectory discontinuity: The field of view of a fixed camera is fixed and may be obstructed, while the field of view of a single drone is flexible but has limited coverage and may be constrained by endurance and flight path. When a target moves between the fields of view of multiple fixed cameras or enters and exits the field of view of a drone, it is difficult to achieve continuous and seamless association of the target's identity. For example, a target moves from the field of view of fixed camera A into the field of view of a drone and then into the field of view of fixed camera B. Due to the drastic changes in viewpoint, resolution differences, and lack of effective coordination between different sensors, existing methods often result in target trajectory discontinuity or frequent ID switching, making it impossible to form a complete and coherent motion trajectory. This greatly limits the ability to analyze cross-regional target behavior and overall situational awareness.

[0003] Inadequate tracking robustness in complex scenarios: In complex scenarios such as dense crowds, severe occlusion, and targets with similar appearance or rapid deformation (such as pedestrians turning or vehicles turning), existing tracking algorithms primarily rely on single appearance features or simple motion models. These features are susceptible to interference from factors such as ambient lighting changes, partial occlusion, and viewpoint differences, leading to high target association errors, unstable tracking trajectories, and even target loss. Recapturing and correctly associating targets is particularly challenging when targets are occluded for extended periods in densely populated areas or undergo significant appearance changes.

[0004] Therefore, there is an urgent need for a multi-target detection and tracking method that can effectively integrate multi-source heterogeneous video data (such as the perspective of aerial drones and the perspective of fixed high points on the ground), overcome cross-perspective correlation obstacles, and maintain high robustness in complex scenarios, so as to achieve continuous, accurate and seamless characterization of target motion trajectories in a wide area, laying a solid foundation for subsequent abnormal behavior identification and linkage response. Summary of the Invention

[0005] To address the above problems, the present invention provides a multi-target detection and tracking method, which aims to achieve accurate detection, stable tracking and rapid response to abnormal behaviors of multiple targets in complex scenarios through multi-source data fusion, multi-scale detection, dynamic feature association and multi-machine collaborative optimization.

[0006] In a first aspect, the present invention provides a multi-target detection and tracking method, comprising the following steps: S1, obtaining multi-source linkage video data, wherein the multi-source linkage video data includes an aerial video frame sequence collected by a drone and a ground video frame sequence collected by a fixed camera at a mid-high point, and performing scene adaptation preprocessing on the video frame sequence; S2: Build a multi-scale detection model that integrates scene context and outputs multi-target detection boxes and confidence scores. S3, extracts the multi-dimensional feature vector of the detected target, including the appearance features based on block enhancement and the geometric relationship features based on the relative positions of the targets, and builds a dynamic feature library; S4, the tracking mechanism based on multi-level adaptive association generates the initial trajectory; S5, trajectory optimization based on parent-child multi-machine collaboration; S6 performs abnormal behavior analysis based on tracking trajectories. When a violation or anomaly is detected, the linkage control mechanism is triggered, controlling the drone to adjust its path for close-range evidence collection and linking it with fixed cameras for multi-angle recording to generate an alarm message.

[0007] Furthermore, the performing scene adaptation preprocessing on the video frame sequence specifically includes: Histogram block local equalization processing is used for UAV video frames to enhance small target details, and feature point matching based on the SuperPoint model is used for image registration of fixed camera video frames to correct camera offset.

[0008] By applying histogram-based local equalization to drone video frames, we can specifically enhance the detailed features of small targets (such as long-range vehicles and low-altitude drones), resolving the difficulty in detecting small targets from a drone's perspective due to their distance and blurred imaging. Image registration of fixed camera video frames using feature point matching based on the SuperPoint model effectively corrects for camera drift caused by installation offset and environmental vibration, ensuring spatial consistency in ground-based video frames. The combination of these two methods provides high-quality preprocessing data for subsequent multi-target detection and cross-device tracking, improving the usability of multi-source video data in complex scenarios.

[0009] Furthermore, the drone video frames are processed using histogram block local equalization to enhance small target details, specifically including: The drone video frames are divided into non-overlapping area blocks. The grayscale histogram is calculated for each area block. The cumulative distribution function is calculated based on the histogram, and the grayscale frequency limit value is set. When the number of pixels at a certain grayscale level exceeds the preset limit value, the number of pixels exceeding the limit is evenly distributed to other grayscale levels that do not reach the limit value. Local equalization is performed based on the cumulative distribution function, and bilinear interpolation is used to smooth the transition between area blocks to enhance the contour details of distant vehicles.

[0010] The histogram-based local equalization applied to drone video frames specifically addresses the core issue of detail loss due to distance and blurry imaging of small targets (such as long-range vehicles and low-altitude small objects) in drone aerial perspectives. By dividing the video frames into non-overlapping blocks and performing separate grayscale histogram analysis and equalization, this approach avoids the drawbacks of global equalization, which can easily lead to overexposure of bright areas and masking of dark details, and accurately enhances the texture and contour details of small targets within each local area. Setting a grayscale frequency limit and redistributing the number of pixels exceeding the threshold effectively suppresses the "overwhelming" of target details caused by excessive grayscale levels in the image, further highlighting the grayscale difference between small targets and the background. Bilinear interpolation smoothing between blocks eliminates possible splicing artifacts caused by block processing, ensuring overall visual coherence and preventing local abruptness from interfering with feature extraction in subsequent detection models. Ultimately, this processing provides subsequent multi-scale detection models with drone video data with high detail clarity and low noise interference, significantly improving the detection accuracy of small targets in complex aerial scenes and laying a high-quality aerial perspective data foundation for multi-source data fusion and tracking.

[0011] Furthermore, the image registration of the fixed camera video frame using feature point matching based on the SuperPoint model to correct the camera offset specifically includes: Extract feature points of two consecutive frames of images through the SuperPoint model; The k-nearest neighbor algorithm is used to find multiple feature points with the closest distance in the reference frame for each feature point in the current frame, and the initial matching point pairs are screened by the nearest neighbor distance ratio threshold; The RANSAC algorithm is used to fit the perspective transformation model, and the incorrect matching point pairs are eliminated with the inlier threshold to obtain the optimized correct matching point pairs; Based on the correct matching point pairs, the homography matrix is ​​calculated, the lane line coordinates and the target detection frame in the current frame are perspective transformed, mapped to the reference frame coordinate system, and the camera offset is corrected.

[0012] The SuperPoint model-based feature point matching image registration scheme used for fixed camera video frames effectively solves the camera offset problem caused by installation offset and environmental vibration, and ensures the spatial consistency of ground video frames. The SuperPoint model can extract robust feature points with scale invariance and rotation invariance, adapting to the feature matching needs of fixed cameras in dynamic environments. Compared with traditional feature extraction methods (such as SIFT and ORB), it is more adaptable to illumination changes and partial occlusion. The k-nearest neighbor algorithm combined with the nearest neighbor distance ratio threshold screening mechanism can initially eliminate erroneous matching point pairs with low similarity, reducing noise interference in subsequent processing. The RANSAC algorithm further improves the accuracy of feature point matching by robustly fitting the perspective transformation model and eliminating erroneous matching point pairs, ensuring that the matching point pairs truly reflect the spatial transformation relationship between the two frames. The homography matrix calculated based on the correctly matched point pairs can accurately map the lane line coordinates and target detection box of the current frame to the reference frame coordinate system, completely offsetting the position deviation caused by camera offset and ensuring that the ground video frames at different times have a unified coordinate reference. This solution effectively avoids cross-frame target position misalignment, trajectory breakage, or ID switching problems caused by camera offset, provides a stable ground perspective spatial reference for subsequent multi-target trajectory association, and significantly improves the continuity and robustness of multi-target tracking in ground scenes.

[0013] Furthermore, the multi-scale detection model uses the YOLO11 model as a benchmark model, including a backbone network, a neck network, and a head network; The backbone network is based on Yolov7-tiny, with three additional multi-scale feature extraction sublayers. It uses convolution kernels of different sizes for objects of different sizes and fuses feature maps of different depths through skip connections. The neck network adopts an improved feature pyramid network and path aggregation network structure to integrate high-level semantics and low-level detail features, while adding scene context features. The head network adopts a two-branch structure. The classification branch outputs category confidence through an improved Resnet-18, and the regression branch outputs the detection box coordinates. The loss function is the weighted sum of the classification loss and the positioning loss.

[0014] The backbone network enhances its ability to perceive objects of varying sizes by adding a multi-scale feature extraction sublayer and employing convolutional kernels of varying sizes, addressing the limited adaptability of traditional models to multi-scale objects. The neck network fuses high-level semantics with low-level detail features and incorporates scene context (such as lane lines and terrain edges), improving the model's understanding of complex scenes. The head network employs a dual-branch structure and optimizes the loss function to increase the weight of the loss for small object localization, further enhancing the accuracy of the detection box and the reliability of the category confidence. The overall model significantly improves the accuracy and robustness of multi-object detection in complex environments.

[0015] Furthermore, the S3 specifically includes: S31: Divide the target image into sub-regions based on morphological features. The Transformer's QKV attention mechanism is used to process the features of each sub-region. Self-attention enhances the texture and color details within a single region, while cross-attention integrates the structural associations of different regions. The fully connected layer outputs the appearance feature vector. S32, calculating the relative distance and angle between the target and surrounding targets, as well as the mean and standard deviation of the target center coordinates in the local area, and constructing a geometric relationship vector; S33, concatenate the appearance feature vector and the geometric relationship vector into a multidimensional feature vector, store it according to the target identity number, update the feature vector in each frame according to the detection result, archive the target features that have not been matched for more than a preset time, and retain the feature history of the last few frames for re-identification and association.

[0016] The QKV attention mechanism of Transformer is used to extract block-enhanced appearance features, which can capture fine-grained differences in targets, such as the structural features of the drone body and arms, and the local texture of the vehicle, solving the identity confusion problem caused by the similar appearance of similar targets; constructing geometric relationship features based on the relative positions of targets enhances the scene adaptability of target association and reduces the limitations of single appearance feature association; the dynamic feature library provides data support for target re-identification association by real-time updating, archiving and retaining historical features, ensuring the identity continuity of targets during short-term occlusion and cross-device tracking, and improving the robustness of multi-target association.

[0017] Furthermore, the S4 specifically includes: S41, calculating the scene dynamic confidence threshold by gradient descent method and classifying the detection frame; S42, construct an improved Kalman filter, the state vector contains position, size, velocity and rotation angular velocity, and combines the affine transformation matrix of global motion estimation to achieve camera offset compensation; S43, for high-confidence detection boxes, double association is performed using intersection-over-union (IoU) and re-identification features, and for low-confidence detection boxes, secondary IoU association is performed, and the initial trajectory is generated through the Hungarian algorithm.

[0018] By calculating the dynamic confidence threshold of the scene through the gradient descent method, scene adaptation of the detection frame classification is achieved, avoiding the inapplicability of fixed thresholds to dynamic scenes; the improved Kalman filter introduces state vectors such as rotational angular velocity and combines them with the affine transformation matrix to accurately compensate for camera offset (such as drone jitter and camera vibration), thereby improving the accuracy of target motion state prediction; for high / low confidence detection frames, a differentiated association strategy (double association / quadratic intersection-over-union association) is adopted, combined with the Hungarian algorithm to generate the initial trajectory, effectively reducing the error rate of target association, solving the problem of insufficient robustness of the traditional single association strategy in complex scenes, and ensuring the coherence of the initial trajectory.

[0019] Furthermore, the S5 specifically includes: The master machine generates the multi-target minimum enclosing circle and plans the global coverage trajectory through the Welzl algorithm, and the slave machine uses the B-spline curve to generate the local tracking trajectory. The algorithm implements dynamic obstacle avoidance and multi-machine collision detection, and outputs optimized trajectories.

[0020] Furthermore, the master machine generates the multi-target minimum enclosing circle and plans the global coverage trajectory through the Welzl algorithm, and the slave machine uses the B-spline curve to generate the local tracking trajectory. The algorithm implements dynamic obstacle avoidance and multi-machine collision detection, and outputs optimized trajectories, including: S51: The master aircraft predicts the future position based on the historical trajectory of multiple targets and uses the Welzl algorithm to generate a minimum enclosing circle containing the future position. If the radius of the enclosing circle is larger than the sensing range of the master aircraft's lidar, outlier targets that are out of range are removed and a new minimum enclosing circle is generated, with the center of the new enclosing circle as the trajectory endpoint. S52, optimizing the B-spline curve control points of the local tracking trajectory of the slave, introducing an obstacle avoidance cost function, and calculating the relative distance between the control points and the nearest obstacle using the Euclidean signed distance field to ensure that the distance between the slave and the obstacle is not less than a first safety threshold, while constraining the relative distance between the slave and the target to be within a preset range; S53, based on the improved The algorithm generates a collision-free local path, introduces a multi-machine collision avoidance potential field into the path evaluation, outputs a path point in combination with the obstacle potential field, and incorporates the path point into the B-spline trajectory optimization of the sub-machine.

[0021] The master machine generates the minimum enclosing circle of multiple targets and plans the global coverage trajectory through the Welzl algorithm, which realizes the global control of the distribution of multiple targets and ensures the comprehensiveness of the tracking range; the slave machine uses the B-spline curve to generate the local tracking trajectory, which ensures the smoothness and continuity of the trajectory; combined with the improved The algorithm implements dynamic obstacle avoidance and multi-machine collision detection, resolving the path conflict issue during multi-machine collaborative tracking. The combination of these three enables the synergy of global coverage and local precision tracking, improving the efficiency and safety of multi-machine collaborative tracking.

[0022] In a second aspect, the present invention further provides a computer terminal, comprising: a memory storing an executable program; A processor is used to run the program, wherein the program executes the multi-target detection and tracking method when running.

[0023] In a third aspect, the present invention also provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the multi-target detection and tracking method.

[0024] In a fourth aspect, the present invention further provides a computer program product, comprising a computer program, which implements the multi-target detection and tracking method when executed by a processor.

[0025] Compared with the existing technology, the beneficial effects of the present invention are as follows: by integrating the multi-source linkage video data of drones and fixed cameras at medium and high points, the complementarity of aerial and ground perspectives is achieved, the limitation of the perspective of a single device is broken through, and the comprehensive coverage capability of complex scenes is improved; by constructing a multi-scale detection model that integrates the scene context, combining multi-dimensional feature extraction and dynamic feature library construction, the ability to accurately identify and distinguish multi-scale targets is enhanced, and the problem of easy confusion of similar targets is solved; the tracking mechanism based on multi-level adaptive association and parent-child multi-machine collaborative trajectory optimization ensures the continuity of the target trajectory and the efficiency of tracking, and reduces the interference caused by equipment offset, target occlusion, etc.; finally, through abnormal behavior analysis and linkage control mechanism, a closed loop from detection and tracking to intelligent response is realized, which effectively supports intelligent management in fields such as traffic control and forestry monitoring, and significantly improves the ability to quickly deal with violations or abnormal behaviors, and has strong practicality and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present drawings or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present drawings. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without any creative work.

[0027] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0029] The present invention provides a multi-target detection and tracking method, which specifically includes the steps of: S1, obtaining multi-source linkage video data, wherein the multi-source linkage video data includes an aerial video frame sequence collected by a drone and a ground video frame sequence collected by a fixed camera at a mid-high point, and performing scene adaptation preprocessing on the video frame sequence.

[0030] Furthermore, histogram block local equalization processing is used for UAV video frames to enhance small target details, and feature point matching based on the SuperPoint model is used for image registration of fixed camera video frames to correct camera offset.

[0031] Specifically, the preprocessing of drone video frames includes: dividing the drone video frames into non-overlapping area blocks, calculating the grayscale histogram for each area block, calculating the cumulative distribution function based on the histogram, setting the grayscale frequency limit value, and when the number of pixels (frequency) of a certain grayscale level exceeds the preset limit value, the number of pixels exceeding the limit will be evenly distributed to other grayscale levels that have not reached the limit value; local equalization is performed based on the cumulative distribution function, and bilinear interpolation is used to smooth the transition between area blocks to enhance the contour details of distant vehicles.

[0032] Preprocessing of video frames from a fixed camera involves extracting feature points from two consecutive frames using the SuperPoint model. The k-nearest neighbor algorithm is then used to find matching pairs of feature points from the two frames. For each feature point in the current frame, the two closest feature points are found in the other frame. By setting a distance threshold, such as the nearest neighbor distance ratio, initial matching point pairs with high similarity are selected, establishing a preliminary correspondence between the feature points in the two frames.

[0033] Due to camera vibration, occlusion, or interference from similar textures, there may be mismatched point pairs in the initial matching. The RANSAC algorithm iteratively and randomly selects matching points to fit the perspective transformation model. The inlier threshold is used to determine whether the points conform to the model. Points with a deviation less than or equal to the inlier threshold after transformation are considered inliers. The algorithm ultimately retains the matching point pairs corresponding to the model with the most inliers, effectively eliminating mismatches and improving matching accuracy.

[0034] Based on the correctly matched point pairs selected, a homography matrix is ​​calculated. This matrix describes the perspective transformation relationship between the two image frames, including parameters such as translation, rotation, and scaling. This matrix is ​​used to perform a perspective transformation on the lane line coordinates and vehicle detection frame in the current frame and map them to the coordinate system of the reference frame (such as the previous frame). This offsets positional offsets caused by camera vibration, ensures spatial consistency of target positions across different video frames, and provides a stable coordinate reference for subsequent multi-target detection and tracking.

[0035] S2 builds a multi-scale detection model that integrates scene context and outputs multi-target detection boxes and confidence levels.

[0036] Specifically, the multi-scale detection model uses the YOLO11 model as the benchmark model, including the backbone network, neck network and head network.

[0037] The backbone network is based on Yolov7-tiny. Three multi-scale feature extraction sublayers are added after its original convolutional layer. Convolution kernels of different sizes are used for targets of different sizes. Feature maps of different depths are fused through jump connections to enhance the perception of multi-scale targets in complex scenes.

[0038] The neck network adopts an improved feature pyramid network FPN and path aggregation network PAN combined structure. The FPN layer upsamples the high-level semantic features and fuses them with the middle-level features. The PAN layer downsamples the middle-level features and fuses them with the low-level detail features. At the same time, scene context features are added in the fusion process, such as lane line features in traffic scenes and terrain edge features in forestry scenes, to improve the detection accuracy of scene-related targets.

[0039] The head network adopts a two-branch structure. The classification branch outputs category confidence through an improved Resnet-18 network, and the regression branch outputs the detection box coordinates. The loss function is the weighted sum of classification loss and positioning loss.

[0040] S3 extracts the multi-dimensional feature vector of the detected target, including the appearance features based on block enhancement and the geometric relationship features based on the relative positions between targets, and builds a dynamic feature library.

[0041] Specifically, the above S3 includes the following contents: S31, the detection target image is divided according to morphological features to obtain sub-regions after division, and the QKV attention mechanism of Transformer is used to process the features of each sub-region. The self-attention mechanism strengthens the texture and color details in a single region, and the cross-attention mechanism fuses the structural associations of different regions. The appearance feature vector is output through the fully connected layer.

[0042] Specifically, when the detection target is a vehicle, it is divided into three sub-regions according to the front, body, and parking space. For each sub-region, the correlation between pixels is calculated through the self-attention matrix. The formula is as follows: ; Where Q is the query matrix; K is the bond matrix, is the transposed matrix of the key matrix; V is the value matrix; is the dimension of each eigenvector in the key matrix K.

[0043] After Transformer encoding, the appearance feature vector is output through two fully connected layers.

[0044] S32, calculating the relative distance and angle between the target and surrounding targets, as well as the mean and standard deviation of the target center coordinates in the local area, and constructing a geometric relationship vector.

[0045] Specifically, a local area is delineated with the target as the center, and the spatial relationship features with other targets in the area are calculated. The specific steps are as follows: Set the center coordinate of target t to , filter to meet The surrounding target i, where is the center coordinate of target i, and R is the preset distance threshold parameter.

[0046] Calculate the Euclidean distance between target t and target i: .

[0047] With target t as the origin and the horizontal right as the positive direction of the x-axis, the angle of target i is: , is the four-quadrant inverse tangent function.

[0048] Calculate the mean and standard deviation of the center coordinates of all targets in the local area: ; ; Where m is the number of surrounding targets in the local area; is the mean x-coordinate of all target centers in the local area; is the mean of the y coordinates of all target centers in the local area; is the center x coordinate of the kth target; is the center y coordinate of the kth target; is the standard deviation of the x-coordinates of all target centers in the local area, which measures the degree of dispersion of the coordinates in the x-direction; It is the standard deviation of the y coordinates of all target centers in the local area, which measures the degree of discreteness of the coordinates in the y direction.

[0049] Take the first three nearest neighbors of m surrounding targets and , combined with the calculated mean and standard deviation, a geometric relationship vector is constructed.

[0050] S33, concatenate the appearance feature vector and the geometric relationship vector into a multidimensional feature vector, store it according to the target identity number, update the feature vector in each frame according to the detection result, archive the target features that have not been matched for more than a preset time, and retain the feature history of the last few frames for re-identification and association.

[0051] Specifically, the appearance feature vector and the geometric relationship vector are merged into a multi-dimensional feature vector, while retaining visual recognition and spatial context information. Each target is assigned a unique identity ID, and its multi-dimensional feature vector is stored in the dynamic feature library. If the target is detected in the current frame, the new feature vector overwrites the old data to reflect the latest state of the target; if the target is not detected, such as temporary occlusion, the last feature is retained and waits for subsequent matching. The system is preset to tolerate the maximum time window for target disappearance. If the target is not re-matched within the set time, its features are removed from the dynamic feature library and transferred to the archive library to reduce the burden of real-time retrieval. The most recent multi-frame historical feature vectors are retained for each target. When the target reappears, the system compares the current features with the historical feature library. If the match is successful, the original ID is restored to ensure trajectory continuity; if the match fails, a new ID is reallocated and it is considered a new target.

[0052] S4, generating the initial trajectory based on the multi-level adaptive association tracking mechanism, specifically including the following steps: S41, calculate the scene dynamic confidence threshold by gradient descent method and classify the detection frame.

[0053] Specifically, we extract detection box samples from the current frame and historical frames, annotate the true target (positive sample) and false positive / background (negative sample), and construct the following binary cross entropy loss function to quantify the deviation between the confidence and the true label: ; is the loss function value; in, is the original confidence of the i-th detection box; θ is the confidence threshold to be optimized; N is the total number of samples; is the true label of the i-th sample; σ is the Sigmoid activation function.

[0054] The above loss function is iteratively optimized through the gradient descent algorithm to solve the confidence threshold that minimizes the classification error. The specific formula is as follows: .

[0055] in, is the confidence threshold of the tth iteration; is the confidence threshold for the t+1th iteration is the learning rate; is the gradient of the loss function.

[0056] The threshold is dynamically adjusted according to the scenario. For example, the threshold is increased in congested scenarios to reduce false detections, and the threshold is lowered in sparse scenarios to avoid missed detections.

[0057] According to the optimization results, a high confidence threshold and a low confidence threshold are set to divide the detection frame into a high confidence detection frame and a low confidence detection frame.

[0058] S42, construct an improved Kalman filter, the state vector contains position, size, speed and rotation angular velocity, combined with the affine transformation matrix of global motion estimation to achieve camera offset compensation.

[0059] Specifically, the state vector dimension of the traditional Kalman filter is expanded to include the target's position, size, movement speed, and rotation angular velocity, comprehensively characterizing the target's movement and morphological changes.

[0060] Based on the uniform motion assumption, a state transfer matrix is ​​constructed. The time interval between adjacent frames is used to update the state vector through the Kalman filter to achieve a preliminary prediction of the target's position, size, speed and rotation angular velocity in the next frame.

[0061] The Oriented Fast and Rotated Brief (ORB) feature matching algorithm is used to extract feature points from two consecutive frames and establish correspondences. The ORB matching process essentially calculates the displacement of feature point pairs, providing sparse optical flow information for estimating global motion. Combined with the Random Sample Consensus (RANSAC) algorithm, the affine transformation matrix describing the camera's motion is robustly estimated.

[0062] The inverse of the affine transformation matrix is ​​used to transform the target position predicted by the Kalman filter. This step aims to eliminate target position prediction errors caused by camera shake or offset, significantly improving prediction accuracy. The predicted target size and angular velocity are directly used in the Kalman prediction results, and the corrected target state prediction is output.

[0063] S43, for high-confidence detection boxes, double association is performed using intersection-over-union (IoU) and re-identification features, and for low-confidence detection boxes, secondary IoU association is performed, and the initial trajectory is generated through the Hungarian algorithm.

[0064] Specifically, the high confidence detection box association adopts the dual measurement of intersection-over-union (IOU) and re-identification features: IOU metric: Calculates the overlap between the current detection box and the historical trajectory prediction box (obtained by the corrected target state prediction result output by S42) to reflect the spatial position consistency; Re-identification feature matching: Extract the historical appearance feature vector of the target trajectory from the dynamic feature library, calculate the cosine similarity between the appearance feature vector of the current detection box and the extracted historical appearance feature vector, and reflect the consistency of the target identity; Fusion score: The weighted fusion of the above two indicators forms a comprehensive correlation score as the basis for correlation reliability.

[0065] Low-confidence detection box association: Perform a secondary IOU association on low-confidence detection boxes and detection boxes that are not matched after high-confidence association to obtain the secondary overlap. Then set the IOU threshold and only retain the overlap matching pairs above the IOU threshold to avoid incorrect associations caused by feature noise.

[0066] Trajectory generation: Construct an association cost matrix, where each element C(i, j) in the matrix represents the cost of associating the i-th detection box with the j-th trajectory. The association cost of a successfully matched high-confidence detection box i and trajectory j is (1-the comprehensive association score of the detection box i and trajectory j). The association cost of a successful quadratic IOU match between a detection box i and trajectory j is (1-the quadratic overlap of the detection box i and trajectory j). The association cost of a detection box i and trajectory j that fails in both matches or is physically impossible to associate is set to infinity.

[0067] The Hungarian algorithm is used to solve the optimal matching of the associated cost matrix through matrix transformation to achieve one-to-one correspondence of targets across frames.

[0068] Successfully matched detection frame and trajectory: Update the status of the trajectory, including position, speed, angular velocity, etc.

[0069] Unmatched high-confidence detection box: Initialize as a new trajectory and assign a new identity ID to the detection box; initialize a new trajectory using the detection box's position, size, and other information; extract the appearance features of the detection box and calculate its geometric relationship features, store them in the dynamic feature library and associate them with the new ID; initialize the motion state of the trajectory.

[0070] Unmatched low-confidence detection boxes: No track is generated for now, pending verification in subsequent frames.

[0071] S5, trajectory optimization based on parent-child multi-machine collaboration, specifically includes the following steps: S51: The mother machine predicts the future position based on the historical trajectory of multiple targets and calls the Welzl algorithm to generate a minimum enclosing circle containing the future position. If the radius of the enclosing circle is larger than the sensing range radius of the mother machine's lidar, the outlier targets that exceed the range are removed and the minimum enclosing circle is regenerated, with the center of the new enclosing circle as the end point of the trajectory.

[0072] S52: Optimize the B-spline curve control points of the slave's local tracking trajectory. At the same time, introduce an obstacle avoidance cost function. Use the Euclidean signed distance field to calculate the relative distance between the control point and the nearest obstacle. Ensure that the distance between the slave and the obstacle is not less than a first safety threshold. At the same time, constrain the relative distance between the slave and the target to be within a preset range. Generate a smooth and obstacle-avoiding local tracking trajectory for each slave to ensure safe approach to the target.

[0073] Specifically, based on the current position of the slave, the current position of the target, and possible intermediate points, a set of control points is initially set. These initial control points define a rough B-spline curve, namely ,in is the coordinate of the i-th control point, is the p-order B-spline basis function, are the coordinates of the point on the curve, u is the curve parameter, and n is the number of control points minus 1.

[0074] Construct a Euclidean Signed Distance Field (ESDF) and input known environmental obstacle information into the ESDF algorithm. The ESDF algorithm then calculates the value of each grid cell in the entire planning space. This value represents the distance from the control point to the nearest obstacle surface. When this value is less than the first safety threshold, a quadratic penalty is imposed. The smaller the distance, the greater the penalty, which strongly drives the control point away from the obstacle, ensuring that the points on the final trajectory (obtained by interpolation of the control points) also meet the safety distance. The cost function is expressed as follows: ; in, is the obstacle avoidance cost of the control point, the larger the value, the more dangerous it is; w is the obstacle avoidance cost weight coefficient; max is the maximum value function; is the first safety threshold; The distance from the control point to the nearest obstacle.

[0075] The gradient descent method is used to find a set of optimal control points to minimize the cost function while ensuring that the relative distance between the slave machine and the target is within a preset range.

[0076] S53, based on the improved The algorithm generates a collision-free local path, introduces a multi-machine collision avoidance potential field into the path evaluation, outputs a path point in combination with the obstacle potential field, and incorporates the path point into the B-spline trajectory optimization of the sub-machine.

[0077] The algorithm is a classic heuristic search algorithm used to find the shortest path between two points in a grid graph or point cloud space. It determines the search direction by evaluating the cost of each candidate node: f(m) = g(m) + h(m), where g(m) is the actual cost of node m and h(m) is the estimated cost from node m to the target point.

[0078] Multi-drone collision avoidance potential field: When the relative distance between any two sub-drones is less than the preset second safety threshold, a repulsive potential field is generated. The value of this potential field increases exponentially as the distance decreases, forcing the sub-drones to automatically maintain a safe distance during collaborative tracking to avoid collision.

[0079] Obstacle potential field: Generates a repulsive potential field based on static / dynamic obstacles in the environment. The potential field value increases as the distance between the slave and the obstacle decreases.

[0080] Improved The algorithm integrates the multi-aircraft collision avoidance potential field and the obstacle potential field into the above cost to form an improved cost function, namely ,in is the improved cost function, 、 are the weight coefficients of the multi-aircraft collision avoidance potential field and the obstacle potential field, balancing the priority of obstacle and multi-aircraft collision avoidance. is the multi-aircraft collision avoidance potential field function, is the obstacle potential field function.

[0081] Using the improved cost function above, run The algorithm searches for a path from the current position of the slave to its local tracking target point. Since the improved cost function penalizes both proximity to obstacles and proximity to other slaves, the searched path tends to stay away from all static / dynamic obstacles and maintain a safe distance from other collaboratively tracked slaves. While ensuring safety, the path length is the shortest.

[0082] The algorithm finally outputs a collision-free local path consisting of a series of path points, which will improve The path points generated by the algorithm are input into S52 for sub-machine B-spline trajectory optimization.

[0083] S6 performs abnormal behavior analysis based on tracking trajectories. When a violation or anomaly is detected, the linkage control mechanism is triggered, controlling the drone to adjust its path for close-range evidence collection and linking it with fixed cameras for multi-angle recording to generate an alarm message.

[0084] Specifically, based on the spatiotemporal characteristics of the target tracking trajectory, pre-set rules for abnormal behavior in multiple scenarios are established. When a trajectory characteristic first meets a certain abnormality rule, it is marked as a suspected abnormality and a trajectory review is initiated. If the abnormality conditions are still met after the load is removed, the abnormal behavior is confirmed and the trigger time, target ID, abnormality type, and trajectory segment are recorded.

[0085] After confirming abnormal behavior, the system automatically triggers a multi-device coordinated response. The specific steps are as follows: The mother machine sends instructions to the daughter machine responsible for the target, predicts the target's future position based on the current tracking trajectory, and uses the improved The algorithm plans the shortest evidence collection path and adjusts the flight altitude to ensure clear capture of details.

[0086] When the sub-machine arrives near the target, it activates the high-definition camera to shoot close-up images: in traffic scenes, it needs to capture the license plate and the moment of violation; in forestry scenes, it needs to capture the target's appearance and activity status.

[0087] Fixed camera linkage recording: The system calls multiple fixed cameras at different angles around the target's current location and starts recording simultaneously to cover the entire process of the target's violation.

[0088] The camera recording data and the drone evidence footage are aligned through time stamps to ensure temporal and spatial consistency.

[0089] Automatically generate structured alarm reports, push them to the management and control terminal through an encrypted protocol, trigger terminal sound and light alarms, and store them on the local server for backup.

[0090] It should be noted that the present invention is not limited to the above-mentioned embodiments. The above-mentioned embodiments are merely examples, and any embodiments having substantially the same structure and effect as the technical concept within the scope of the technical solution of the present invention are all included in the technical scope of the present invention. In addition, without departing from the scope of the present invention, other embodiments that can be conceived by those skilled in the art and that combine some of the constituent elements in the embodiments are also included in the scope of the present invention.

Claims

1. A multi-target detection and tracking method, characterized in that: The following steps are involved: S1, obtaining multi-source linkage video data, wherein the multi-source linkage video data includes an aerial video frame sequence collected by a drone and a ground video frame sequence collected by a fixed camera at a mid-high point, and performing scene adaptation preprocessing on the video frame sequence; S2: Build a multi-scale detection model that integrates scene context and outputs multi-target detection boxes and confidence scores. S3, extracts the multi-dimensional feature vector of the detected target, including the appearance features based on block enhancement and the geometric relationship features based on the relative positions of the targets, and builds a dynamic feature library; S4, the tracking mechanism based on multi-level adaptive association generates the initial trajectory; S5, trajectory optimization based on parent-child multi-machine collaboration; S6 performs abnormal behavior analysis based on tracking trajectories. When a violation or anomaly is detected, the linkage control mechanism is triggered, controlling the drone to adjust its path for close-range evidence collection and linking it with fixed cameras for multi-angle recording to generate an alarm message.

2. A multi-target detection and tracking method according to claim 1, characterized in that: The performing scene adaptation preprocessing on the video frame sequence specifically includes: Histogram block local equalization processing is used for UAV video frames to enhance small target details, and feature point matching based on the SuperPoint model is used for image registration of fixed camera video frames to correct camera offset.

3. A multi-target detection and tracking method according to claim 2, characterized in that: The above method uses histogram block local equalization processing on the drone video frame to enhance the details of small targets, specifically including: The drone video frames are divided into non-overlapping area blocks. The grayscale histogram is calculated for each area block. The cumulative distribution function is calculated based on the histogram, and the grayscale frequency limit value is set. When the number of pixels at a certain grayscale level exceeds the preset limit value, the number of pixels exceeding the limit is evenly distributed to other grayscale levels that do not reach the limit value. Local equalization is performed based on the cumulative distribution function, and bilinear interpolation is used to smooth the transition between area blocks to enhance the contour details of distant vehicles.

4. A multi-target detection and tracking method according to claim 2, characterized in that: The method of performing image registration on a fixed camera video frame by using feature point matching based on a SuperPoint model to correct camera offset specifically includes: Extract feature points of two consecutive frames of images through the SuperPoint model; The k-nearest neighbor algorithm is used to find multiple feature points with the closest distance in the reference frame for each feature point in the current frame, and the initial matching point pairs are screened by the nearest neighbor distance ratio threshold; The RANSAC algorithm is used to fit the perspective transformation model, and the incorrect matching point pairs are eliminated with the inlier threshold to obtain the optimized correct matching point pairs; Based on the correct matching point pairs, the homography matrix is ​​calculated, the lane line coordinates and the target detection frame in the current frame are perspective transformed, mapped to the reference frame coordinate system, and the camera offset is corrected.

5. The multi-target detection and tracking method according to claim 1, wherein: The multi-scale detection model uses the YOLO11 model as the benchmark model, including the backbone network, the neck network and the head network; The backbone network is based on Yolov7-tiny, with three additional multi-scale feature extraction sublayers. It uses convolution kernels of different sizes for objects of different sizes and fuses feature maps of different depths through skip connections. The neck network adopts an improved feature pyramid network and path aggregation network structure to integrate high-level semantics and low-level detail features, while adding scene context features. The head network adopts a two-branch structure. The classification branch outputs category confidence through an improved Resnet-18, and the regression branch outputs the detection box coordinates. The loss function is the weighted sum of the classification loss and the positioning loss.

6. A multi-target detection and tracking method according to claim 1, characterized in that: The S3 specifically includes: S31: Divide the target image into sub-regions based on morphological features. The Transformer's QKV attention mechanism is used to process the features of each sub-region. Self-attention enhances the texture and color details within a single region, while cross-attention integrates the structural associations of different regions. The fully connected layer outputs the appearance feature vector. S32, calculating the relative distance and angle between the target and surrounding targets, as well as the mean and standard deviation of the target center coordinates in the local area, and constructing a geometric relationship vector; S33, concatenate the appearance feature vector and the geometric relationship vector into a multidimensional feature vector, store it according to the target identity number, update the feature vector in each frame according to the detection result, archive the target features that have not been matched for more than a preset time, and retain the feature history of the last few frames for re-identification and association.

7. The multi-target detection and tracking method according to claim 1, wherein: The S4 specifically includes: S41, calculating the scene dynamic confidence threshold by gradient descent method and classifying the detection frame; S42, construct an improved Kalman filter, the state vector contains position, size, velocity and rotation angular velocity, and combines the affine transformation matrix of global motion estimation to achieve camera offset compensation; S43, for high-confidence detection boxes, double association is performed using intersection-over-union (IoU) and re-identification features, and for low-confidence detection boxes, secondary IoU association is performed, and the initial trajectory is generated through the Hungarian algorithm.

8. The multi-target detection and tracking method according to claim 1, wherein: The S5 specifically includes: The master machine generates the multi-target minimum enclosing circle and plans the global coverage trajectory through the Welzl algorithm, and the slave machine uses the B-spline curve to generate the local tracking trajectory. The algorithm implements dynamic obstacle avoidance and multi-machine collision detection, and outputs optimized trajectories.

9. A multi-target detection and tracking method according to claim 8, characterized in that: The master machine generates the multi-target minimum enclosing circle and plans the global coverage trajectory through the Welzl algorithm, and the slave machine generates the local tracking trajectory using the B-spline curve. The algorithm implements dynamic obstacle avoidance and multi-machine collision detection, and outputs optimized trajectories, including: S51: The master aircraft predicts the future position based on the historical trajectory of multiple targets and uses the Welzl algorithm to generate a minimum enclosing circle containing the future position. If the radius of the enclosing circle is larger than the sensing range of the master aircraft's lidar, outlier targets that are out of range are removed and a new minimum enclosing circle is generated, with the center of the new enclosing circle as the trajectory endpoint. S52, optimizing the B-spline curve control points of the local tracking trajectory of the slave, introducing an obstacle avoidance cost function, and calculating the relative distance between the control points and the nearest obstacle using the Euclidean signed distance field to ensure that the distance between the slave and the obstacle is not less than a first safety threshold, while constraining the relative distance between the slave and the target to be within a preset range; S53, based on the improved The algorithm generates a collision-free local path, introduces a multi-machine collision avoidance potential field into the path evaluation, outputs a path point in combination with the obstacle potential field, and incorporates the path point into the B-spline trajectory optimization of the sub-machine.

Images