A multi-target detection and tracking method

Abstract

The application discloses a multi-target detection and tracking method, and relates to the technical field of computer vision and intelligent monitoring. The method obtains multi-source video data of an unmanned aerial vehicle and a high point fixed camera, carries out scene adaptive preprocessing, and outputs a target detection frame by using a multi-scale detection model fusing scene context; a dynamic feature library is constructed by extracting a block enhanced appearance feature and a geometric relationship feature of the target, and an initial trajectory is generated based on a multi-level adaptive correlation mechanism; the trajectory is optimized by a sub-mother type multi-machine cooperation, and linkage control is triggered in combination with abnormal behavior analysis, so that the unmanned aerial vehicle is controlled to take evidence at a close distance and linkage fixed equipment recording is performed. The application improves the multi-source data fusion capability, multi-scale target detection precision and trajectory correlation robustness in a complex scene, and can efficiently support intelligent management and control requirements in the fields of traffic and forestry.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and intelligent monitoring technology, and particularly relates to a multi-target detection and tracking method. BACKGROUND

[0002] In the fields of smart city management, public security monitoring, large-scale event security, etc., it is of great significance to continuously and accurately detect and track multiple targets (pedestrians, vehicles, etc.) in a wide-area scene (such as a transportation hub, a city square, or a large-scale event site). Existing technologies usually rely on a single-view monitoring system, such as widely deployed fixed high-point cameras or a single unmanned aerial vehicle (UAV) cruise. However, these traditional methods have significant limitations when dealing with complex and dynamic large-scale scenes:

[0003] Cross-view trajectory breakage problem: fixed cameras have fixed fields of view and can be blocked, while a single UAV has a flexible field of view but limited coverage and can be constrained by endurance and flight path. When a target moves between the fields of view of multiple fixed cameras or enters and exits the field of view of a UAV, it is difficult to continuously and seamlessly associate the target's identity. For example, a target enters the field of view of a UAV after moving out of the field of view of a fixed camera A and then enters the field of view of a fixed camera B. Due to the dramatic change in viewing angle, the difference in resolution, and the lack of effective coordination between different sensors, existing methods often result in broken target trajectories or frequent ID switching, making it impossible to form a complete and coherent motion trajectory. This greatly limits the ability to analyze target behavior across regions and to perceive the overall situation.

[0004] Insufficient tracking robustness in complex scenes: In complex scenes with dense crowds, severe occlusions, similar target appearances, or rapid appearance changes (such as pedestrians turning around or vehicles turning), existing tracking algorithms mainly rely on a single appearance feature or a simple motion model. These features are easily disturbed by factors such as environmental light changes, local occlusions, and viewing angle differences, resulting in high target association error rates, unstable tracking trajectories, and even lost targets. Especially when a target is heavily occluded in a dense area for a long time or its appearance changes significantly, it is extremely difficult to recapture and correctly associate the target.

[0005] Therefore, there is an urgent need for a multi-target detection and tracking method that can effectively integrate multi-source heterogeneous video data (such as aerial UAV views and ground fixed high-point views), overcome cross-view association barriers, and maintain high robustness in complex scenes, to continuously, accurately, and seamlessly depict target motion trajectories in a wide area, laying a solid foundation for subsequent abnormal behavior identification and response. SUMMARY

[0006] In view of the above problems, the present application provides a multi-target detection and tracking method, which aims to realize accurate detection, stable tracking and rapid response to abnormal behavior of multi-target in complex scenes through multi-source data fusion, multi-scale detection, dynamic feature correlation and multi-machine collaborative optimization.

[0007] In a first aspect, the present application provides a multi-target detection and tracking method, comprising the following steps:

[0008] S1, acquiring multi-source linkage video data, wherein the multi-source linkage video data comprises an aerial video frame sequence collected by a UAV and a ground video frame sequence collected by a middle-high point fixed camera, and performing scene adaptive preprocessing on the video frame sequence;

[0009] S2, constructing a multi-scale detection model fused with scene context, and outputting multi-target detection boxes and confidence;

[0010] S3, extracting a multi-dimensional feature vector of the detected target, including appearance features based on block enhancement and geometric relationship features based on relative positions between targets, and constructing a dynamic feature library;

[0011] S4, generating an initial trajectory based on a multi-level adaptive correlation tracking mechanism;

[0012] S5, performing trajectory optimization based on a sub-mother multi-machine collaboration;

[0013] S6, performing abnormal behavior analysis based on the tracking trajectory, triggering a linkage control mechanism to control the UAV to adjust the path for close-range evidence collection and link the fixed camera for multi-angle recording when a violation or abnormality is detected, and generating an alarm information.

[0014] Further, the scene adaptive preprocessing on the video frame sequence specifically comprises:

[0015] The UAV video frame is subjected to histogram block local equalization processing to enhance small target details, and the fixed camera video frame is subjected to feature point matching based on a SuperPoint model for image registration to correct camera offset.

[0016] By adopting histogram block local equalization processing on the UAV video frame, the details of small targets (such as remote vehicles and low-altitude UAVs) can be enhanced, and the detection difficulty of small targets caused by long distance and blurred imaging in the UAV view can be solved; by adopting feature point matching based on the SuperPoint model for image registration on the fixed camera video frame, the camera offset caused by installation offset and environmental vibration can be effectively corrected, and the spatial consistency of the ground video frame can be ensured. The combination of the two provides high-quality preprocessing data for subsequent multi-target detection and cross-device tracking, and improves the availability of multi-source video data in complex scenes.

[0017] Further, the unmanned aerial vehicle video frame is subjected to histogram block local equalization processing to enhance small target details, specifically including:

[0018] The unmanned aerial vehicle video frame is divided into non-overlapping region blocks, the gray histogram of each region block is calculated, the cumulative distribution function is calculated according to the histogram, the gray level frequency limit value is set, when the pixel number of a certain gray level exceeds the preset limit value, the pixel number exceeding the limit is evenly distributed to other gray levels that do not reach the limit value; local equalization is carried out based on the cumulative distribution function, and the region blocks are smoothly transitioned by bilinear interpolation, enhancing the outline details of the remote vehicle.

[0019] The histogram block local equalization processing of the unmanned aerial vehicle video frame specifically solves the core problem of small targets (such as remote vehicles and low-altitude small targets) losing details due to long distance and blurred imaging from the aerial perspective of the unmanned aerial vehicle: by dividing the video frame into non-overlapping region blocks and separately performing gray histogram analysis and equalization, the global equalization is avoided, which is prone to overexposure in bright areas and covering up details in dark areas, and the texture and outline details of small targets in each local region can be accurately enhanced; by setting the gray level frequency limit value and redistributing the pixel number exceeding the threshold value, the "submersion" of the gray level with a high proportion in the image to the target details can be effectively inhibited, further highlighting the gray difference between the small target and the background; the bilinear interpolation smooth transition between region blocks eliminates the stitching marks that may be generated by the block processing, ensures the overall visual coherence of the image, and avoids the interference of local abruptness to the feature extraction of the subsequent detection model. Ultimately, the processing provides the unmanned aerial vehicle video data with high detail clarity and low noise interference for the subsequent multi-scale detection model, significantly improves the detection accuracy of small targets in complex aerial scenes, and lays a high-quality aerial perspective data foundation for multi-source data fusion tracking.

[0020] Further, the fixed camera video frame is subjected to feature point matching based on the SuperPoint model for image registration to correct camera offset, specifically including:

[0021] The feature points of the two consecutive frames of images are extracted by the SuperPoint model;

[0022] The k-nearest neighbor algorithm is used to find the nearest feature points in the reference frame for each feature point of the current frame, and the initial matching point pairs are screened by the nearest neighbor distance ratio threshold value;

[0023] The RANSAC algorithm is used to fit the perspective transformation model, and the inlier threshold value is used to remove the false matching point pairs to obtain the optimized correct matching point pairs;

[0024] Calculate a homography matrix based on the correct matching point pairs, and perform perspective transformation on the lane line coordinates and target detection box in the current frame to map them to the reference frame coordinate system to correct the camera offset.

[0025] The feature point matching image registration scheme based on the SuperPoint model used by the pair of fixed camera video frames efficiently solves the camera offset problem caused by installation offset and environmental vibration of the fixed camera, ensuring the spatial consistency of the ground video frames: the SuperPoint model can extract robust feature points with scale invariance and rotation invariance, which adapts to the feature matching needs of the fixed camera in a dynamic environment, and is more adaptable to changes in light and local occlusion than traditional feature extraction methods (such as SIFT and ORB); the k-nearest neighbor algorithm combined with the nearest neighbor distance ratio threshold screening mechanism can initially eliminate low-similarity false matching point pairs, reducing noise interference in subsequent processing; the RANSAC algorithm further improves the accuracy of feature point matching by robustly fitting a perspective transformation model and eliminating false matching point pairs, ensuring that the matching point pairs can truly reflect the spatial transformation relationship between the two frames of images; the homography matrix calculated based on the correct matching point pairs can accurately map the lane line coordinates and target detection box of the current frame to the reference frame coordinate system, completely offsetting the positional deviation caused by camera offset, so that the ground video frames at different times have a unified coordinate reference. This scheme effectively avoids the problems of cross-frame target position misalignment, trajectory breakage, or ID switching caused by camera offset, provides a stable ground perspective spatial reference for subsequent multi-target trajectory association, and significantly improves the continuity and robustness of multi-target tracking in a ground scene.

[0026] Further, the multi-scale detection model takes YOLO11 model as a benchmark model, including a backbone network, a neck network and a head network.

[0027] The backbone network is based on Yolov7-tiny, and three multi-scale feature extraction sublayers are added, which use convolution kernels of different sizes for targets of different sizes, and fuse feature maps of different depths through skip connection;

[0028] The neck network adopts a combined structure of an improved feature pyramid network and a path aggregation network, which fuses high-level semantic and low-level detail features, and adds scene context features.

[0029] The head network adopts a double-branch structure, the classification branch outputs class confidence through an improved Resnet-18, and the regression branch outputs detection box coordinates, and the loss function is the weighted sum of classification loss and positioning loss.

[0030] The backbone network enhances the perception ability of different size targets by adding multi-scale feature extraction sub-layers and using different size convolution kernels, solving the problem of insufficient adaptability of traditional models to multi-scale targets. The neck network fuses high-level semantic and low-level detail features and adds scene context features (such as lane lines and terrain edges), improving the model's understanding of complex scenes. The head network uses a double-branch structure and optimizes the loss function, increasing the small target positioning loss weight, further improving the detection box positioning accuracy and class confidence reliability. The overall model significantly improves the accuracy and robustness of multi-target detection in complex environments.

[0031] Further, the S3 specifically comprises:

[0032] S31, dividing the detection target image according to morphological features to obtain a divided sub-region, and using a QKV attention mechanism of a Transformer to process features of each sub-region, wherein self-attention strengthens texture and color details within a single region, and cross-attention fuses structural correlations of different regions, and an appearance feature vector is output through a fully connected layer;

[0033] S32, calculating a relative distance, an included angle, and a mean and a standard deviation of a target center coordinate of a target and surrounding targets in a local region to construct a geometric relationship vector;

[0034] S33, concatenating the appearance feature vector and the geometric relationship vector into a multi-dimensional feature vector, storing according to a target identity number, updating the feature vector according to a detection result of each frame, archiving a target feature that is not matched for more than a preset time length, and retaining a feature history of a plurality of recent frames for re-identification association.

[0035] The QKV attention mechanism of the Transformer extracts the appearance features of the divided blocks, which can capture fine-grained differences of targets, such as structural features of a drone body and arms, and local textures of a vehicle, solving the problem of identity confusion caused by similar appearances of similar targets. The geometric relationship features based on relative positions between targets are constructed, which enhances the scene adaptability of target association and reduces the limitations of single appearance feature association. The dynamic feature library provides data support for target re-identification association by real-time updating, archiving, and retaining historical features, ensuring the identity continuity of targets in temporary occlusion and cross-device tracking, and improving the robustness of multi-target association.

[0036] Further, the S4 specifically comprises:

[0037] S41, calculating a scene dynamic confidence threshold value by a gradient descent method, and grading detection boxes;

[0038] S42, an improved Kalman filter is constructed, the state vector includes position, size, velocity and angular velocity, and the affine transformation matrix of global motion estimation is combined to realize camera offset compensation;

[0039] S43, for high confidence detection boxes, a double association of intersection over union and re-identification features is adopted, and for low confidence detection boxes, a secondary intersection over union association is performed, and an initial trajectory is generated by the Hungarian algorithm.

[0040] The scene dynamic confidence threshold is calculated by the gradient descent method, the scene self-adaptation of detection box grading is realized, and the inadaptability of the fixed threshold to dynamic scenes is avoided; the improved Kalman filter introduces state vectors such as angular velocity and combines an affine transformation matrix to accurately compensate for camera offset (such as unmanned aerial vehicle jitter and camera vibration), and the accuracy of target motion state prediction is improved; for high / low confidence detection boxes, a differential association strategy (double association / secondary intersection over union association) is adopted, and the initial trajectory is generated by the Hungarian algorithm, which effectively reduces the error rate of target association, solves the problem of insufficient robustness of traditional single association strategy in complex scenes, and ensures the continuity of the initial trajectory.

[0041] Further, the S5 specifically comprises:

[0042] The mother machine generates a minimum enclosing circle of multiple targets by the Welzl algorithm and plans a global coverage trajectory, the child machine generates a local tracking trajectory by a B-spline curve, and the improved algorithm realizes dynamic obstacle avoidance and multi-machine collision detection, and outputs the optimized trajectory.

[0043] Further, the mother machine generates a minimum enclosing circle of multiple targets by the Welzl algorithm and plans a global coverage trajectory, the child machine generates a local tracking trajectory by a B-spline curve, and the improved algorithm realizes dynamic obstacle avoidance and multi-machine collision detection, and outputs the optimized trajectory, specifically comprising:

[0044] S51, the mother machine predicts future positions based on multiple target historical trajectories, calls the Welzl algorithm to generate a minimum enclosing circle containing the future positions, and if the radius of the enclosing circle is greater than the radius of the mother machine laser radar sensing range, removes the outlying point targets beyond the range and regenerates the minimum enclosing circle, and takes the center of the new enclosing circle as the trajectory endpoint;

[0045] S52, the B-spline curve control points of the local tracking trajectory of the child machine are optimized, and an obstacle avoidance cost function is introduced, the relative distance between the control points and the nearest obstacle is calculated by the Euclidean signed distance field, and the distance between the child machine and the obstacle is ensured to be not less than a first safety threshold, and the relative distance between the child machine and the target is constrained within a preset interval;

[0046] S53, based on the improved The algorithm generates a collision-free local path, introduces a multi-robot collision avoidance potential field in path evaluation, outputs a path point in combination with an obstacle potential field, and integrates the path point into a B-spline trajectory optimization of the sub-robot.

[0047] The mother robot generates a multi-target minimum enclosing circle through the Welzl algorithm and plans a global coverage trajectory, realizes global control of the multi-target distribution, and ensures the comprehensiveness of the tracking range; the sub-robot generates a local tracking trajectory through a B-spline curve, ensures the smoothness and continuity of the trajectory, and realizes the improved The algorithm realizes dynamic obstacle avoidance and multi-robot collision detection, solves the path conflict problem in multi-robot cooperative tracking, realizes the cooperation of global coverage and local accurate tracking through the combination of the three, and improves the efficiency and safety of multi-robot cooperative tracking.

[0048] In a second aspect, the present application further provides a computer terminal, comprising:

[0049] A memory storing an executable program;

[0050] A processor for running the program, wherein the program performs the multi-target detection and tracking method when running.

[0051] In a third aspect, the present application further provides a computer readable storage medium, comprising a stored executable program, wherein the executable program controls the device where the computer readable storage medium is located to perform the multi-target detection and tracking method when running.

[0052] In a fourth aspect, the present application further provides a computer program product comprising a computer program, which realizes the multi-target detection and tracking method when executed by a processor.

[0053] Compared with the prior art, the present application has the beneficial effects that: by fusing the multi-source linkage video data of unmanned aerial vehicles and high-point fixed cameras, the complementary of aerial and ground perspectives is realized, the limitation of single device perspective is broken through, and the comprehensive coverage capability of complex scenes is improved; by constructing a multi-scale detection model integrating scene context, in combination with multi-dimensional feature extraction and dynamic feature library construction, the accurate identification and differentiation capability of multi-scale targets is enhanced, and the problem of easy confusion of similar targets is solved; based on the tracking mechanism of multi-level adaptive association and the sub-mother multi-robot cooperative trajectory optimization, the coherence of target trajectory and the efficiency of tracking are ensured, and the interference caused by device deviation, target occlusion, etc. is reduced; finally, through the abnormal behavior analysis and linkage control mechanism, a closed loop from detection and tracking to intelligent response is realized, the intelligent management in the fields of traffic control, forestry monitoring, etc. is effectively supported, the rapid disposal capability of illegal or abnormal behaviors is significantly improved, and the present application has strong practicality and application value. BRIEF DESCRIPTION OF DRAWINGS

[0054] The technical solutions in the embodiments or the prior art will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings in the following description only show some embodiments of the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative effort based on the structures shown in the drawings are within the protection scope of the present application.

[0055] Figure 1 A flowchart of the method of the present application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described and explained below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort are within the protection scope of the present application.

[0057] The present application provides a multi-target detection and tracking method, which specifically comprises the following steps:

[0058] S1, acquiring multi-source linkage video data, wherein the multi-source linkage video data comprises an aerial video frame sequence collected by a UAV and a ground video frame sequence collected by a middle-high point fixed camera, and performing scene adaptation preprocessing on the video frame sequences.

[0059] Further, the UAV video frame is subjected to histogram block local equalization processing to enhance small target details, and the fixed camera video frame is subjected to feature point matching based on a SuperPoint model for image registration to correct camera offset.

[0060] Specifically, the preprocessing of the UAV video frame comprises: dividing the UAV video frame into non-overlapping region blocks, calculating a gray histogram for each region block, calculating a cumulative distribution function according to the histogram, setting a gray level frequency limit value, when the number of pixels (frequency) of a certain gray level exceeds the preset limit value, the number of pixels exceeding the limit is evenly distributed to other gray levels that do not reach the limit value, and performing local equalization based on the cumulative distribution function, and using bilinear interpolation smooth transition between region blocks to enhance the outline details of a long-distance vehicle.

[0061] The preprocessing of the fixed camera video frame includes: extracting feature points of two continuous frames by a SuperPoint model, and finding the nearest two feature points in another frame for each feature point of the current frame by using a k-nearest neighbor algorithm. By setting a distance threshold, such as a nearest neighbor distance ratio, the initial matching point pairs with high similarity are screened out, and the correspondence relationship between the feature points of the two frames is preliminarily established.

[0062] Due to camera vibration, occlusion or similar texture interference, there are false matching point pairs in the initial matching. The RANSAC algorithm fits a perspective transformation model by iteratively randomly selecting part of the matching points, and judges whether a point conforms to the model by an inlier threshold, that is, a point whose deviation after transformation is less than or equal to the inlier threshold is an inlier. Finally, the matching point pairs corresponding to the model with the most inliers are retained, effectively eliminating false matches and improving matching accuracy.

[0063] Based on the correct matching point pairs screened out, a homography matrix is calculated, which can describe the perspective transformation relationship between the two frames of images and contains parameters such as translation, rotation and scaling. Using this matrix, the lane line coordinates and vehicle detection box of the current frame are subjected to perspective transformation, and are mapped into the coordinate system of the reference frame (such as the previous frame), so as to offset the position deviation caused by camera vibration and ensure the spatial consistency of the target position in different time video frames, thereby providing a stable coordinate reference for subsequent multi-target detection and tracking.

[0064] S2, a multi-scale detection model integrating scene context is constructed, and multi-target detection boxes and confidence are output.

[0065] Specifically, the multi-scale detection model takes YOLO11 model as a benchmark model, and includes a backbone network, a neck network and a head network.

[0066] The backbone network is based on Yolov7-tiny, and three multi-scale feature extraction sublayers are added after the original convolutional layer. Different sizes of convolutional kernels are used for different size targets, and features of different depths are fused through a skip connection to enhance the perception ability of multi-scale targets in complex scenes.

[0067] The neck network adopts a combined structure of an improved feature pyramid network FPN and a path aggregation network PAN. The FPN layer up-samples high-level semantic features and fuses them with middle-level features, and the PAN layer down-samples middle-level features and fuses them with low-level detailed features. Meanwhile, scene context features such as lane line features in a traffic scene and terrain edge features in a forestry scene are added in the fusion process to improve the detection accuracy of scene-related targets.

[0068] The head network adopts a double-branch structure. The classification branch outputs class confidence through an improved Resnet-18 network, and the regression branch outputs detection box coordinates. The loss function is the weighted sum of the classification loss and the positioning loss.

[0069] S3, extract the multi-dimensional feature vector of the detection target, including the appearance feature based on the block enhancement and the geometric relationship feature based on the relative position between targets, and construct a dynamic feature library.

[0070] Specifically, the above S3 includes the following contents:

[0071] S31, divide the detection target image according to the morphological features to obtain the divided sub-regions, and use the QKV attention mechanism of the Transformer to process the features of each sub-region, wherein the self-attention strengthens the texture and color details in a single region, and the cross-attention fuses the structural correlation of different regions, and the appearance feature vector is output through a fully connected layer.

[0072] Specifically, when the detection target is a vehicle, it is divided into 3 sub-regions according to the vehicle head, body and position, and for a single sub-region, the pixel correlation is calculated through a self-attention matrix, and the formula is as follows:

[0073] ;

[0074] Q is the query matrix;

[0075] K is the key matrix, is the transpose matrix of the key matrix K;

[0076] V is the value matrix;

[0077] is the dimension of each feature vector in the key matrix K.

[0078] After the Transformer encoding, the appearance feature vector is output through 2 fully connected layers.

[0079] S32, calculate the relative distance, angle and mean and standard deviation of the center coordinates of the target and the surrounding targets in the local region to construct a geometric relationship vector.

[0080] Specifically, the local region is defined with the target as the center, and the spatial relationship features of other targets in the region are calculated, and the specific steps are as follows:

[0081] The center coordinates of the target t are set as , and the surrounding targets i satisfying are selected, wherein is the center coordinates of the target i, and R is a preset distance threshold parameter.

[0082] The Euclidean distance between the target t and the target i is calculated as: .

[0083] The angle of the target i with the target t as the origin and the horizontal right as the positive direction of the x axis is:

[0084] , is the four-quadrant arctangent function.

[0085] Calculate the mean and standard deviation of all target center coordinates in the local region:

[0086] ;

[0087] ;

[0088] where m is the number of peripheral targets in the local region;

[0089] is the mean of all target center x coordinates in the local region;

[0090] is the mean of all target center y coordinates in the local region;

[0091] is the center x coordinate of the kth target;

[0092] is the center y coordinate of the kth target;

[0093] is the standard deviation of all target center x coordinates in the local region, measuring the dispersion of coordinates in the x direction;

[0094] is the standard deviation of all target center y coordinates in the local region, measuring the dispersion of coordinates in the y direction.

[0095] Take the first 3 nearest neighbors of the m peripheral targets and , combined with the calculated mean and standard deviation, construct the geometric relationship vector.

[0096] S33, concatenate the appearance feature vector and the geometric relationship vector into a multi-dimensional feature vector, store according to the target identity number, update the feature vector according to the detection result every frame, archive the target features that have not been matched for more than a preset time length, and keep the feature history of the recent frames for re-identification association.

[0097] Specifically, the appearance feature vector and the geometric relationship vector are combined into a multi-dimensional feature vector while retaining visual recognition and spatial context information. Each target is assigned a unique ID, and its multi-dimensional feature vector is stored in a dynamic feature library. If the target is detected in the current frame, the new feature vector replaces the old data, reflecting the latest state of the target. If the target is not detected, such as temporary occlusion, the last feature is retained and waits for subsequent matching. A preset system tolerates the maximum time window for target disappearance. If the target is not re-matched within the set time, its feature is removed from the dynamic feature library and stored in the archive library, reducing the real-time retrieval burden. For each target, the latest multi-frame historical feature vector is retained. When the target reappears, the system compares the current feature with the historical feature library. If the matching is successful, the original ID is restored, ensuring the continuity of the trajectory. If the matching fails, a new ID is assigned, and it is considered a new target.

[0098] S4, generating an initial trajectory based on a multi-level adaptive association tracking mechanism, specifically comprising the following steps:

[0099] S41, calculating the scene dynamic confidence threshold by gradient descent method, and classifying the detection frame.

[0100] Specifically, detection frame samples are extracted from the current frame and historical frames, and real targets (positive samples) and false positives / backgrounds (negative samples) are labeled. The following binary cross-entropy loss function is constructed to quantify the deviation between confidence and true label:

[0101] ;

[0102] The loss function value is:

[0103] wherein, is the original confidence of the ith detection frame;

[0104] θ is the confidence threshold to be optimized;

[0105] N is the total number of samples;

[0106] is the true label of the ith sample;

[0107] σ is the Sigmoid activation function.

[0108] The above loss function is iteratively optimized by gradient descent algorithm to solve the confidence threshold that minimizes the classification error. The specific formula is as follows:

[0109] .

[0110] wherein, is the confidence threshold of the tth iteration;

[0111] confidence threshold for the t+1th iteration

[0112] learning rate

[0113] gradient of the loss function

[0114] The threshold is dynamically adjusted according to the scene, such as increasing the threshold in a congestion scene to reduce false positives, and reducing the threshold in a sparse scene to avoid missing detection.

[0115] According to the optimization result, a high confidence threshold and a low confidence threshold are set, and the detection frame is divided into a high confidence detection frame and a low confidence detection frame.

[0116] S42, an improved Kalman filter is constructed, the state vector includes position, size, speed and angular velocity, and the affine transformation matrix of global motion estimation is combined to realize camera offset compensation.

[0117] Specifically, the state vector dimension of the traditional Kalman filter is expanded to include the position, size, motion speed and angular velocity of the target, and the motion and morphological changes of the target are comprehensively described.

[0118] Based on the assumption of uniform motion, a state transition matrix is constructed, and the time interval between adjacent frames is used to update the state vector through the Kalman filter to realize preliminary prediction of the position, size, speed and angular velocity of the target in the next frame.

[0119] The ORB (Oriented FAST and Rotated BRIEF) feature matching algorithm is used to extract feature points of two consecutive frames of images and establish a corresponding relationship. The essence of the ORB matching process is to calculate the displacement of the feature point pair, that is, to provide sparse optical flow information for estimating the global motion. Combined with the RANSAC (Random Sample Consensus) algorithm, an affine transformation matrix describing the camera motion is robustly estimated.

[0120] The inverse matrix of the above affine transformation matrix is used to perform coordinate transformation on the target position coordinates predicted by the Kalman filter. This step aims to eliminate the target position prediction error caused by camera jitter or offset, and significantly improve the prediction accuracy. The predicted target size and angular velocity directly use the Kalman prediction result, and output the corrected target state prediction result.

[0121] S43, the high confidence detection frame is associated with the intersection over union and the re-identification feature, and the low confidence detection frame is associated with the secondary intersection over union, and the initial trajectory is generated by the Hungarian algorithm.

[0122] Specifically, the high confidence detection frame association uses the intersection over union IOU and the re-identification feature for double measurement:

[0123] IOU metric: calculate the overlap between the current detection box and the predicted box of the historical trajectory (obtained from the corrected target state prediction output by S42), reflecting the spatial position consistency;

[0124] Re-identification feature matching: extract the historical appearance feature vector of the target trajectory from the dynamic feature library, calculate the cosine similarity between the appearance feature vector of the current detection box and the extracted historical appearance feature vector, reflecting the target identity consistency;

[0125] Fusion score: weighted fusion of the above two indicators to form a comprehensive correlation score as the basis for correlation reliability.

[0126] Low confidence detection box association: for low confidence detection boxes and high confidence detection boxes that are not matched after association, perform secondary IOU association to obtain secondary overlap, and set an IOU threshold to only keep the overlap matching pairs above the IOU threshold, avoiding false associations caused by feature noise.

[0127] Trajectory generation: construct an association cost matrix, where each element C(i, j) represents the cost of associating the ith detection box with the jth trajectory. For a high-confidence detection box i that is successfully matched with a trajectory j, the association cost is (1-comprehensive correlation score of detection box i and trajectory j). For a detection box i that is successfully matched with a trajectory j through secondary IOU matching, the association cost is (1-secondary overlap of detection box i and trajectory j). For a detection box i and a trajectory j that are not successfully matched through the above two matches or are physically impossible to associate, the association cost is set to infinity.

[0128] Use the Hungarian algorithm to solve the optimal matching of the association cost matrix through matrix transformation to achieve one-to-one correspondence of cross-frame targets.

[0129] Matched detection box and trajectory: update the state of this trajectory, including position, velocity, angular velocity, etc.

[0130] Unmatched high-confidence detection box: initialize as a new trajectory, assign a new ID to this detection box; initialize a new trajectory with the position, size, etc. of this detection box; extract the appearance feature of this detection box and calculate its geometric relationship feature, store it in the dynamic feature library associated with this new ID; initialize the motion state of this trajectory.

[0131] Unmatched low-confidence detection box: do not generate a trajectory, wait for verification in subsequent frames.

[0132] S5, trajectory optimization based on sub-multiple multi-machine cooperation, specifically including the following steps:

[0133] S51, the mother machine predicts the future position based on the multi-target historical trajectory, calls the Welzl algorithm to generate a minimum enclosing circle containing the future position, and if the radius of the enclosing circle is greater than the radius of the mother machine laser radar perception range, the outlier target beyond the range is removed and the minimum enclosing circle is regenerated, and the center of the new enclosing circle is taken as the trajectory endpoint.

[0134] S52, the optimization sub-machine locally tracks the B-spline curve control point of the trajectory, introduces an obstacle avoidance cost function, calculates the relative distance between the control point and the nearest obstacle through the Euclidean signed distance field, ensures that the distance between the sub-machine and the obstacle is not less than the first safety threshold, and at the same time, the relative distance between the sub-machine and the target is constrained in a preset interval, generates a smooth and obstacle-avoiding local tracking trajectory for each sub-machine, and ensures safe approach to the target.

[0135] Specifically, based on the current position of the sub-machine, the current position of the target and the possible intermediate point, a set of control points is initially set. These initial control points define a rough B-spline curve, that is, wherein is the coordinate of the i-th control point, is the p-order B-spline basis function, is the coordinate of the point on the curve, u is the curve parameter, and n is the number of control points minus 1.

[0136] The Euclidean signed distance field (ESDF) is constructed, the known environmental obstacle information is input into the ESDF algorithm, and the value of each grid cell in the entire planning space is calculated through the ESDF algorithm. The value represents the distance from the control point to the surface of the nearest obstacle. When the value is less than the first safety threshold, a quadratic penalty cost is applied, and the smaller the distance, the greater the cost, and the greater the punishment. This strongly drives the control point away from the obstacle, ensuring that the points on the final trajectory (obtained by interpolation of the control points) also satisfy the safety distance. The cost function is represented by the following formula:

[0137] ;

[0138] wherein, is the obstacle avoidance cost value of the control point, and the greater the value, the more dangerous it is;

[0139] w is the obstacle avoidance cost weight coefficient;

[0140] max is the maximum value function;

[0141] is the first safety threshold;

[0142] the distance from the control point to the nearest obstacle.

[0143] A set of optimal control points are found using gradient descent method to minimize the cost function while satisfying the relative distance between the sub-machines and the target within the preset interval.

[0144] S53, based on the improved algorithm generates a collision-free local path, introduces a multi-machine collision avoidance potential field in path evaluation, combines obstacle potential field to output path points, and integrates the path points into sub-machine B-spline trajectory optimization.

[0145] A classic heuristic search algorithm is used to find the shortest path between two points in a grid map or point cloud space. It decides the search direction by evaluating the cost of each candidate node f(m)=g(m)+h(m), where g(m) is the actual cost of node m and h(m) is the estimated cost of node m to the target point.

[0146] Multi-machine collision avoidance potential field: when the relative distance between any two sub-machines is less than the preset second safety threshold, a repulsive potential field is generated, which increases exponentially with the decrease of distance, forcing the sub-machines to maintain a safe distance when tracking cooperatively and avoiding collision.

[0147] Obstacle potential field: a repulsive potential field is generated based on static / dynamic obstacles in the environment, which increases with the decrease of distance between the sub-machine and the obstacle.

[0148] Improved The multi-machine collision avoidance potential field and the obstacle potential field are integrated into the above cost function to form an improved cost function, i.e. where is the improved cost function, , are the weight coefficients of the multi-machine collision avoidance potential field and the obstacle potential field, respectively, balancing the priority of obstacle avoidance and multi-machine collision avoidance, is the multi-machine collision avoidance potential field function, is the obstacle potential field function.

[0149] Run the algorithm using the improved cost function above, which searches for a path from the current position of the sub-machine to its local tracking target point. Since the improved cost function penalizes both approaching obstacles and approaching other sub-machines, the searched path tends to avoid all static / dynamic obstacles and maintain a safe distance from other cooperatively tracking sub-machines, while being the shortest under the premise of safety.

[0150] The algorithm finally outputs a collision-free local path composed of a series of path points, which are input into S52 to optimize the sub-machine B-spline trajectory. The path points generated by the improved algorithm are input into S52 for sub-machine B-spline trajectory optimization.

[0151] S6, based on the tracking trajectory, abnormal behavior analysis is carried out, when the violation or abnormality is detected, the linkage control mechanism is triggered, the unmanned aerial vehicle adjusts the path to take close-up evidence and links the fixed camera to record multiple angles, and alarm information is generated.

[0152] Specifically, based on the spatiotemporal characteristics of the target tracking trajectory, a plurality of scene abnormal behavior determination rules are preset. When the trajectory characteristics first meet a certain type of abnormal rule, it is marked as suspected abnormality, and trajectory review is started. If the abnormal condition is still met after the load, the abnormal behavior is confirmed, and the trigger time, target ID, abnormal type and trajectory segment are recorded.

[0153] After confirming the abnormal behavior, the system automatically triggers a multi-device cooperative response, and the specific steps are as follows:

[0154] The mother machine sends instructions to the child machine responsible for the target, predicts the future position of the target based on the current tracking trajectory, plans the shortest evidence path using an improved algorithm, and adjusts the flight height to ensure clear shooting details.

[0155] After the child machine arrives near the target, the high-definition camera is enabled to shoot close-up pictures: traffic scenes need to capture license plates and violation moments; forestry scenes need to shoot target appearance and activity status.

[0156] Fixed camera linkage recording: the system calls multiple fixed cameras at different angles around the target's current location, and starts recording synchronously, covering the complete process of the target's violation behavior.

[0157] The camera recording data and the unmanned aerial vehicle evidence pictures are aligned by timestamp to ensure spatiotemporal consistency.

[0158] A structured alarm report is automatically generated, the report is pushed to the control terminal through an encryption protocol, and the terminal sound and light alarm is triggered, and the report is stored in the local server backup.

[0159] It should be noted that the present application is not limited to the above-mentioned embodiments. The above-mentioned embodiments are only examples, and embodiments having the same technical idea and playing the same role within the scope of the technical solution of the present application are included in the technical scope of the present application. In addition, within the scope of the main idea of the present application, various modifications of the embodiments that can be thought of by those skilled in the art, and other ways constructed by combining part of the components in the embodiments are also included in the scope of the present application.

Claims

1. A multi-target detection and tracking method, characterized in that, The method comprises the following steps: S1, acquiring multi-source linkage video data, wherein the multi-source linkage video data comprises an aerial video frame sequence collected by a UAV and a ground video frame sequence collected by a fixed camera at a medium-high point, and performing scene adaptation preprocessing on the video frame sequences; S2, constructing a multi-scale detection model fused with scene context, and outputting multi-target detection boxes and confidence; S3, extracting a multi-dimensional feature vector of a detected target, comprising appearance features based on block enhancement and geometric relationship features based on relative positions between targets, and constructing a dynamic feature library; specifically comprising: S31, dividing a detected target image according to morphological features to obtain divided sub-regions, and processing features of each sub-region by using a QKV attention mechanism of a Transformer, wherein self-attention strengthens texture and color details in a single region, and cross-attention fuses structural correlations of different regions, and appearance feature vectors are output through a fully connected layer; S32, calculating relative distances, included angles, and mean values and standard deviations of target center coordinates of a target and surrounding targets in a local region to construct geometric relationship vectors; S33, concatenating the appearance feature vectors and the geometric relationship vectors into a multi-dimensional feature vector, storing the multi-dimensional feature vector according to a target identity number, updating the feature vector according to a detection result every frame, archiving target features that are not matched for more than a preset time length, and retaining feature histories of recent frames for re-identification and correlation; S4, generating an initial trajectory based on a multi-level adaptive correlation tracking mechanism; S5, performing trajectory optimization based on a child-parent type multi-machine cooperation; specifically comprising: The mother machine generates multi-objective minimum enclosing circle by Welzl algorithm and plans global coverage trajectory, the sub-machine generates local tracking trajectory by B-spline curve, and the improved algorithm is used to realize dynamic obstacle avoidance and multi-machine collision detection, and output the optimized trajectory; S6, performing abnormal behavior analysis based on a tracking trajectory, triggering a linkage control mechanism when a violation or anomaly is detected, controlling a UAV to adjust a path for close-range evidence collection and linkage of fixed camera multi-angle recording, and generating an alarm information.

2. The multi-target detection and tracking method of claim 1, wherein, The scene adaptation preprocessing on the video frame sequences specifically comprises: performing histogram block local equalization processing on the UAV video frames to enhance small target details, and performing feature point matching based on a SuperPoint model on the fixed camera video frames to correct camera deviation.

3. The multi-target detection and tracking method of claim 2, wherein, The histogram block local equalization processing on the UAV video frames to enhance small target details specifically comprises: dividing the UAV video frames into non-overlapping region blocks, calculating a gray histogram for each region block, calculating a cumulative distribution function according to the histogram, setting a gray level frequency limit value, when the number of pixels of a certain gray level exceeds the preset limit value, evenly distributing the number of pixels exceeding the limit to other gray levels that do not reach the limit value, performing local equalization based on the cumulative distribution function, and using bilinear interpolation to smoothly transition between region blocks to enhance the outline details of distant vehicles.

4. The multi-target detection and tracking method of claim 2, wherein, The feature point matching based on the SuperPoint model on the fixed camera video frames to correct camera deviation specifically comprises: extracting feature points of two consecutive images by using the SuperPoint model; finding multiple feature points closest to each feature point of a current frame in a reference frame by using a k-nearest neighbor algorithm, and screening initial matching point pairs through a nearest neighbor distance ratio threshold. A perspective transformation model is fitted by using a RANSAC algorithm, and the incorrect matching point pairs are removed by using an inlier threshold, so as to obtain the correct matching point pairs after optimization; A homography matrix is calculated based on the correct matching point pairs, and the lane line coordinates and the target detection frame in the current frame are subjected to perspective transformation and mapped to a reference frame coordinate system to correct the camera offset.

5. The multi-target detection and tracking method of claim 1, wherein, The multi-scale detection model takes a YOLOv1 model as a benchmark model, and includes a backbone network, a neck network and a head network. The backbone network takes a Yolov7-tiny as a basis, and additionally adds three multi-scale feature extraction sub-layers, uses different sizes of convolution kernels for different sizes of targets, and fuses feature maps of different depths through a jump connection. The neck network adopts a combined structure of an improved feature pyramid network and a path aggregation network, fuses high-level semantic features and low-level detail features, and simultaneously adds scene context features. The head network adopts a double-branch structure, a classification branch outputs a category confidence by using an improved Resnet-18, and a regression branch outputs a detection frame coordinate, and a loss function is a weighted sum of a classification loss and a positioning loss.

6. The multi-target detection and tracking method of claim 1, wherein, The S4 specifically includes: S41, a scene dynamic confidence threshold is calculated by using a gradient descent method, and the detection frame is classified; S42, an improved Kalman filter is constructed, a state vector includes a position, a size, a speed and an angular velocity, and a camera offset compensation is realized by combining an affine transformation matrix of global motion estimation; S43, a high-confidence detection frame is associated by using an intersection over union and a re-identification feature, a low-confidence detection frame is associated by using a secondary intersection over union, and an initial trajectory is generated by using a Hungarian algorithm.

7. The multi-target detection and tracking method of claim 1, wherein, The parent machine generates a multi-target minimum enclosing circle by a Welzl algorithm and plans a global covering trajectory, the child machine generates a local tracking trajectory by a B-spline curve, and an improved algorithm is used to realize dynamic obstacle avoidance and multi-machine collision detection, and output an optimized trajectory, which specifically includes: S51, a mother machine predicts a future position based on multi-target historical trajectories, calls a Welzl algorithm to generate a minimum enclosing circle containing the future position, and if a radius of the enclosing circle is greater than a radius of a perception range of a laser radar of the mother machine, removes outlier targets beyond the range and regenerates a minimum enclosing circle, so as to take a center of the new enclosing circle as a trajectory end point; S52, a B-spline curve control point of a local tracking trajectory of a sub-machine is optimized, a cost function for avoiding obstacles is introduced, a relative distance between the control point and a nearest obstacle is calculated by using a Euclidean signed distance field, so as to ensure that a distance between the sub-machine and the obstacle is not less than a first safety threshold, and a relative distance between the sub-machine and the target is constrained in a preset interval. S53, based on improvement The algorithm generates a collision-free local path, introduces a multi-robot collision avoidance potential field in path evaluation, outputs path points combined with obstacle potential field, and integrates the path points into sub-machine B-spline trajectory optimization.

Images