Multi-target tracking method in complex scene based on adaptive association

Abstract

The invention provides a multi-target tracking method in a complex scene based on adaptive association, which comprises the following steps of: 1, acquiring a video frame sequence containing multi-target motion, and preprocessing the video frame sequence; 2, constructing a new target detection network model; the new target detection network model takes a YOLO11 model as a reference model and comprises a backbone network, a neck network and a head network; 3, performing overall multi-scale feature training and learning on the new target detection network model; 4, performing performance evaluation and evaluation on the target detection network model; and 5, inputting the detection frame object into a tracking algorithm, and predicting a target trajectory. According to the invention, an adaptive threshold value and a camera offset compensation mechanism are introduced, so that the anti-interference capability of tracking in different environments is enhanced. The model provided by the invention is used for an ablation experiment, and compared with a common deep learning model, the model has excellent detection and tracking performance under the same test condition.

Description

Technical Field

[0001] The present invention relates to the technical field of video analysis, and in particular to a multi-target tracking method in complex scenes based on adaptive association. Background Art

[0002] Multi-object tracking (MOT) is an important research hotspot in the field of computer vision. Its purpose is to detect and estimate the spatiotemporal motion trajectories of multiple targets in an image video stream. With the rapid development of computer vision technology, multi-object tracking has been widely used in video surveillance, autonomous driving, intelligent transportation and other fields.

[0003] At present, multi-target tracking based on detection (tracking-by-detection) has become an effective paradigm for MOT tasks. Multi-target tracking based on detection can be seen as an extension of the target detection problem, or can be described as a matching problem of the target position boxes of the previous and next frames. With the rapid development of target detection technology, more and more methods have begun to use more powerful detectors to obtain higher tracking performance. For example, SORT first uses FRCNN to detect objects, and uses the Kalman filter algorithm and the Hungarian algorithm to predict the current frame position and update the position, which helps to reduce the missed detection rate of occluded targets, and the ID switching effect is very good when the object moves less. DeepSORT uses the same detector and adds cascade matching and new trajectory confirmation on the basis of SORT. These methods have promoted the development of multi-target tracking.

[0004] Although the above methods have achieved good results, the following problems still exist: (1) Most detection methods select parallel-view tracking targets, ignoring some small targets at a distance or other complex scenes, and are not suitable for other scene changes, such as drone overhead shooting angles and bad weather interference. (2) In addition, during the tracking process, the data association stage is affected by many factors, such as the size and aggregation of the target, and changes in the camera perspective. Most methods classify high-scoring and low-scoring bounding boxes through a single predefined threshold, and then perform data association. This threshold can be determined by repeated experiments to obtain the threshold with the highest tracking accuracy, achieving good tracking results. However, in practical applications, the scene is usually not understood, making it difficult to select a suitable confidence threshold. Therefore, setting a fixed confidence threshold will limit the tracking performance. (3) In terms of camera perspective, most methods consider the motion model of the target pedestrian or car in an ideal situation, that is, the camera has a fixed perspective. The tracked target can be approximated as linear movement. By inputting the motion model into the Kalman filter, the trajectory of the object's movement is estimated, and good tracking results can be achieved. However, whether it is handheld or drone shooting, it is difficult to avoid camera movement, or even violent shaking, which will undoubtedly affect the tracking effect. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for multi-target tracking in complex scenes based on adaptive association in view of the shortcomings of the prior art. The method of the present invention aims to effectively extract scene context information and individual features on the basis of focusing on key information, and input the detection frame into the improved data association method for trajectory prediction, so as to achieve high-precision multi-target tracking.

[0006] The method of the present invention comprises the following steps: Step 1, obtaining a video frame sequence containing multiple target motions and preprocessing the video frame sequence; Step 2, constructing a new target detection network model to further process the multi-target images of continuous frames; the new target detection network model uses the YOLO11 model as a benchmark model, including a backbone network, a neck network and a head network; The backbone network is used to extract the contextual relationship information between the target object and the scene, and obtains fusion features with enhanced representation capabilities by using spatial and channel information; four context information extraction modules CM (Context Module) are used to replace the four C3k2 modules in the backbone network of the original YOLO11 model; The neck network introduces a hybrid channel attention module ML to integrate features of different scales and extract local and global target features based on hybrid attention; The head network adopts the LIOU loss function to ensure the diagonal ratio while controlling the intersection-over-union ratio, and finally outputs the confidence and bounding box position; Step 3: Perform overall multi-scale feature training and learning on the new target detection network model, using the loss function Accelerate model convergence; Step 4: Perform performance evaluation and assessment on the target detection network model obtained after training in step 3 to obtain the final target detection network model. Use the target detection network model to predict the labels of multi-target detection, namely, the detection box and confidence score (x, y, w, h, score), where x, y represent the horizontal and vertical coordinates of the target, w represents the width of the detection box, h represents the height of the detection box, and score represents the confidence score of the detection box (the value is 0~1); In step 5, the detection box object is input into the tracking algorithm, and the target trajectory is predicted by constructing more than two adaptive matches, dynamically adjusting the matching threshold according to the confidence level, and performing camera offset compensation at the same time.

[0007] Step 1 includes: using the VisDrone2019-MOT and VisDrone2019-Det drone datasets, which were open-sourced by the Tianjin University team in 2019, to collect small target motions in multiple scenes from the drone's perspective, and provide 10 types of targets. This dataset was collected using different models of drones in different scenes, lighting and weather conditions, and the image quality was ensured to be suitable for the input of the deep learning model. According to the intensity of motion in the scene, the frame interval is set, the original video is dynamically sparsely sampled, and the sampling ratio is dynamically set according to the number of targets and the scene to obtain a video frame sequence; the detected objects are annotated, and the video frames are sorted in chronological order to obtain multi-target images containing more than two consecutive frames, and histogram block local equalization preprocessing is performed to obtain a small target image dataset, and the small target image dataset is divided into a training set, a validation set, and a test set. The test set is used to finally evaluate the performance of the model. Considering that the target motion speed is fast from the drone's perspective, the frame interval is set to 1 to avoid losing important information. When computing resources are limited, the frame interval can be appropriately increased to 5-10 frames. Considering the large number of targets and high scene complexity, the sampling ratio is set to 30%-40% according to the number of targets and scene dynamics.

[0008] In step 1, the histogram block local equalization preprocessing specifically includes the following steps: Step 1-1, divide the image into non-overlapping area blocks tiles (usually 16×16 non-overlapping area blocks tiles); Step 1-2, calculate the histogram for each tile, and calculate the cumulative distribution function CDF based on the histogram; Step 1-3, set the grayscale limit frequency (usually 2). If the grayscale frequency of a tile's histogram exceeds the limit frequency, reduce the frequency and redistribute the reduced frequency to other grayscales, keeping the histogram sum unchanged. Step 1-4, use the cumulative distribution function CDF to perform local equalization and map the value of each pixel to a new value; In steps 1-5, bilinear interpolation is used to smooth the area between tiles.

[0009] In step 2, the backbone network is constructed by the following steps: Step 2-1, construct a convolution layer (64, 3, 2) for the input image, where 64 represents the number of channels (filters) output by the convolution layer, 3 represents the size of the convolution kernel, that is, a 3x3 convolution kernel, which is used to process the local area of ​​the input image, and 2 represents the stride, that is, the stride of the convolution kernel sliding on the input image is 2; Then construct 4 consecutive convolution downsampling, where the downsampled convolution kernel size is 3×3 and the stride is 2; Then build a context information extraction module to reduce the resolution and extract high-level features; Step 2-2, build a multi-scale feature utilization structure, connect the second context information extraction module CM to the neck network, and perform a concat operation with the upsample module of the neck network (in deep learning, concat refers to the operation of splicing multiple tensors along a specific dimension); Step 2-3, the backbone network is connected to a hybrid spatial channel attention module ML, and the hybrid spatial channel attention module ML is connected to the neck network; In step 2-2, the context information extraction module CM includes a large-kernel depthwise convolution and a pointwise convolution; The number of groups of the large kernel depthwise convolution is the same as the number of channels; The large kernel depthwise convolution first connects the ReLU activation function and the normalization operation to extract the global information of each channel; then a residual connection is performed, and then the pointwise convolution is repeated twice, and then the activation function ReLU and the normalization operation are connected; The two pointwise convolutions adopt an inverted bottleneck design, that is, the hidden dimension between the two pointwise convolutions is four times the input dimension, thereby achieving a mixture of spatial and channel information; this structure can improve the model's ability to detect small targets, especially in complex backgrounds; In step 2-3, the mixed spatial channel attention module ML includes a local average pooling module LAP, a global average pooling module GAP, an anti-average pooling module UNGAP and a one-dimensional convolution module Conv1d; The hybrid spatial channel attention module ML first transforms the input feature vector into a vector of 1*C*ks*ks, and then extracts local spatial information through local average pooling LAP, where C represents the channel dimension and ks (kernel size) represents the convolution kernel size; two branches are used to convert the extracted local spatial information into one-dimensional vectors, wherein the first branch uses the global average pooling module GAP operation to contain global information, and the second branch directly performs a reshape operation to contain local spatial information; the first branch passes through the one-dimensional convolution module Conv1d, and the second branch passes through the reshape, and then the original resolution of the two branches is restored through the unaverage pooling module UNGAP, and then information fusion is performed, and finally the feature vector of the original input is residually connected, so as to output a feature vector of hybrid global, local space and channel attention; wherein the convolution kernel size k of the one-dimensional convolution module Conv1d is proportional to the channel dimension C, which means that when capturing local cross-channel interaction information, only the relationship between each channel and k adjacent channels is considered; The convolution kernel size k is determined by the following formula: , where b and These are all hyperparameters, with a default value of 2, and if k is an even number, it is increased by 1; is a proportional function.

[0010] In step 3, the overall multi-scale feature training and learning of the new target detection network model includes: Step 3-1: Use the COCO image recognition dataset for pre-training of the new target detection network model, and use the VisDrone2019-Det drone-view image recognition dataset for overall training; Step 3-2, set the training round epochs, batch size batch, input image size, learning rate and intersection over union (IoU), and use stochastic gradient descent SGD as the optimizer. In the present invention, the training round epochs=100, batch size batch=4, input image size is (640, 640), learning rate is 0.01, and IoU is 0.7.

[0011] In step 3, most commonly used loss functions (SIoU, DIoU, CIoU) measure the overlap between the predicted box and the true box by IoU, Euclidean distance, aspect ratio and angle. Since the aspect ratio is defined as a relative value, errors may occur, especially for small targets at a long distance of the drone in this embodiment. When the true box and the predicted box have the same aspect ratio but different width and height, the value of the loss function is the same, which will make the loss function invalid. Therefore, the present invention designs a more robust loss function To solve this problem, the loss function for: , in It represents the intersection over union (IoU) loss. represents the Euclidean distance loss, represents the aspect ratio loss, are the true bounding box coordinates, are the bounding box coordinates of the predicted value, , , , ,in Respectively represent the upper left corner horizontal coordinate, ordinate and lower right corner horizontal coordinate, ordinate, Respectively represent the upper left corner horizontal coordinate, ordinate and lower right corner horizontal coordinate, ordinate of the predicted value bounding box; in , , Indicates an adjustable proportional factor, which can be freely adjusted according to actual needs and final effects; Represents the Euclidean distance between two points. By introducing the logarithmic difference between the length and width of the predicted box and the real box and the length and width of the minimum bounding box, the present invention can avoid the failure of the loss function in certain circumstances, thereby achieving faster convergence speed and better positioning results.

[0012] Step 4 includes: evaluating the average detection accuracy and detection speed of the target detection network model obtained after training in step 3, inputting the sample images in the training set into the target detection network model by setting the training rounds epochs, batch size batch, input image size, learning rate and intersection over union ratio IoU, training to obtain the best weight file of the target detection network model, continuously adjusting the training direction of the target detection network model through the loss function, and verifying whether the training has achieved the expected effect by calculating the mean average precision mAP (mean of Average Precision) value of the validation set. The best weight file is selected by screening the trained weight file, and the best weight file is loaded into the target detection network model to obtain a model suitable for small target detection in drone scenarios.

[0013] In step 5, the tracking algorithm process includes: Step 5-1, starting from the first frame of the input video frame, the video frame is input into the target detection network model trained in step 4, and the confidence threshold conf based on scene change distinction is obtained through an adaptive threshold matching method; The adaptive threshold matching method comprises: given a confidence score list corresponding to the detected object { },right{ } to sort in ascending order, if ,but ; Use the gradient descent method to find the confidence function distinction point, find the confidence threshold by calculating the gradient, and calculate the first discrete weighted difference ,in and is the set weight, represents the confidence function, represents the confidence score of the i-th target, represents the confidence score of the jth target; Find the maximum value of the confidence gradient to get the threshold ; Step 5-2, set the state vector of the Kalman filter KF to , where r represents the ratio of height h to width w, r=h / w; They represent the velocity component in the x-direction of the horizontal coordinate and the velocity component in the y-direction of the vertical coordinate, respectively. represents the speed component of the width change, Represents the angular velocity component of rotation; the width of the detection frame is used to replace the aspect ratio of the original Kalman filter state vector, and the quaternion aspect ratio r is not changed. This can compensate for the aspect ratio change caused by the movement of the drone or the relative motion of the object, and will not cause errors due to the small target of the object; Step 5-3, use global motion estimation GMC to represent background motion, extract image key points, and use sparse optical flow to perform feature tracking with local outlier suppression for translation; calculate the affine transformation matrix to transform the predicted bounding box from the coordinates of the t-1 frame to the coordinates of the next frame t; The calculating of the affine transformation matrix comprises: Input data is image point pairs , initialization parameters maximum number of iterations N (usually 1000), interior point threshold (usually 2 pixels), minimum number of samples s (usually 3), optimal affine transformation matrix and the optimal interior point set ; Respectively represent the horizontal and vertical coordinates of the initial point in the i-th image point pair, Respectively represent the horizontal and vertical coordinates of the update point in the i-th image point pair; For each iteration t=1,2,...,N, perform the following steps: Step 5-3-1, random sampling: randomly select s pairs of points from the image point pairs, and obtain the affine transformation matrix by solving the linear equations : , in and They represent the scaling and rotation of the horizontal and vertical axes respectively. and represent the shear amount of the horizontal axis and the shear amount of the vertical axis respectively, represents the translation of the horizontal axis, Indicates the translation of the vertical coordinate; , Respectively represent the horizontal and vertical coordinates of the initial point in the sth image point pair, , Respectively represent the horizontal and vertical coordinates of the update point in the sth image point pair; Step 5-3-2, for all image point pairs P, calculate the error between the transformed point and the target point : , If the error between the image point and the target point is less than the threshold , then mark the image point pair as an internal point; Step 5-3-3, if the current affine transformation matrix The number of interior points is greater than the optimal interior point set The number of internal points in , then update , ,in is the current interior point set; After the iteration is completed, use all the internal point sets Re-estimate the affine transformation matrix using the least squares method and return the optimal affine transformation matrix and the interior point set ; Step 5-4, divided according to the confidence threshold conf, the detection frames above the confidence threshold conf are compensated for camera offset, and then the Kalman filter is used to perform the first complete intersection-over-union (CIoU) and re-identification (ReID) data association; the detection frames below the confidence threshold conf are prepared for the second data CIoU association; The camera offset compensation includes: predicting the state vector according to step 5-2 and performing translation and rotation to obtain the final state quantity: , = , in is a matrix containing the scale and rotation parts of the affine matrix. The superscript T indicates the transpose. , is a transformation matrix describing the scaling and rotation of a two-dimensional plane. is the matrix of the translation part, , R represents the real space, Tr represents the translation matrix, ; It is the estimated value of the state at time t obtained by prediction based on the information at time t-1; is the result after Kalman's prediction. represents the predicted covariance matrix at time t, represents the covariance matrix at time t.

[0014] Beneficial effects: Compared with the prior art, the present invention realizes multi-target detection and trajectory prediction with high accuracy in special scenarios based on the deep learning benchmark model and the detection-based tracking framework. By introducing the context information extraction module CM, the context information and global scene extraction capabilities can be effectively improved. The key objects in the scene are extracted by introducing the hybrid attention module ML, providing good conditions for multi-target tracking. At the same time, the present invention introduces an adaptive threshold and camera offset compensation mechanism to enhance the anti-interference ability of tracking in different environments. The model proposed in the present invention was subjected to an ablation experiment and compared with the commonly used deep learning model. Under the same test conditions, the present invention has excellent detection and tracking performance. The present invention has broad prospects in practical applications, especially in the fields of intelligent monitoring, autonomous driving, and robot navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flow chart of the method of the present invention.

[0016] Figure 2 This is a diagram of the detector network architecture of the present invention.

[0017] Figure 3 It is a structural diagram of the tracking algorithm module in the present invention.

[0018] Figure 4 This is a structural diagram of the context information extraction module in the present invention.

[0019] Figure 5 This is a structural diagram of the mixed spatial channel attention module in the present invention.

[0020] Figure 6 This is a visualization diagram of tracking from the perspective of the drone of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0022] like Figure 1As shown, an embodiment of the present invention provides a method for tracking multiple targets in complex scenarios based on adaptive association, comprising the following steps: Step 1: obtain a video frame sequence containing multi-target motion and preprocess the video frame sequence; sort the video frames in chronological order to obtain a multi-target image containing multiple continuous frames.

[0023] Step 2, constructing a new target detection network model to further process the multi-target images of continuous frames; the new target detection network model uses the YOLO11 model as a benchmark model, including a backbone network, a neck network and a head network; The backbone network is used to extract the contextual relationship information between the target object and the scene, and obtains fusion features with enhanced representation capabilities by using spatial and channel information; four context information extraction modules CM (Context Module) are used to replace the four C3k2 modules in the backbone network of the original YOLO11 model; The neck network introduces a hybrid channel attention module ML to integrate features of different scales and extract local and global target features based on hybrid attention; The head network adopts the LIOU loss function to ensure the diagonal ratio while controlling the intersection-over-union ratio, and finally outputs the confidence and bounding box position; like Figure 2 As shown in the figure, the backbone structure first processes the input image through a convolutional layer (Conv). Next, four consecutive convolution downsampling (Conv) and context information extraction (CM) are constructed to extract features. After that, the feature pyramid pooling (SPPF) module is used to further enhance the expression ability of multi-scale features. In the neck structure part, the operation of the feature extraction component (C3k2) is first applied, and then upsampling (Upsample) is performed to restore the spatial resolution of the feature map. Next, the feature maps from different paths are merged through the concatenation operation (Concat) to generate a richer feature representation. Finally, in the head structure, the head module (LIoU-Detect) with the LIoU loss function is used to perform the final detection prediction and output the label of the target. The entire network structure can achieve efficient target detection tasks through the effective connection of the backbone, neck and head.

[0024] Step 3: Perform overall multi-scale feature training and learning on the new target detection network model, using the loss function Accelerate model convergence; Step 4: Perform performance evaluation and assessment on the target detection network model obtained after training in step 3 to obtain the final target detection network model. Use the target detection network model to predict the labels of multi-target detection, namely, the detection box and confidence score (x, y, w, h, score), where x, y represent the horizontal and vertical coordinates of the target, w represents the width of the detection box, h represents the height of the detection box, and score represents the confidence score of the detection box (the value is 0~1); In step 5, the detection box object is input into the tracking algorithm, and the target trajectory is predicted by constructing more than two adaptive matches, dynamically adjusting the matching threshold according to the confidence level, and performing camera offset compensation at the same time.

[0025] In step 2, the backbone network is constructed by the following steps: Step 2-1, construct a convolution layer (64, 3, 2) for the input image, where 64 represents the number of channels (filters) output by the convolution layer, 3 represents the size of the convolution kernel, that is, a 3x3 convolution kernel, which is used to process the local area of ​​the input image, and 2 represents the stride, that is, the stride of the convolution kernel sliding on the input image is 2; Then, four consecutive convolution downsamplings are constructed, where the downsampled convolution kernel size is 3×3 and the stride is 2.

[0026] Then build a context information extraction module to reduce the resolution and extract high-level features; Step 2-2, build a multi-scale feature utilization structure, connect the second context information extraction module CM to the neck network, and perform a concat operation with the upsample module of the neck network (in deep learning, concat refers to the operation of splicing multiple tensors along a specific dimension); Step 2-3, the backbone network is connected to a hybrid spatial channel attention module ML, and the hybrid spatial channel attention module ML is connected to the neck network; In step 2-2, the context information extraction module CM is as follows Figure 4 The context information extraction module CM includes a large kernel depthwise convolution and a pointwise convolution; The number of groups of the large kernel depthwise convolution is the same as the number of channels; The large kernel depthwise convolution first connects the ReLU activation function and the normalization operation to extract the global information of each channel; then a residual connection is performed, and then the pointwise convolution is repeated twice, and then the activation function ReLU and the normalization operation are connected; The two pointwise convolutions adopt an inverted bottleneck design, that is, the hidden dimension between the two pointwise convolutions is four times the input dimension, thereby achieving a mixture of spatial and channel information; this structure can improve the model's ability to detect small targets, especially in complex backgrounds; In step 2-3, the mixed spatial channel attention module ML is as follows Figure 5 The hybrid spatial channel attention module ML includes a local average pooling module LAP, a global average pooling module GAP, an anti-average pooling module UNGAP and a one-dimensional convolution module Conv1d; The hybrid spatial channel attention module ML first transforms the input feature vector into a vector of 1*C*ks*ks, and then extracts local spatial information through local average pooling LAP, where C represents the channel dimension and ks (kernel size) represents the convolution kernel size; two branches are used to convert the extracted local spatial information into one-dimensional vectors, wherein the first branch uses the global average pooling module GAP operation to contain global information, and the second branch directly performs a reshape operation to contain local spatial information; the first branch passes through the one-dimensional convolution module Conv1d, and the second branch passes through the reshape, and then the original resolution of the two branches is restored through the unaverage pooling module UNGAP, and then information fusion is performed, and finally the feature vector of the original input is residually connected, so as to output a feature vector of hybrid global, local space and channel attention; wherein the convolution kernel size k of the one-dimensional convolution module Conv1d is proportional to the channel dimension C, which means that when capturing local cross-channel interaction information, only the relationship between each channel and k adjacent channels is considered; The convolution kernel size k is determined by the following formula: , where b and These are all hyperparameters, with a default value of 2, and if k is an even number, it is increased by 1; is a proportional function.

[0027] In step 3, the overall multi-scale feature training and learning of the new target detection network model includes: Step 3-1: Use the COCO image recognition dataset for pre-training of the new target detection network model, and use the VisDrone2019-Det drone-view image recognition dataset for overall training; Step 3-2, set the training round epochs, batch size batch, input image size, learning rate and intersection over union (IoU), and use stochastic gradient descent SGD as the optimizer. In the present invention, the training round epochs=100, batch size batch=4, input image size is (640, 640), learning rate is 0.01, and IoU is 0.7.

[0028] In step 3, the loss function for: , in It represents the intersection over union (IoU) loss. represents the Euclidean distance loss, represents the aspect ratio loss, are the true bounding box coordinates, are the bounding box coordinates of the predicted value, , , , ,in Respectively represent the upper left corner horizontal coordinate, ordinate and lower right corner horizontal coordinate, ordinate, Respectively represent the upper left corner horizontal coordinate, ordinate and lower right corner horizontal coordinate, ordinate of the predicted value bounding box; in , , Indicates an adjustable proportional factor, which can be freely adjusted according to actual needs and final effects; Represents the Euclidean distance between two points. By introducing the logarithmic difference between the length and width of the predicted box and the real box and the length and width of the minimum bounding box, the present invention can avoid the failure of the loss function in certain circumstances, thereby achieving faster convergence speed and better positioning results.

[0029] In step 5, the overall framework of the tracking algorithm process is as follows Figure 3 The tracking algorithm process includes: Step 5-1, starting from the first frame of the input video frame, the video frame is input into the target detection network model trained in step 4, and the confidence threshold conf based on scene change distinction is obtained through an adaptive threshold matching method; The adaptive threshold matching method comprises: given a confidence score list corresponding to the detected object { },right{ } to sort in ascending order, if ,but ; Use the gradient descent method to find the confidence function distinction point, find the confidence threshold by calculating the gradient, and calculate the first discrete weighted difference ,in and is the set weight, represents the confidence function, represents the confidence score of the i-th target, represents the confidence score of the jth target; Find the maximum value of the confidence gradient to get the threshold ; Step 5-2, set the state vector of the Kalman filter KF to , where r represents the ratio of height h to width w, r=h / w; They represent the velocity component in the x-direction of the horizontal coordinate and the velocity component in the y-direction of the vertical coordinate, respectively. represents the speed component of the width change, Represents the angular velocity component of rotation; the width of the detection frame is used to replace the aspect ratio of the original Kalman filter state vector, and the quaternion aspect ratio r is not changed. This can compensate for the aspect ratio change caused by the movement of the drone or the relative motion of the object, and will not cause errors due to the small target of the object; Step 5-3, use global motion estimation GMC to represent background motion, extract image key points, and use sparse optical flow to perform feature tracking with local outlier suppression for translation; calculate the affine transformation matrix to transform the predicted bounding box from the coordinates of the t-1 frame to the coordinates of the next frame t; The calculating of the affine transformation matrix comprises: Input data is image point pairs , initialization parameters maximum number of iterations N (usually 1000), interior point threshold (usually 2 pixels), minimum number of samples s (usually 3), optimal affine transformation matrix and the optimal interior point set ; Respectively represent the horizontal and vertical coordinates of the initial point in the i-th image point pair, Respectively represent the horizontal and vertical coordinates of the update point in the i-th image point pair; For each iteration t=1,2,...,N, perform the following steps: Step 5-3-1, random sampling: randomly select s pairs of points from the image point pairs, and obtain the affine transformation matrix by solving the linear equations : , in and They represent the scaling and rotation of the horizontal and vertical axes respectively. and represent the shear amount of the horizontal axis and the shear amount of the vertical axis respectively, represents the translation of the horizontal axis, Indicates the translation of the vertical coordinate; , Respectively represent the horizontal and vertical coordinates of the initial point in the sth image point pair, , Respectively represent the horizontal and vertical coordinates of the update point in the sth image point pair; Step 5-3-2, for all image point pairs P, calculate the error between the transformed point and the target point : , If the error between the image point and the target point is less than the threshold , then mark the image point pair as an internal point; Step 5-3-3, if the current affine transformation matrix The number of interior points is greater than the optimal interior point set The number of internal points in , then update , ,in is the current interior point set; After the iteration is completed, use all the internal point sets Re-estimate the affine transformation matrix using the least squares method and return the optimal affine transformation matrix and the interior point set ; Step 5-4, divided according to the confidence threshold conf, the detection frames above the confidence threshold conf are compensated for camera offset, and then the Kalman filter is used to perform the first complete intersection-over-union (CIoU) and re-identification (ReID) data association; the detection frames below the confidence threshold conf are prepared for the second data CIoU association; The camera offset compensation includes: predicting the state vector according to step 5-2 and performing translation and rotation to obtain the final state quantity: , = , in is a matrix containing the scale and rotation parts of the affine matrix. The superscript T indicates the transpose. , is a transformation matrix describing the scaling and rotation of a two-dimensional plane. is the matrix of the translation part, , R represents the real space, Tr represents the translation matrix, ; It is the estimated value of the state at time t obtained by prediction based on the information at time t-1; is the result after Kalman's prediction. represents the predicted covariance matrix at time t, represents the covariance matrix at time t.

[0030] In this embodiment, the datasets used are VisDrone-DET2019, Visdrone2019-MOT and COCO. The VisDrone-DET2019 dataset uses a drone platform to collect moving targets at different locations and altitudes, including 8599 images. 540k target bounding boxes were found and annotated with 10 predefined categories: pedestrian, person1, car, van, bus, truck, motor, bicycle, awning-tricycle, tricycle. The dataset is divided into three subsets: training, validation and testing (6471 training subsets, 548 validation subsets, and 1580 test subsets). The VisDrone-MOT2019 dataset also uses a drone platform to collect moving targets at different locations and altitudes, consisting of 79 video clips and a total of 33366 frames. According to the feature differences of the moving targets in the scene, the dataset focuses on 5 selected label categories: pedestrian, car, van, bus, and truck. The COCO (Common Objects in Context) dataset is a large-scale image recognition dataset for object detection, segmentation, and captioning tasks. It contains more than 330,000 images, each annotated with 80 object categories and 5 captions describing scenes. The COCO dataset is widely used in computer vision research and has been used to train and evaluate many state-of-the-art object detection and segmentation models.

[0031] On the VisDrone-DET2019 dataset, the input image size is (640, 640), a total of 100 rounds of training and testing are set, the initial learning rate is set to 0.01, the batch size is 4, and the IoU is set to 0.7. The mAP0.5 value on the test set of VisDrone-DET2019 reached 34.7%, an increase of 1.5% compared to the baseline model, and the mAP0.5:0.95 value reached 20%, an increase of 1.6% compared to the baseline model. The HOTA value of VisDrone-MOT2019 reached 43.6%, MOTP reached 76.6%, and IDF1 reached 53.3%. Compared with the baseline model, they are improved by 7.1%, 2.1%, and 12.6%, respectively. The results of the ablation experiment are shown in Table 1. The tracking visualization effect from the perspective of the drone is shown in the figure. Figure 6 shown.

[0032] Table 1

[0033] The results show that the present invention has excellent detection and tracking performance, realizes multi-target detection and trajectory prediction with high accuracy in special scenarios, can effectively improve the ability to extract contextual information and global scenes, and extracts key figures in the scene through the attention mechanism, providing good conditions for multi-target tracking.

[0034] The present invention provides a method for tracking multiple targets in complex scenes based on adaptive association. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A multi-target tracking method in complex scenes based on adaptive association, characterized in that: The following steps are involved: Step 1, obtaining a video frame sequence containing multiple target motions and preprocessing the video frame sequence; Step 2, constructing a new target detection network model to further process the multi-target images of continuous frames; the new target detection network model uses the YOLO11 model as a benchmark model, including a backbone network, a neck network and a head network; The backbone network is used to extract the contextual relationship information between the target object and the scene, and obtains fusion features with enhanced representation capabilities by using spatial and channel information; four contextual information extraction modules CM are used to replace four C3k2 modules in the backbone network of the original YOLO11 model; The neck network introduces a hybrid channel attention module ML to integrate features of different scales and extract local and global target features based on hybrid attention; The head network adopts the LIOU loss function to ensure the diagonal ratio while controlling the intersection-over-union ratio, and finally outputs the confidence and bounding box position; Step 3: Perform overall multi-scale feature training and learning on the new target detection network model, using the loss function Accelerate model convergence; Step 4: Perform performance evaluation and assessment on the target detection network model obtained after training in step 3 to obtain the final target detection network model. Use the target detection network model to predict the labels of multi-target detection, namely, the detection box and confidence score (x, y, w, h, score), where x, y represent the horizontal and vertical coordinates of the target, respectively, w represents the width of the detection box, h represents the height of the detection box, and score represents the confidence score of the detection box; In step 5, the detection box object is input into the tracking algorithm, and the target trajectory is predicted by constructing more than two adaptive matches, dynamically adjusting the matching threshold according to the confidence level, and performing camera offset compensation at the same time.

2. The method according to claim 1, characterized in that Step 1 includes: setting the frame interval according to the motion intensity in the scene, dynamically sparsely sampling the original video, dynamically setting the sampling ratio according to the number of targets and the scene, and obtaining a video frame sequence; marking the detected objects, sorting the video frames in chronological order, obtaining a multi-target image containing more than two consecutive frames, performing histogram block local equalization preprocessing to obtain a small target image data set, and dividing the small target image data set into a training set, a validation set, and a test set.

3. The method according to claim 2, characterized in that In step 1, the histogram block local equalization preprocessing specifically includes the following steps: Step 1-1, divide the image into non-overlapping area blocks tiles; Step 1-2, calculate the histogram for each tile, and calculate the cumulative distribution function CDF based on the histogram; Step 1-3, set the gray level limit frequency. If the gray level frequency of a histogram of a tile exceeds the limit frequency, reduce the frequency and redistribute the reduced frequency to other gray levels, keeping the histogram sum unchanged. Step 1-4, use the cumulative distribution function CDF to perform local equalization and map the value of each pixel to a new value; In steps 1-5, bilinear interpolation is used to smooth the area between tiles.

4. The method according to claim 3, characterized in that In step 2, the backbone network is constructed by the following steps: Step 2-1, construct a convolution layer (64, 3, 2) for the input image, where 64 represents the number of channels of the convolution layer output filters, 3 represents the size of the convolution kernel, that is, a 3x3 convolution kernel, which is used to process the local area of ​​the input image, and 2 represents the stride, that is, the step length of the convolution kernel sliding on the input image is 2; Then construct 4 consecutive convolution downsampling, where the downsampled convolution kernel size is 3×3 and the stride is 2; Then build a context information extraction module to reduce the resolution and extract high-level features; Step 2-2, construct a multi-scale feature utilization structure, starting from the second context information extraction module CM connected to the neck network, and perform concat operation with the upsample module of the neck network; In step 2-3, the backbone network is connected to a hybrid spatial channel attention module ML, which is connected to the neck network.

5. The method according to claim 4, characterized in that In step 2-2, the context information extraction module CM includes a large kernel depthwise convolution and a pointwise convolution; The number of groups of the large kernel depthwise convolution is the same as the number of channels; The large kernel depthwise convolution first connects the ReLU activation function and the normalization operation to extract the global information of each channel; Then a residual connection is performed, followed by two pointwise convolutions, followed by the activation function ReLU and normalization operation. The two pointwise convolutions adopt an inverted bottleneck design, that is, the hidden dimension between the two pointwise convolutions is four times the input dimension, thereby achieving a mixture of spatial and channel information.

6. The method according to claim 5, characterized in that In step 2-3, the mixed spatial channel attention module ML includes a local average pooling module LAP, a global average pooling module GAP, an anti-average pooling module UNGAP and a one-dimensional convolution module Conv1d; The hybrid spatial channel attention module ML first transforms the input feature vector into a vector of 1*C*ks*ks, and then extracts local spatial information through local average pooling LAP, where C represents the channel dimension and ks represents the convolution kernel size; two branches are used to convert the extracted local spatial information into one-dimensional vectors, wherein the first branch uses the global average pooling module GAP operation to contain global information, and the second branch directly performs a reshape operation to contain local spatial information; the first branch passes through the one-dimensional convolution module Conv1d, and the second branch passes through the reshape, and then the original resolution of the two branches is restored through the un-average pooling module UNGAP, and then information fusion is performed, and finally the residual connection of the original input feature vector is used to output a feature vector of hybrid global, local space and channel attention; The convolution kernel size k of the one-dimensional convolution module Conv1d is proportional to the channel dimension C; The convolution kernel size k is determined by the following formula: ， where b and All are hyperparameters; is a proportional function.

7. The method according to claim 6, characterized in that In step 3, the overall multi-scale feature training and learning of the new target detection network model includes: Step 3-1: Use the COCO image recognition dataset for pre-training of the new target detection network model, and use the VisDrone2019-Det drone-view image recognition dataset for overall training; In step 3-2, set the training rounds epochs, batch size batch, input image size, learning rate and intersection over union (IoU), and use stochastic gradient descent SGD as the optimizer.

8. The method according to claim 7, characterized in that In step 3, the loss function for: ， in It represents the intersection over union (IoU) loss. represents the Euclidean distance loss, represents the aspect ratio loss, are the true bounding box coordinates, are the bounding box coordinates of the predicted value, , , , ,in Respectively represent the upper left corner horizontal coordinate, ordinate and lower right corner horizontal coordinate, ordinate, Respectively represent the upper left corner horizontal coordinate, ordinate and lower right corner horizontal coordinate, ordinate of the predicted value bounding box; in , , represents an adjustable scaling factor, Represents the Euclidean distance between two points.

9. The method according to claim 8, characterized in that Step 4 includes: evaluating the average detection accuracy and detection speed of the target detection network model obtained after training in step 3, inputting the sample images in the training set into the target detection network model by setting the training rounds epochs, batch size batch, input image size, learning rate and intersection over Union (IoU), training to obtain the best weight file of the target detection network model, continuously adjusting the training direction of the target detection network model through the loss function, and verifying whether the training has achieved the expected effect by calculating the average precision mAP value of the validation set; selecting the best weight file by screening the trained weight file, and loading the best weight file into the target detection network model to obtain a model suitable for small target detection in drone scenarios.

10. The method according to claim 9, characterized in that In step 5, the tracking algorithm process includes: Step 5-1, starting from the first frame of the input video frame, the video frame is input into the target detection network model trained in step 4, and the confidence threshold conf based on scene change distinction is obtained through an adaptive threshold matching method; The adaptive threshold matching method comprises: given a confidence score list corresponding to the detected object { },right{ } to sort in ascending order, if ,but ; Use the gradient descent method to find the confidence function distinction point, find the confidence threshold by calculating the gradient, and calculate the first discrete weighted difference ,in and is the set weight, represents the confidence function, represents the confidence score of the i-th target, represents the confidence score of the jth target; Find the maximum value of the confidence gradient to get the threshold ; Step 5-2, set the state vector of the Kalman filter KF to , where r represents the ratio of height h to width w, r=h / w; They represent the velocity component in the x-direction of the horizontal coordinate and the velocity component in the y-direction of the vertical coordinate, respectively. represents the speed component of the width change, Represents the angular velocity component of rotation; the width of the detection frame is used to replace the aspect ratio of the original Kalman filter state vector, without changing the aspect ratio r; Step 5-3, use global motion estimation GMC to represent background motion, extract image key points, and use sparse optical flow to perform feature tracking with local outlier suppression for translation; calculate the affine transformation matrix to transform the predicted bounding box from the coordinates of the t-1 frame to the coordinates of the next frame t; The calculating of the affine transformation matrix comprises: Input data is image point pairs , initialization parameters maximum number of iterations N, interior point threshold , minimum number of samples s, optimal affine transformation matrix and the optimal interior point set ; Respectively represent the horizontal and vertical coordinates of the initial point in the i-th image point pair, Respectively represent the horizontal and vertical coordinates of the update point in the i-th image point pair; For each iteration t=1,2,...,N, perform the following steps: Step 5-3-1, random sampling: randomly select s pairs of points from the image point pairs, and obtain the affine transformation matrix by solving the linear equations : ， in and They represent the scaling and rotation of the horizontal and vertical axes respectively. and represent the shear amount of the horizontal axis and the shear amount of the vertical axis respectively, represents the translation of the horizontal axis, Indicates the translation of the vertical coordinate; , Respectively represent the horizontal and vertical coordinates of the initial point in the sth image point pair, , Respectively represent the horizontal and vertical coordinates of the update point in the sth image point pair; Step 5-3-2, for all image point pairs P, calculate the error between the transformed point and the target point : ， If the error between the image point and the target point is less than the threshold , then mark the image point pair as an internal point; Step 5-3-3, if the current affine transformation matrix The number of interior points is greater than the optimal interior point set The number of internal points in , then update , ,in is the current interior point set; After the iteration is completed, use all the internal point sets Re-estimate the affine transformation matrix using the least squares method and return the optimal affine transformation matrix and the interior point set ; Step 5-4, divided according to the confidence threshold conf, the detection frames above the confidence threshold conf are compensated for camera offset, and then the Kalman filter is used to perform the first complete intersection-over-union (CIoU) and re-identification (ReID) data association; the detection frames below the confidence threshold conf are prepared for the second data CIoU association; The camera offset compensation includes: predicting the state vector according to step 5-2 and performing translation and rotation to obtain the final state quantity: ， = ， in is a matrix containing the scale and rotation parts of the affine matrix. The superscript T indicates the transpose. , is a transformation matrix describing the scaling and rotation of a two-dimensional plane. is the matrix of the translation part, , R represents the real space, Tr represents the translation matrix, ; It is the estimated value of the state at time t obtained by prediction based on the information at time t-1; is the result after Kalman's prediction. represents the predicted covariance matrix at time t, represents the covariance matrix at time t.

Images