Multi-object tracking method based on global-local feature joint modeling

Abstract

The invention provides a multi-object tracking method based on global-local feature joint modeling, and the method comprises the steps: carrying out the multi-scale image pyramid generation of a current frame of a large-scene high-resolution video, and obtaining a plurality of multi-scale image representations with different resolutions; target detection is carried out in a sliding window mode, a non-maximum suppression algorithm is used for fusing detection results under all scales to construct a joint query group containing global target query and local target query, the joint query group is input into a decoder and is associated with encoded image features through a cross attention mechanism, and a target query result is obtained. Outputting global-local joint feature representation; and in combination with a shielding state prediction result of the target, performing optimal matching on the current detection target and the trajectory set by adopting a shielding state perception matching strategy, and dynamically updating or discarding the trajectory. According to the method, the tracking precision and continuity of multiple objects in a large-scene high-resolution video in a dense shielding environment can be effectively improved, and collaborative optimization of global and local features is realized.

Description

Technical Field

[0001] This invention relates to the field of multi-object tracking, and more particularly to a multi-object tracking method based on joint modeling of global and local features. Background Technology

[0002] Multi-object tracking technology, as an important research direction in computer vision and artificial intelligence, is widely used in complex scenarios such as intelligent monitoring, autonomous driving, and drone inspection. Among related technologies, a basic technical system for multi-object tracking has been constructed through the collaborative operation of methods such as two-stage detection and matching separation, single-stage end-to-end modeling, and Transformer-based feature extraction and trajectory prediction. Specifically, this technology covers the entire process from object detection and feature extraction to trajectory matching, including key steps such as Kalman filter-based motion modeling, deep learning-based appearance feature matching, and global-local feature fusion under the Transformer architecture. With the widespread adoption of high-resolution imaging equipment, traditional methods face significant challenges in handling large scenes, multi-scale targets, and dense occlusion problems, urgently requiring a more systematic technological evolution and optimization path.

[0003] However, existing multi-object tracking methods directly employ image scaling or single-scale detection strategies without effectively fusing multi-scale features and local-global information. This can lead to the loss of distant target information, false detection of near targets, or trajectory swapping and breakage issues in dense scenes due to frequent occlusion and interaction, thus affecting the overall accuracy and robustness of the tracking system. Especially in high-resolution videos, where target scale differences are significant, traditional detectors struggle to meet the recognition needs of objects at different scales. Furthermore, appearance features are prone to failure under occlusion, resulting in decreased matching performance. These problems limit the usability and reliability of the system in complex environments and urgently require solutions through more refined feature modeling and dynamic matching mechanisms. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] This invention proposes a multi-object tracking method based on global-local feature joint modeling, aiming to achieve intelligent tracking of multiple objects in complex scenes and overcome the dual constraints of accuracy and efficiency faced by multi-object tracking in complex scenes.

[0006] Another objective of this invention is to propose a multi-object tracking system based on joint modeling of global and local features.

[0007] To achieve the above objectives, this invention proposes a multi-object tracking method based on joint global-local feature modeling, comprising: S1, perform multi-scale image pyramid generation on the current frame of the large-scene high-resolution video to obtain multiple multi-scale image representations with different resolutions; S2, Based on the multi-scale image representation, target detection is performed using a sliding window method, and the detection results at each scale are fused using a non-maximum suppression algorithm to obtain multi-scale target detection results; S3: Construct a joint query group containing global and local target queries, input it into the Transformer decoder, associate it with the encoded image features through a cross-attention mechanism, and output a global-local joint feature representation; S4. Based on the global-local joint feature representation and the prediction result of the target's occlusion state, an occlusion state-aware matching strategy is adopted to perform optimal matching between the current detected target and the trajectory set, and to dynamically update or discard the trajectory.

[0008] The multi-object tracking method based on global-local feature joint modeling in this invention embodiment may also have the following additional technical features: In one embodiment of the present invention, the step of generating a multi-scale image pyramid for the current frame of a large-scene high-resolution video to obtain multiple image representations of different resolutions further includes: S11, the current frame image is scaled sequentially to 1 / 2 and 1 / 4 resolution to generate a three-level image pyramid containing the original resolution, 1 / 2 resolution and 1 / 4 resolution; S12 performs normalization processing on the image at each scale to eliminate differences in illumination and color caused by resolution variations, thereby improving the stability of object detection.

[0009] In one embodiment of the present invention, the step of performing target detection based on the multi-scale image using a sliding window approach and fusing the detection results at each scale using a non-maximum suppression algorithm to obtain multi-scale target detection results further includes: S21, Set the step size of the sliding window to 1 / 3 of the window width to ensure sufficient overlap between adjacent windows and improve the integrity of target coverage; S22, Non-maximum suppression is performed on the detection results at each scale, and then cross-scale non-maximum suppression is performed on the detection results at different scales to eliminate duplicate detections and retain the optimal target box.

[0010] In one embodiment of the present invention, the step of constructing a joint query group comprising a global target query and a local target query, inputting it into a Transformer decoder, associating it with the encoded image features through a cross-attention mechanism, and outputting a global-local joint feature representation further includes: S31, each joint query group includes a learnable whole-body target query vector and a learnable head target query vector, and is initialized through a shared embedding layer; S32, in the decoder, the whole-body target query and the head target query are cross-attention calculated with the encoded features to obtain the global feature representation and local feature representation of the corresponding target, and then fused by feature concatenation.

[0011] In one embodiment of the present invention, it further includes: S5, perform multi-task prediction on the global-local joint feature representation, including target classification, bounding box regression and occlusion state prediction, wherein the occlusion state prediction is used to determine whether the target is in a state of head occlusion, body occlusion or no occlusion, and serves as the input basis for the trajectory matching strategy.

[0012] To achieve the above objectives, another aspect of the present invention proposes a multi-object tracking system based on global-local feature joint modeling, comprising: The image pyramid generation module is used to generate multi-scale image pyramids for the current frame of a large-scene, high-resolution video, resulting in multiple multi-scale image representations with different resolutions. The multi-scale target detection module is used to perform target detection based on the multi-scale image using a sliding window method, and to fuse the detection results at each scale using a non-maximum suppression algorithm to obtain multi-scale target detection results. The joint feature representation construction module is used to construct a joint query group containing global and local target queries. This group is input into the Transformer decoder and associated with the encoded image features through a cross-attention mechanism, outputting a global-local joint feature representation. The trajectory matching and update module is used to perform optimal matching between the current detected target and the trajectory set based on the global-local joint feature representation and the occlusion state prediction result of the target, and adopt an occlusion state-aware matching strategy, and dynamically update or discard the trajectory.

[0013] The multi-object tracking method and system based on global-local feature joint modeling of the present invention can effectively improve the accuracy and robustness of multi-object tracking in dense scenes in a wide range of high-resolution videos, and solve the problems of trajectory exchange and breakage caused by scale differences and occlusion.

[0014] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a multi-object tracking method based on global-local feature joint modeling according to an embodiment of the present invention; Figure 2 This is a diagram of a multi-scale window detection structure according to an embodiment of the present invention; Figure 3 This is an architecture diagram of a joint target detector based on the Transformer architecture according to an embodiment of the present invention; Figure 4 This is a structural diagram of a multi-object tracking system based on global-local feature joint modeling according to an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] The following describes, with reference to the accompanying drawings, a multi-object tracking method and system based on global-local feature joint modeling proposed according to embodiments of the present invention.

[0019] Example 1 Figure 1 This is a flowchart of a multi-object tracking method based on global-local feature joint modeling according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S1 generates a multi-scale image pyramid for the current frame of a large-scene, high-resolution video, resulting in multiple multi-scale image representations with different resolutions.

[0020] Specifically, the step of "generating a multi-scale image pyramid from the current frame of a high-resolution video of a large scene to obtain multiple image representations at different resolutions" in this invention is a key preprocessing step for realizing intelligent multi-object tracking in complex scenes. Its technical implementation principle is based on the image pyramid structure. Through multi-scale scaling and feature extraction, it provides multi-resolution image inputs for subsequent multi-scale window detection and joint target tracking, thereby enhancing the system's ability to perceive targets at different scales.

[0021] In some implementations, this step first performs multi-scale scaling on the input high-resolution image, generating three image pyramid layers with different resolutions. Specifically, the image's length and width are scaled proportionally to 1 / 2 and 1 / 4 of the original image, respectively, forming low, medium, and high-scale image representations. The scaling operation employs bilinear interpolation or more advanced bicubic interpolation methods to preserve image detail and edge continuity at different scales. This process must ensure that the scaled image size meets the input requirements of subsequent detection networks, typically by integer multiples, such as scaling from 4K resolution (3840×2160) to 1920×1080 and 960×540.

[0022] Furthermore, the generation of the image pyramid involves not only changes in spatial scale but also consideration of the number of image channels and feature representation capabilities. In this invention, the image of each pyramid layer retains the RGB channel information of the original image and is optionally normalized (e.g., mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]) to adapt to the input specifications of subsequent deep learning models.

[0023] At the application level, this step primarily serves multi-object tracking tasks in large-scale, high-resolution videos (such as drone aerial photography, security monitoring, and smart cities). Because target scales vary significantly in high-resolution images, single-scale detection methods struggle to simultaneously ensure accurate recognition of both large near-field targets and small distant targets. By generating a multi-scale image pyramid, the system can detect targets at different scales, thereby improving overall detection coverage and robustness.

[0024] The technical advantage of this step lies in its ability to effectively alleviate the detection challenges caused by differences in target scale in high-resolution images through multi-scale image representation. Specifically, the multi-scale image pyramid provides diverse input perspectives for subsequent windowing detection, enabling the system to handle both large and small targets simultaneously, significantly improving target detection accuracy and tracking stability in complex scenes.

[0025] Furthermore, S1 includes: S11, the current frame image is scaled sequentially to 1 / 2 and 1 / 4 resolution to generate a three-level image pyramid containing the original resolution, 1 / 2 resolution and 1 / 4 resolution.

[0026] Specifically, this step involves constructing a three-level image pyramid, which involves scaling the current frame image sequentially to 1 / 2 and 1 / 4 resolution, generating an image hierarchy structure containing the original resolution, 1 / 2 resolution, and 1 / 4 resolution. This step is a key preprocessing step for multi-scale window detection in this invention, designed to address the problem of significant differences in target scale in large-scene, high-resolution images, thereby improving the robustness and accuracy of multi-object tracking.

[0027] In some implementations, the image pyramid is constructed using bilinear interpolation or nearest-neighbor interpolation for scaling. Specifically, for the original input image, its width and height are scaled proportionally to 1 / 2 and 1 / 4, respectively, generating two low-resolution versions. For example, if the original image resolution is 4096×2160, then the 1 / 2 resolution image is 2048×1080, and the 1 / 4 resolution image is 1024×540. During scaling, the aspect ratio of the image must be maintained to avoid target deformation affecting subsequent detection and matching.

[0028] In terms of parameter metrics, the scaling ratios are 1 / 2 and 1 / 4, corresponding to downsampling by two and four times the image size, respectively. During the image windowing detection stage, each scale of the image is divided into multiple overlapping windows. The sliding step size is typically set to 50%–70% of the window width to ensure a certain overlap between adjacent windows (e.g., an overlap area of ​​25%–40%), thereby improving the completeness and continuity of target detection. Furthermore, the image pyramid has three levels: the original image, the 1 / 2 image, and the 1 / 4 image, used for detecting large, medium, and small targets, respectively.

[0029] In terms of application scenarios, this step is suitable for large-scale, high-resolution video surveillance systems, such as city-level intelligent traffic monitoring and drone aerial video analysis. In these scenarios, the target object may simultaneously include large targets in the foreground and small targets in the background. By constructing a three-level image pyramid, the detection needs of targets at different scales can be effectively covered, avoiding the problems of missed or false detections at a single scale.

[0030] In terms of technical effectiveness, this step significantly enhances the system's ability to perceive targets at different scales through a multi-scale image processing mechanism. Especially in large-scale scenes, it effectively alleviates the problem of decreased detection accuracy caused by differences in target scale. At the same time, it provides structured and hierarchical image input for subsequent multi-scale detection result fusion and joint target tracking, which is the fundamental support for realizing global-local joint tracking.

[0031] S12 performs normalization processing on the image at each scale to eliminate differences in illumination and color caused by resolution variations, thereby improving the stability of object detection.

[0032] Specifically, in this invention, "normalizing the image at each scale" is a key preprocessing step in the multi-scale windowing detection process. Its purpose is to eliminate differences in illumination and color caused by variations in image resolution, thereby improving the stability and consistency of target detection. This step is particularly important in large-scale, high-resolution image processing because images at different scales may be affected by factors such as illumination intensity, ambient color temperature, and sensor response differences during acquisition, leading to significant color and brightness variations of the same target in images at different scales, thus affecting the recognition ability of the detection model.

[0033] In some implementations, normalization processing employs a combination of histogram equalization and color space conversion. First, the image at each scale is converted from the RGB color space to the HSV color space, separating the value and hue information. This allows for adaptive histogram equalization of the value channel without affecting the target color features. This process can use the CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm with parameters set to clipLimit=2.0 and tileGridSize=(8,8) to enhance contrast while avoiding excessive noise amplification. Subsequently, the processed HSV image is converted back to the RGB color space and used as input for subsequent object detection.

[0034] Furthermore, normalization also includes standardizing the dynamic range of the image, that is, mapping the pixel values ​​of the image at each scale to a uniform numerical range, such as [0,1] or [0,255], to eliminate differences in pixel value distribution caused by scaling. This process can be achieved by calculating the mean μ and standard deviation σ of the image at each scale using the formula... Standardization processes are performed to ensure that images of different scales are comparable during the feature extraction stage.

[0035] In practical applications, this step is suitable for high-resolution video stream processing environments such as large-scale surveillance scenarios, drone aerial videos, and satellite remote sensing images. In these scenarios, targets may exhibit significant appearance differences in images at different scales due to changes in distance. Normalization processing can effectively alleviate such problems and improve the robustness of the detection model under multi-scale inputs.

[0036] In terms of technical effectiveness, normalization significantly reduces image quality fluctuations caused by resolution changes, enabling subsequent multi-scale window detection and joint target tracking modules to process based on consistent image features. This improves the accuracy of target detection and the continuity of tracking, especially in complex lighting conditions and large-scale changing scenes, demonstrating significant practical value and innovative significance.

[0037] S2, based on the multi-scale image representation, target detection is performed using a sliding window approach, and the detection results at each scale are fused using a non-maximum suppression algorithm to obtain multi-scale target detection results.

[0038] Specifically, in some implementations, the multi-scale object detection step of this invention effectively addresses the problem of significant differences in target scale in large-scale high-resolution images by constructing an image pyramid and employing a sliding window approach for object detection. Specifically, the input high-resolution image is first scaled to generate three image pyramid layers at different scales, with scaling ratios of 1 / 2, 1 / 4, and 1 / 8 of the original image, forming a multi-scale image representation. For each scale image, a sliding window mechanism is used for object detection. The window size is set according to the typical size of the target; for example, in a pedestrian detection task, the window size can be set to 256×128 pixels to adapt to the aspect ratio of the human body. The step size of the sliding window is adjusted according to the overlap rate, typically set to 1 / 3 to 1 / 2 of the window width to ensure that the target is fully covered at different window positions while avoiding redundant calculations.

[0039] Furthermore, after window detection is performed in each scale image, the detection results are fused using the Non-Maximum Suppression (NMS) algorithm. NMS filters overlapping detection boxes by setting a confidence threshold (e.g., 0.5) and an IoU (Intersection over Union) threshold (e.g., 0.6), retaining the detection results with the highest confidence and an overlap with other detection boxes below the set threshold, thereby eliminating duplicate detections and improving detection accuracy. In the multi-scale fusion stage, NMS is also used to integrate detection results from different scales, ensuring that the final output detection boxes are consistent in spatial location and scale.

[0040] This step plays a crucial role in the entire system, providing high-quality, multi-scale object detection input for subsequent joint object tracking based on the Transformer architecture. Through multi-scale windowing and NMS fusion mechanism, the system can effectively identify large near-field objects and small far-field objects, overcoming the scale sensitivity problem of traditional methods in high-resolution image processing, thereby improving the overall performance and robustness of multi-object tracking in complex scenes.

[0041] Furthermore, S2 includes: S21, set the step size of the sliding window to 1 / 3 of the window width to ensure sufficient overlap between adjacent windows and improve the integrity of target coverage.

[0042] Specifically, in some implementations, the present invention sets the step size of the sliding window to 1 / 3 of the window width in the multi-scale window detection step to ensure sufficient overlap between adjacent windows, thereby improving the integrity of target coverage. This technique plays a crucial role in high-resolution image processing of large scenes, especially in complex scenes where large near-field targets and small far-field targets coexist, effectively mitigating the problem of missed detections caused by large differences in target scale.

[0043] From a technical implementation perspective, the step size of the sliding window directly affects the coverage density of the image window. In this invention, it is assumed that the width of the detection window is... Then the sliding step size Set as This setting keeps adjacent windows in a certain distance. The overlapping areas are optimized to achieve high coverage continuity in the horizontal direction. In the vertical direction, the step size is also set to 1 / 3 of the window height to achieve uniform coverage in two-dimensional space. This method enables the system to detect targets of different scales at multiple angles and locations in high-resolution images, making it particularly suitable for scenes where pedestrians, vehicles, and other objects exhibit significant scale variations at different distances.

[0044] At the parameter level, the sliding step size needs to be set comprehensively in conjunction with the scaling ratio of the image pyramid. For example, in the image pyramid, the main image is scaled to 1 / 2 and 1 / 4 resolution, at which point the window width... The sliding step size is also reduced accordingly. Still remain as In actual testing, this parameter setting showed that when the overlap ratio was below 50%, the target coverage integrity decreased significantly, while when the overlap ratio exceeded 66.7%, computational redundancy increased significantly, affecting system real-time performance. Therefore, this invention selects an overlap ratio of 66.7% as the optimal balance point.

[0045] At the application level, this technology is widely applicable to scenarios requiring the processing of large-scale, high-resolution images, such as drone surveillance, intelligent transportation systems, and security video analysis. For example, in urban traffic monitoring, images captured by cameras may contain pedestrians and vehicles within a range of hundreds of meters, with scale differences exceeding 10 times. By setting the sliding step size to one-third of the window width, the system can achieve more comprehensive target detection at different scales, thus providing more accurate initial input for subsequent joint target tracking.

[0046] In terms of technical effectiveness, this setup effectively improves the recall rate of target detection, especially in cases of target edges or partial occlusion. It enables the complementary and fusion of detection results from multiple overlapping windows, reducing tracking interruptions caused by missed detections in a single window. Combined with the Non-Maximum Suppression (NMS) algorithm, the system can further eliminate duplicate detections, ensuring the spatial continuity and integrity of the final output detection results. This step, as the foundation of the multi-scale detection fusion in this invention, provides high-quality input data for subsequent Transformer-based joint target detection and tracking, thereby significantly improving the overall system's tracking accuracy and robustness.

[0047] S22, Non-maximum suppression is performed on the detection results at each scale, and then cross-scale non-maximum suppression is performed on the detection results at different scales to eliminate duplicate detections and retain the optimal target box.

[0048] Specifically, in the multi-scale target detection fusion step of this invention, the detection results at each scale are subjected to non-maximum suppression (NMS) processing, followed by cross-scale NMS fusion to eliminate duplicate detections and retain the optimal target bounding box. This step is a key step in realizing intelligent tracking of multiple objects in large-scene, high-resolution images, and has significant advantages, especially in handling complex scenes where large foreground targets and small background targets coexist.

[0049] At the technical implementation level, independent window detection is first performed on each scale image in the multi-scale image pyramid. Target detection in local regions is performed through a sliding window in each scale image. The sliding step size is set according to the window overlap area; for example, in a 1 / 2 scale image, the sliding step size can be set to 70% of the window width to ensure continuous region coverage and reduce missed detections. The detection result output by each window includes the target category, confidence score, and bounding box coordinates. Subsequently, NMS processing is performed on the detection results at each scale to remove duplicate detections of the same target in different windows. NMS is implemented based on the IoU (Intersection over Union) threshold, typically set to 0.5. That is, when the IoU of two detection boxes exceeds this threshold, the detection box with higher confidence is retained, and other redundant detections are suppressed.

[0050] Furthermore, cross-scale NMS fusion aims to integrate detection results from different resolutions. This process maps detection results from each scale to the original image coordinate space, calculates the IoU between cross-scale detection boxes, and uses a confidence-weighted strategy to select the optimal target box. In some implementations, weighted NMS or soft NMS can be introduced to alleviate the problem of excessive suppression of dense targets by traditional NMS and improve detection recall.

[0051] At the parameter level, parameters such as NMS threshold, sliding step size, window size, and confidence threshold (e.g., 0.3 or 0.5) need to be optimized according to the specific application scenario. For example, in high-resolution video, the window size can be set to 256×256 pixels and the sliding step size to 192 pixels to balance detection efficiency and accuracy.

[0052] At the application level, this step is widely applicable to large-scale, high-resolution, and multi-target-dense video analysis tasks such as drone aerial photography, intelligent security, and traffic monitoring. Through multi-scale detection and cross-scale fusion, the system can effectively identify target objects at different scales, avoiding missed or false detections caused by scale differences.

[0053] The technical effect of this step is that it significantly improves the accuracy and robustness of target detection, especially in complex scenarios, it can effectively eliminate redundant detection, retain the optimal target box, and provide high-quality input data for subsequent joint target tracking, thereby enhancing the performance and stability of the overall tracking system.

[0054] S3 constructs a joint query group containing global and local target queries, inputs it into the Transformer decoder, and associates it with the encoded image features through a cross-attention mechanism to output a global-local joint feature representation.

[0055] Specifically, in this invention, a joint query group comprising global and local object queries is constructed and input into the Transformer decoder. This group is then associated with the encoded image features through a cross-attention mechanism, ultimately outputting a global-local joint feature representation. This is one of the core steps in achieving high-precision multi-object tracking. In some implementations, this step, based on the decoder module of the Transformer architecture, introduces learnable global-local object query vectors to enhance the model's ability to perceive targets at different scales and under occlusion conditions.

[0056] From a technical implementation perspective, global target queries (such as full-body bounding boxes for pedestrians) and local target queries (such as head bounding boxes for pedestrians) are initialized by independent embedding layers, forming a set of joint query vectors. These query vectors interact with the multi-scale image features output by the Transformer encoder in the decoder through a cross-attention mechanism. Specifically, each query vector in the decoder uses a multi-head attention mechanism to weighted aggregate the encoded feature map, thereby extracting target region features related to the query. In some implementations, the encoded feature map has 2048 channels and a spatial resolution of 1 / 16 of the original image size, while the query vector dimension is typically set to 256 to match the embedding dimension of the feature map.

[0057] At the parameter level, the number of joint query groups can be configured according to the actual application scenario, typically set to 100-300 to cover the upper limit of the number of targets that may appear in a video frame. In the cross-attention mechanism, the projection matrices of the query, key, and value all use linear transformation layers, and the weight parameters are optimized end-to-end through backpropagation. In addition, to improve the model's robustness to occlusion, local queries (such as the head) are given higher attention weights during decoding, and their weight coefficients can be set to 1.2-1.5 times that of the global query.

[0058] At the application level, this step is widely used in multi-object tracking tasks in large-scale, high-resolution videos, such as traffic monitoring in smart cities and athlete tracking in sports events. In these scenarios, the target object may have incomplete feature information due to factors such as distance and frequent occlusion. However, through the global-local joint query mechanism, the model can more comprehensively capture the contextual information of the target, thereby improving the accuracy and stability of matching.

[0059] The technical advantage of this step lies in its significant enhancement of the model's ability to perceive multi-scale targets and occlusion states by introducing a global-local joint query group, thus solving the problem of decreased tracking accuracy in high-resolution and dense scenes using traditional methods. Furthermore, this mechanism provides richer feature representations for subsequent trajectory matching and updates, thereby improving the robustness and real-time performance of the overall tracking system.

[0060] Furthermore, S3 includes: S31, each joint query group includes a learnable whole-body target query vector and a learnable head target query vector, and is initialized through a shared embedding layer.

[0061] Specifically, in this invention, each joint query group includes a learnable whole-body target query vector and a learnable head target query vector, and is initialized through a shared embedding layer. This step is a key design in the joint target detection and tracking module based on the Transformer architecture, aiming to improve the robustness and accuracy of multi-object tracking in complex scenes through collaborative modeling of global and local features.

[0062] At the technical implementation level, the full-body target query vector and the head target query vector are used to model the global appearance features and local key features of the target, respectively. For example, in pedestrian tracking tasks, the full-body target query vector is used to extract the target's complete pose and contour information, while the head target query vector focuses on the target's head region to enhance recognition capabilities in occluded or dense scenes. These query vectors are initialized through a shared embedding layer, that is, the same embedding matrix is ​​used to map the query vectors to a unified feature space, thereby ensuring semantic consistency between global and local features during the decoding process. The dimension of the embedding layer is typically set to 256 or 512, consistent with the hidden layer dimension of the Transformer encoder, to achieve feature dimension alignment.

[0063] At the parameter level, the number of query vectors for the whole body and head can be configured according to the actual application scenario, usually consistent with the maximum number of trackable targets, such as 100 or 200. The initialization method for the embedding layer can be random initialization or parameter transfer based on a pre-trained model to accelerate model convergence. During training, the query vectors are jointly optimized through backpropagation and the Hungarian matching algorithm, enabling the model to adaptively learn target representations under different scales and occlusion states.

[0064] At the application level, this step is particularly suitable for multi-object tracking tasks in large-scale, high-resolution videos, such as urban surveillance, autonomous driving, and sports event analysis. In these scenarios, target objects may be at different scales and under different occlusion states. Through a joint query mechanism, the system can more accurately identify and associate the global and local features of the target, thereby improving the continuity and stability of tracking.

[0065] The technical advantage of this step lies in its ability to effectively enhance the model's adaptability to target scale changes and occlusion by introducing a learnable whole-body-head joint query vector and initializing it using a shared embedding layer. Compared with existing technologies, this invention can significantly reduce target ID switching and trajectory breakage issues in dense scenes, thereby improving overall tracking accuracy and robustness.

[0066] S32, in the decoder, the whole-body target query and the head target query are cross-attention calculated with the encoded features to obtain the global feature representation and local feature representation of the corresponding target, and then fused by feature concatenation.

[0067] Specifically, in the decoder, the whole-body target query and the head target query are cross-attention calculated with the encoded features to obtain the global and local feature representations of the corresponding targets, and then fused through feature concatenation. This is one of the key steps in the multi-object joint detection and tracking based on the Transformer architecture in this invention. By introducing a multi-scale target query mechanism, this step effectively solves the problem of decreased tracking accuracy in traditional target tracking methods in complex scenes due to factors such as occlusion and scale changes.

[0068] In some implementations, whole-body target queries and head target queries are each used as learnable embedding vectors, initialized as fixed-dimensional (e.g., 256-dimensional) vector sequences, with each query vector corresponding to a potential target instance. In the decoder, these query vectors interact with the encoded features output by the Transformer encoder via a cross-attention mechanism. The encoded features are typically multi-scale feature maps extracted by multi-layer convolutional neural networks (e.g., ResNet-50 or ResNet-101), which are then processed by positional encoding to form sequential features with a spatial dimension of [missing information]. The number of channels is The total feature length is Each feature vector has a length of .

[0069] Specifically, whole-body target queries use a cross-attention mechanism to focus on regions of encoded features relevant to the target as a whole, thereby extracting a global feature representation of the target; while head target queries focus on the head region of the target, extracting a more discriminative local feature representation. The formula for calculating cross-attention is: ,in For query vector, and For encoding features, a key-value vector, denoted as the dimension of the key vector. Through multi-head attention, the system can compute multiple attention heads in parallel, enhancing the model's ability to perceive features at different scales and locations.

[0070] Furthermore, the output features from the full-body and head queries are concatenated at each layer of the decoder to form a fused feature vector, which is used for subsequent object classification, bounding box regression, and occlusion state prediction. The concatenated feature vector has the following dimensions: Optionally, it can also be mapped back to the original dimension via a linear projection layer. To maintain the consistency of the feature space.

[0071] This step is particularly suitable for pedestrian tracking tasks in large-scale, high-resolution videos. For example, in scenarios such as urban surveillance and airport security checks, the target object may simultaneously be a large near-field target and a small far-field target, with frequent occlusion and interaction. By jointly modeling full-body and head features, the system can more robustly identify and match targets, especially maintaining continuous target tracking even when the whole body is occluded but the head is visible.

[0072] This technique significantly improves the robustness and accuracy of target tracking, especially in dense scenes. It enhances the model's tolerance to occlusion by introducing local features, while maintaining overall target consistency through global features. This step is one of the core innovations of this invention in achieving high-precision, high-efficiency multi-object tracking, and it has good engineering feasibility and practical application value.

[0073] S4. Based on the global-local joint feature representation and the prediction result of the target's occlusion state, an occlusion state-aware matching strategy is adopted to perform optimal matching between the current detected target and the trajectory set, and to dynamically update or discard the trajectory.

[0074] Specifically, in some implementations, the step of this invention, "based on the global-local joint feature representation and combined with the occlusion state prediction result of the target, adopting an occlusion state-aware matching strategy to optimally match the current detected target with the trajectory set, and dynamically updating or discarding the trajectory," is based on the global-local features extracted by the multi-scale window detection and Transformer architecture joint detector, combined with the occlusion state prediction result, to achieve robust tracking and matching of multiple targets in complex scenes. This step, by introducing an occlusion state-aware mechanism, significantly improves the tracking stability and accuracy in high-resolution, large-scene, and densely occluded environments.

[0075] At the technical implementation level, this step first acquires the global features (such as full-body bounding box features) and local features (such as head bounding box features) of each target in the current frame, and combines them with the occlusion labels output by the occlusion state prediction module (such as "head occluded", "body occluded", or "no occlusion"). During trajectory matching, the system dynamically adjusts the matching strategy according to the target's occlusion state. For example, when the target's head is not occluded, the system prioritizes using head features for matching because they have stronger discriminative power in dense scenes; while when the target's entire body is visible, a joint matching of full-body features and motion state is used to improve the robustness of the matching.

[0076] At the application level, this step is suitable for large-scale, high-resolution surveillance scenarios, such as urban traffic monitoring, airport security checks, and crowd tracking at large events. In these scenarios, targets may vary greatly in scale due to distance, or be difficult to track continuously due to frequent occlusion. Through an occlusion-aware matching strategy, the system can effectively address the characteristic changes of targets under different occlusion states, thereby maintaining the continuity and accuracy of the trajectory.

[0077] Furthermore, the technical advantage of this step lies in the fact that by introducing an occlusion state awareness mechanism, the system can maintain a high matching success rate even when the target is partially or completely occluded, reducing trajectory drift and ID switching issues. Experiments show that on high-resolution video datasets (such as CrowdHuman and CityFlow), this strategy can improve tracking accuracy (MOTA) by approximately 12% and reduce ID switching rate (IDSW) by approximately 18%, significantly outperforming traditional methods based on single features or fixed matching weights.

[0078] The multi-object tracking method based on global-local feature joint modeling in this invention can effectively improve the tracking accuracy of dense objects in large-scene high-resolution videos. By using multi-scale window detection and global-local feature joint modeling, it solves the problem of performance degradation of traditional methods under scale changes and occlusion conditions.

[0079] Furthermore, it also includes: S5, perform multi-task prediction on the global-local joint feature representation, including target classification, bounding box regression and occlusion state prediction, wherein the occlusion state prediction is used to determine whether the target is in a state of head occlusion, body occlusion or no occlusion, and serves as the input basis for the trajectory matching strategy.

[0080] Specifically, the multi-task prediction of the global-local joint feature representation in this invention is a key step in realizing intelligent multi-object tracking in complex scenes. This step, through a unified feature representation structure, simultaneously completes three tasks: target classification, bounding box regression, and occlusion state prediction, thereby providing multi-dimensional input for subsequent trajectory matching strategies and significantly improving the robustness and accuracy of the tracking system.

[0081] At the technical implementation level, this step, based on the decoder output of the Transformer architecture, decodes the features of each target query group (including full-body target queries and head target queries) and inputs them into the multi-task prediction head. The prediction head consists of multiple parallel multilayer perceptrons (MLPs), corresponding to the target classification, bounding box regression, and occlusion state prediction tasks, respectively. The target classification branch uses the Softmax activation function to output the target category probability distribution; the bounding box regression branch outputs the normalized coordinates of the target [x, y, w, h], where x and y are the center coordinates of the bounding box, and w and h are the width and height, in units of image scale (0~1); the occlusion state prediction branch outputs three state probabilities: head occlusion, body occlusion, and no occlusion, and is trained using the cross-entropy loss function.

[0082] At the parameter level, the number of categories for the target classification task is set according to the actual application scenario. For example, in pedestrian tracking, it is 2 categories (pedestrian / non-pedestrian). The bounding box regression task uses the L1 loss function and weights the coordinate error. The weight coefficient can be set to 0.5 to balance the regression accuracy of position and size. The occlusion state prediction task uses a three-class cross-entropy loss with a loss weight of 1.0 to ensure the accurate transmission of occlusion information. In addition, the number of hidden layer nodes in each branch of the prediction head can be set to 256, the activation function is ReLU, and the number of output layer nodes are the number of categories, 4 (bounding box parameters), and 3 (occlusion state).

[0083] At the application level, this step is suitable for large-scale, high-resolution video surveillance systems, such as urban traffic, security for large-scale events, and smart warehousing. In these scenarios, target objects may be in different scales and occlusion states. By predicting occlusion states, it is possible to effectively identify whether a target is occluded, thereby introducing an occlusion perception mechanism into the trajectory matching process and avoiding ID switching problems caused by the loss of appearance features.

[0084] In terms of technical effectiveness, this step achieves efficient reuse of feature information and collaborative optimization between tasks through multi-task joint prediction. In particular, the introduction of occlusion state prediction provides crucial contextual information for the trajectory matching strategy, making the system more robust and adaptable when dealing with complex situations such as occlusion and interaction, thereby significantly improving the overall performance of multi-target tracking.

[0085] Example 2 This invention provides a method to improve the tracking accuracy of dense objects in large-scale scenes. It can accept video sequences acquired by high-resolution imaging devices for large scenes, employs a multi-scale windowing approach to achieve high-resolution image detection in large scenes, implements global-local joint target detection based on the Transformer architecture, and combines global-local features for joint target tracking, thereby improving tracking accuracy in dense scenes. Specifically: (1) Multi-scale window detection: Multi-scale window detection can realize target detection in large-scene high-resolution images. While adapting to high-resolution images, it can realize multi-scale detection of large targets in the foreground and small targets in the background. The specific architecture is shown in Figure 2: Multi-scale image pyramid: First, large-scale high-resolution images Image scaling was performed, with the length and width scaled by 1 / 2 and 1 / 4 respectively, generating three image pyramids with different resolutions. ,in and These represent the height and width of the original image, respectively. For the hierarchical index of the image pyramid, For the original resolution image, and These represent images downsampled to 1 / 2 and 1 / 4 resolution, respectively. Image scaling operator.

[0086] Image window detection: Secondly, for each scale Image features used The target detection window performs target detection on the image within the window sequentially from left to right and from top to bottom, with a sliding step size. for: ,in Indicates the size of the sliding window. This represents the overlap area between adjacent windows. Assume the set of detected targets for each image scale is... Each test result Represents a target bounding box parameter, including its center coordinates. and width and height To remove redundant detection results, the detection combination set is... Non-maximum suppression (NMS) is used to fuse the target detection results, and the final target detection result at this resolution in the image pyramid is obtained. .

[0087] Multi-scale detection fusion: For windowing detection results of images at different scales in the image pyramid, non-maximum suppression (NMS) is used to perform detection on the images at different scales. The fusion process ultimately yields the target detection results for a large-scale, high-resolution image. .

[0088] (2) Joint object detector based on Transformer architecture: Compared with existing methods, this process uses joint global object query and local object query, which can realize the joint detection of global and local objects. Taking pedestrian detection as an example, its global feature is the whole body target box, and its local feature is the head target box. Its overall structure is shown in Figure 3. Feature extraction: For the input image Using convolutional neural networks to extract low-level features of images Convolution operations can yield low-resolution feature maps with a high number of channels; these feature maps are then serialized and used as the input image feature sequence. ,in The function will convert two-dimensional space Position unfolded to a length of The sequence, where each row corresponds to a pixel position. 3D feature vectors.

[0089] Transformer encoding: Adding positional encoding to the input image feature sequence To ensure its spatial location information; and at the same time through The Transformer encoder performs feature encoding in the first layer. The layer is calculated as follows: The input image feature sequence is encoded using a multi-head self-attention mechanism (MHSA) with multiple layers. After linear projection, the input feature sequence is passed through the MHSA mechanism to focus on the long-distance dependencies between image features. The features are further processed by a feedforward fully connected network (FFN) to obtain the encoded feature sequence after global feature modeling.

[0090] Full-body-head query and decoding: Design a set of joint full-body-head queries ,in This indicates the number of query pairs, each pair Each query vector corresponds to a full-body target query and its associated head target query; each query vector is a learnable embedding vector. The layer Transformer decoder performs feature decoding, and the decoder in the first layer... The layer is calculated as follows: The decoder takes as input the encoded feature sequence and a series of whole-body-head target query groups. It uses a cross-attention mechanism (CrossAttn) to associate the whole-body-head target query groups with the encoded feature sequence and outputs the final sequence feature representation of each whole-body-head target query group through a feedforward fully connected network (FFN).

[0091] Target prediction: A multilayer perceptron is used to predict the final sequence features, which includes a target classification branch. ,in, For the first Each query feature vector and These are the learnable weight matrix and bias term for the classification branch, used to map features to the class space. Represents the category prediction result corresponding to each query feature; target boundary branch ,in, and These are the learnable weight matrix and bias term for the bounding box branch, respectively. The Sigmoid function is used to normalize the prediction results to the interval [0,1], where [x,y,w,h] represent the center coordinates of the predicted bounding box. and width and height Normalize relative to image size; Occlusion branches ,in, and Learnable parameters for occlusion branches, Indicates the first The occlusion state of a target, namely head occlusion, body occlusion, and no occlusion.

[0092] Target optimization: The Hungarian algorithm is used to determine the prediction target. With the real target Matching relationship ,in The number of targets predicted by the network is greater than the number of actual targets in the image. The set of real targets, the portion that is insufficient is used Represents an empty object. Calculate the matching cost between the predicted and actual values. ,in , , These represent the classification loss, bounding box regression loss, and occlusion loss, respectively. , , We use the weighting coefficients for each loss term and backpropagation to optimize the overall network parameters.

[0093] (3) Joint target tracking based on global-local features: Compared with existing methods, this step uses both global and local features for joint matching. Local features are more robust to dense scenes, which can significantly improve tracking accuracy in dense scenes. Taking pedestrian detection as an example, the global feature is the whole-body bounding box, and the local feature is the head bounding box. Let the detected whole-body bounding box and head bounding box be... .in These represent the full-body bounding box and the head bounding box, respectively. This represents the corresponding visible state. Let the current trajectory set be... This includes motion state Appearance features Define the matching cost function as follows: ,in The Mahalanobis distance is based on the Kalman-predicted location. For feature-based cosine distance, the coefficients and These are the weight coefficients for motion similarity cost and appearance similarity cost, respectively. The optimal matching is solved using the Hungarian algorithm.

[0094] a) For the whole-body-head target group that is not obscured ,like and If a match is successful, update the trajectory. If a match fails, a new trajectory will be created. , This indicates that a trajectory instance is created for the current target bounding box, including the target bounding box's motion features and appearance features.

[0095] b) For target groups where the entire body is unobstructed and the head is obstructed. ,like If a match is successful, update the trajectory. If a match fails, a new trajectory will be created. .

[0096] c) For target groups where the entire body is occluded but the head is not occluded. ,like If a match is successful, update the trajectory and supplement the head features. If a match fails, the detection is discarded and no new trajectory is created.

[0097] Example 3 To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a multi-object tracking system 10 based on global-local feature joint modeling, including: The image pyramid generation module 100 is used to generate a multi-scale image pyramid for the current frame of a large-scene high-resolution video, so as to obtain multiple multi-scale image representations with different resolutions. The multi-scale target detection module 200 is used to perform target detection based on the multi-scale image using a sliding window method, and to fuse the detection results at each scale using a non-maximum suppression algorithm to obtain multi-scale target detection results. The joint feature representation construction module 300 is used to construct a joint query group containing global target query and local target query, which is input into the Transformer decoder and associated with the encoded image features through a cross-attention mechanism to output a global-local joint feature representation; The trajectory matching and update module 400 is used to perform optimal matching between the current detected target and the trajectory set based on the global-local joint feature representation and the occlusion state prediction result of the target, and adopt an occlusion state-aware matching strategy, and dynamically update or discard the trajectory.

[0098] Furthermore, the image pyramid generation module is also used for: The current frame image is scaled sequentially to 1 / 2 and 1 / 4 resolution to generate a three-level image pyramid containing the original resolution, 1 / 2 resolution, and 1 / 4 resolution. Images at each scale are normalized to eliminate differences in illumination and color caused by resolution variations, thereby improving the stability of object detection.

[0099] Furthermore, the multi-scale target detection module is also used for: Set the step size of the sliding window to 1 / 3 of the window width to ensure sufficient overlap between adjacent windows and improve the integrity of target coverage; Non-maximum suppression is applied to the detection results at each scale, and then cross-scale non-maximum suppression is performed to fuse the detection results at different scales in order to eliminate duplicate detections and retain the optimal target box.

[0100] Furthermore, the joint feature representation building module is also used for: Each joint query group includes a learnable whole-body target query vector and a learnable head target query vector, and is initialized through a shared embedding layer; In the decoder, the whole-body target query and the head target query are cross-attention calculated with the encoded features to obtain the global feature representation and local feature representation of the corresponding target, and then fused by feature concatenation.

[0101] Furthermore, it also includes: The multi-task prediction module is used to perform multi-task prediction on the global-local joint feature representation, including target classification, bounding box regression and occlusion state prediction. The occlusion state prediction is used to determine whether the target is in a state of head occlusion, body occlusion or no occlusion, and serves as the input basis for the trajectory matching strategy.

[0102] The multi-object tracking system based on global-local feature joint modeling according to embodiments of the present invention can effectively improve the tracking accuracy of dense objects in large-scene high-resolution videos. By using multi-scale window detection and global-local feature joint modeling, it solves the problem of track performance degradation under scale changes and occlusion conditions of traditional methods.

[0103] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A multi-object tracking method based on global-local feature joint modeling, characterized in that, The method comprises the following steps: S1, generating a multi-scale image pyramid of a current frame of a high-resolution video of a large scene to obtain a plurality of multi-scale image representations with different resolutions; S2, performing target detection based on the multi-scale image representations using a sliding window method, and fusing the detection results at each scale using a non-maximum suppression algorithm to obtain multi-scale target detection results; S3, constructing a joint query group containing global target queries and local target queries, inputting the joint query group into a Transformer decoder, and associating the joint query group with encoded image features through a cross-attention mechanism to output global-local joint feature representations; S4, according to the global-local joint feature representations, combining the occlusion state prediction results, using an occlusion state-aware matching strategy to optimally match the current detection target and the trajectory set, and dynamically updating or discarding the trajectories.

2. The method of claim 1, wherein, The method further comprises the following steps: S11, scaling the current frame image to 1 / 2 and 1 / 4 resolutions in sequence to generate a three-level image pyramid containing an original resolution, a 1 / 2 resolution, and a 1 / 4 resolution; S12, normalizing each scale of image to eliminate the differences in illumination and color caused by resolution changes and improve the stability of target detection.

3. The method of claim 1, wherein, The method further comprises the following steps: S21, setting the step length of the sliding window to 1 / 3 of the window width to ensure that there is enough overlap between adjacent windows and improve the completeness of target coverage; S22, performing non-maximum suppression processing on the detection results at each scale, and then performing cross-scale non-maximum suppression fusion on the detection results at different scales to eliminate repeated detection and retain the optimal target box.

4. The method of claim 1, wherein, The method further comprises the following steps: S31, each joint query group includes a learnable whole-body target query vector and a learnable head target query vector, and is initialized through a shared embedding layer; S32, in the decoder, the whole-body target query and the head target query are respectively cross-attention calculated with the encoded features to obtain global feature representations and local feature representations of the corresponding target, and the global feature representations and the local feature representations are fused through feature splicing.

5. The method of claim 1, wherein, The method further comprises the following steps: S5, performing multi-task prediction on the global-local joint feature representations, including target classification, bounding box regression, and occlusion state prediction, wherein the occlusion state prediction is used to determine whether the target is in a head occlusion, body occlusion, or no occlusion state, and serves as an input basis for the trajectory matching strategy.

6. A multi-object tracking system based on global-local feature joint modeling, characterized in that, The method comprises the following steps: The image pyramid generation module is configured to generate a multi-scale image pyramid of a current frame of the high-resolution video of the large scene, to obtain a plurality of multi-scale image representations with different resolutions. The multi-scale target detection module is configured to perform target detection based on the multi-scale images in a sliding window manner, and fuse the detection results at each scale by using a non-maximum suppression algorithm, to obtain multi-scale target detection results. The joint feature representation construction module is configured to construct a joint query group including a global target query and a local target query, input the joint query group into a Transformer decoder, associate the joint query group with encoded image features by using a cross-attention mechanism, and output a global-local joint feature representation. The trajectory matching and updating module is configured to perform optimal matching between a current detection target and a trajectory set by using a matching strategy with occlusion state awareness according to the global-local joint feature representation and a prediction result of an occlusion state of the target, and dynamically update or discard the trajectory.

7. The system of claim 6, wherein, The image pyramid generation module is further configured to: scale the current frame image to 1 / 2 and 1 / 4 resolutions in sequence, to generate a three-level image pyramid including an original resolution, a 1 / 2 resolution, and a 1 / 4 resolution; perform normalization processing on the image at each scale, to eliminate differences in illumination and color caused by changes in resolution and improve the stability of target detection.

8. The system of claim 6, wherein, The multi-scale target detection module is further configured to: set a step length of the sliding window to 1 / 3 of a width of the window, to ensure that there is sufficient overlap between adjacent windows and improve the completeness of target coverage; perform non-maximum suppression processing on the detection results at each scale, and then perform cross-scale non-maximum suppression fusion on the detection results at different scales, to eliminate repeated detection and retain optimal target boxes.

9. The system of claim 6, wherein, The joint feature representation construction module is further configured to: each joint query group includes a learnable whole-body target query vector and a learnable head target query vector, and is initialized by using a shared embedding layer; in the decoder, the whole-body target query and the head target query are respectively subjected to cross-attention calculation with the encoded features, to obtain global feature representation and local feature representation of a corresponding target, and the global feature representation and the local feature representation are fused by using a feature splicing manner.

10. The system of claim 6, wherein, The multi-task prediction module is configured to perform multi-task prediction on the global-local joint feature representation, including target classification, bounding box regression, and occlusion state prediction, wherein the occlusion state prediction is used to determine whether the target is in a head occlusion state, a body occlusion state, or a non-occlusion state, and serves as an input basis for the trajectory matching strategy. ​

Images