Multi-camera multi-target tracking method and device, equipment and storage medium

Abstract

The invention provides a multi-camera multi-target tracking method and device, equipment and a storage medium, and relates to the technical field of visual tracking. According to the method, in a multi-camera multi-target tracking scene, the target object cannot be continuously tracked in the first video stream collected by the initial camera, and the second video stream of the associated camera is decoded in parallel, so that the image data processing efficiency is improved. And inputting the image frame data into an image processing model corresponding to each associated camera to carry out feature extraction and mask prediction, thereby realizing efficient target identification and positioning. The current image features are compared and matched with the target template features, the image frames containing the target object are quickly screened out, and the recognition efficiency is improved. The target tracking information is extracted from the associated camera image frame data of the detected target object, and the target tracking trajectory is generated, so that cross-camera continuous tracking of the target object is realized, the integrity and accuracy of the trajectory are ensured, and the tracking efficiency and tracking continuity of multi-camera multi-target tracking are effectively improved.

Description

Technical Field

[0001] This application relates to the field of visual tracking technology, and in particular to a multi-camera, multi-target tracking method, apparatus, device, and storage medium. Background Technology

[0002] Multi-target tracking (MOT) technology is crucial in fields such as smart security, intelligent transportation, and robot navigation. However, in the face of increasingly complex real-world scenarios, especially edge computing environments with large-scale coverage by multiple cameras and high concurrency and real-time requirements, existing technologies face many challenges.

[0003] On the one hand, traditional tracking methods often rely on bounding box detection, which can easily lead to problems such as target loss and ID switching when the target is occluded, moves across viewpoints, or its appearance changes significantly.

[0004] On the other hand, traditional tracking methods lack versatility and adaptability, and need to be optimized for specific application scenarios (monitoring, industrial inspection).

[0005] On the other hand, in multi-camera scenarios, the target may move between different cameras, or be occluded or briefly out of view from the same camera's perspective, making it impossible for traditional single-camera tracking methods to continuously track the target. Existing target re-identification (Re-ID) methods rely on simple visual feature matching, which is easily affected by factors such as changes in viewpoint, lighting conditions, and target pose, resulting in insufficient robust cross-camera association capabilities and difficulty in coping with complex and changing environments.

[0006] In addition, while mainstream vision models such as SAM2 perform well in image and video segmentation, their large number of parameters and complex Vision Transformer (ViT) encoder structure result in high computational requirements and slow inference speed, which limits their widespread application in scenarios with high concurrency and real-time requirements for multi-camera operation at the edge.

[0007] Therefore, how to improve the efficiency of multi-target tracking in multi-camera scenarios has become an urgent technical problem to be solved. Summary of the Invention

[0008] This application provides a multi-camera multi-target tracking method, apparatus, device, and storage medium, aiming to improve the efficiency of multi-target tracking in multi-camera scenarios.

[0009] In a first aspect, this application provides a multi-camera multi-target tracking method, which includes the following steps: When the target object cannot be continuously tracked in the first video stream initially captured by the camera, the target template feature of each target object is extracted from the image frames of the first video stream. Image frames are decoded in parallel from a second video stream acquired by at least one associated camera that is associated with the initial camera to obtain image frame data corresponding to each associated camera; The image frame data corresponding to each of the associated cameras is input into the image processing model corresponding to each associated camera for feature extraction and mask prediction, so as to obtain the current image features of the current image frame of each associated camera. The current image features of the current image frame of each associated camera are compared and matched with the target template features, and the detection result of the target object in the current image frame of each associated camera is output. When the detection result indicates that the target object exists in the current image frame of the associated camera, the target tracking information of the target object in the image frame data corresponding to the associated camera is extracted, and the target tracking trajectory of the target object is generated.

[0010] Secondly, this application also provides a multi-camera multi-target tracking device, the multi-camera multi-target tracking device comprising: The template feature extraction module is used to extract the target template features of each target object in the image frames of the first video stream when the target object cannot be continuously tracked in the first video stream initially captured by the camera. An image frame decoding module is used to decode image frames in parallel from a second video stream acquired by at least one associated camera associated with the initial camera, to obtain image frame data corresponding to each associated camera; The image frame feature extraction module is used to input the image frame data corresponding to each of the associated cameras into the image processing model corresponding to each associated camera for feature extraction and mask prediction, so as to obtain the current image features of the current image frame of each associated camera. The target object detection module is used to compare and match the current image features of the current image frame of each associated camera with the target template features, and output the detection result of the target object in the current image frame of each associated camera; The trajectory generation module is used to extract the target tracking information of the target object in the image frame data corresponding to the associated camera when the detection result indicates that the target object exists in the current image frame of the associated camera, and generate the target tracking trajectory of the target object.

[0011] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the multi-camera multi-target tracking method described above.

[0012] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the multi-camera multi-target tracking method described above.

[0013] This application provides a multi-camera multi-target tracking method, apparatus, computer device, and storage medium. In multi-camera multi-target tracking scenarios, where the target object cannot be continuously tracked in the initial video stream acquired by the first camera, parallel decoding of the second video stream from associated cameras accelerates the acquisition of image frame data, saves decoding waiting time, and improves image data processing efficiency. Inputting the image frame data into the image processing models corresponding to each associated camera for feature extraction and mask prediction allows for the simultaneous acquisition of image frame feature information from each associated camera, achieving efficient target recognition and localization. Comparing and matching the current image features with target template features quickly filters out image frames containing the target object, improving recognition efficiency. Extracting target tracking information and generating a target tracking trajectory from the associated camera image frame data where the target object is detected enables continuous cross-camera tracking of the target object in multi-camera scenarios, ensuring the integrity and accuracy of the trajectory and effectively improving the tracking efficiency and continuity of multi-camera multi-target tracking. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a first embodiment of a multi-camera multi-target tracking method provided in this application; Figure 2 A schematic diagram of the hardware architecture of the multi-camera, multi-target tracking system based on SAM2 and multiple TPU inference cores provided in this application; Figure 3 A schematic diagram of the lightweight improved SAM2 model structure provided in this application; Figure 4 A schematic diagram of the calculation process for the multi-camera multi-target tracking method provided in this application; Figure 5 This is a schematic diagram of the structure of a first embodiment of a multi-camera multi-target tracking device provided in this application; Figure 6 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0019] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of a multi-camera, multi-target tracking method provided in this application.

[0021] like Figure 1 As shown, the multi-camera multi-target tracking method includes steps S101 to S105.

[0022] S101. When the target object cannot be continuously tracked in the first video stream initially captured by the camera, extract the target template feature of each target object in the image frame of the first video stream. In one embodiment, in a multi-camera scenario, cameras distributed at different locations can capture scene images of different areas. There can be overlapping captured images between different cameras, or the target tracking area can be covered by stitching. Depending on the application scenario, the target object also varies. For example, it could be different pedestrian targets in a shopping mall, moving targets such as pedestrians and vehicles tracked on roads, flying targets such as birds, drones, and airplanes in the sky, or targets such as self-moving robots in a factory.

[0023] In one embodiment, the camera that first tracks the target object can be used as the initial camera, or a camera can be manually selected as the initial camera. The trajectory of the target object is tracked in the first video stream captured by the initial camera. The target object can be filtered according to preset tracking conditions, or it can be manually selected. For example, tracking conditions can include the type of the object being tracked (such as pedestrians, vehicles, robots, birds, drones, etc.) and its appearance (such as shape, labels, colors, icons, etc.).

[0024] In one embodiment, there may be one or more target objects, meaning that multiple continuously tracked target objects may exist simultaneously in the initial camera or other associated cameras. Therefore, a unique tracking ID is assigned to each target object to accurately distinguish and identify different target objects in multi-target scenarios, ensuring that the tracking trajectory of each target object can be accurately recorded and analyzed.

[0025] In one embodiment, based on the target selection instruction from the user terminal, the selected target is segmented in real time to generate a target mask corresponding to the selected target; the feature information of the selected target is extracted; and the target mask and the feature information corresponding to the selected target are cached as the target template feature.

[0026] When an object appears in the initial camera's field of view and is tracked for the first time, it can be identified as a target object requiring continuous tracking, based on tracking conditions or manual selection. At this point, appropriate image processing algorithms, such as object detection and feature extraction algorithms, are invoked to analyze the target object. Specifically, the object detection algorithm can quickly locate the target object's position within an image frame, such as by selecting a rectangular bounding box around the target object and using the displacement of the box's center point across consecutive image frames as the target object's trajectory. The feature extraction algorithm, on the other hand, extracts the target object's feature vector from the image data, forming a target template feature. This template feature is then used for target matching and recognition in the video streams of other associated cameras, enabling cross-camera target tracking.

[0027] For example, when a user selects a target object in a camera video frame using a point prompt or box prompt, a target selection instruction is generated. Based on this instruction, a lightweight, improved SAM2 algorithm on the TPU computing card can be loaded to perform real-time inference segmentation of the target based on the user-provided point or box prompt, generating a target mask. Global features (such as macroscopic attributes like color and shape) and local features (such as key areas like texture details) of the target are extracted from the mask region, and this feature information, along with the target location, is cached in a shared memory bank as target template features.

[0028] In terms of hardware deployment, it is based on the BM1684X TPU computing card and features multiple inference cores. It also includes a main control CPU or dedicated processing unit responsible for non-inference tasks such as memory bank management and multi-target tracking. The tpu-mlir toolchain is used to convert lightweight SAM2 models (ONNX format) into TPU-specific .bmodel files and perform INT8 quantization optimization. It supports multiple camera or video stream inputs and parallel decoding via hardware decoders.

[0029] In practical applications, the VPP dedicated processing unit preprocesses the decoded image by resizing and normalizing, and then transmits the processed data to the TPU. Each core of the TPU receives the preprocessed image frame and independently loads and runs a lightweight SAM2 model for image feature extraction and mask prediction. The TPU core only performs model inference calculations. After completing feature extraction and mask prediction, the TPU core transmits the extracted target features and related information to the Memory Bank. The Memory Bank is stored in the host machine's DDR memory, but its management and logical updates are handled by the CPU. The CPU processing unit receives the TPU's output in real time and updates the cache according to the Memory Bank management strategy. The CPU independently runs a multi-target tracking management module. This module receives segmentation results and features from the TPU cores, as well as information from the Memory Bank, performs trajectory association, re-identification, and state updates, and makes global tracking decisions.

[0030] In this embodiment, in order to improve the inference performance on the TPU, the core components of SAM2 are made lightweight.

[0031] Specifically, regarding the encoder, a Plain non-hierarchical Vision Transformer (ViT) architecture is adopted to replace the native multi-stage hierarchical ViT (HieraB+) of SAM2 as the image encoder. Furthermore, the number of layers in the Transformer encoder is reduced to 16 to decrease model complexity. For the attention mechanism, a 14×14 local window attention and four global attention blocks are combined to balance the capture of local and global information. The encoder output is simplified to a single-scale feature map with a resolution of 64x64 to reduce the computational cost of subsequent processing. In addition, INT8 quantization-aware training (QAT) is integrated during the model training phase to ensure that the model maintains high accuracy performance when performing INT8 quantization inference on the TPU, while fully utilizing the quantization acceleration capabilities of the TPU.

[0032] For the Prompt encoder, two Transformer layers were reduced to decrease computational complexity. For the mask decoder, the final segmentation mask was generated directly from the 64x64 single-scale feature map output by the lightweight encoder and the simplified cue embedding, avoiding complex upsampling paths and multi-scale feature fusion. Furthermore, when performing cross-attention computation on the TPU, cached features (keys and values) provided by the external CPU in the Memory Bank were utilized, and these keys and values ​​were subjected to a 2×2 coarse-grained spatial pooling operation to optimize computational efficiency. These improvements significantly enhanced the inference speed and efficiency of SAM2 on the TPU while maintaining model performance. In one embodiment, to maximize the utilization of TPU computing cores and inference performance, a multi-core TPU resource allocation scheme is adopted. The multi-camera multi-target tracking task is subdivided into subtasks such as video decoding, SAM2 model inference (feature extraction and mask prediction), memory bank management (feature update, re-identification query), and high-level tracking management.

[0033] like Figure 2 and Figure 3 As shown, each VPU of the TPU computing card executes encoding / decoding subtasks, while the TPU core executes inference tasks in SAM2, achieving specialized task processing. Memory Bank management and high-level tracing management tasks are handled by an external main control CPU.

[0034] Specifically, each TPU core runs its own lightweight SAM2 model instance, performing image feature extraction and mask prediction in parallel to ensure that inference tasks from different video streams do not interfere with each other, achieving maximum parallel computing. The main control CPU or dedicated processing unit is responsible for reading and writing to the Memory Bank, dynamic pruning, and target re-identification queries. The TPU core transmits the extracted features and generated masks to the CPU for Memory Bank management via a high-speed interconnect channel, ensuring the real-time performance and consistency of the Memory Bank. The main control CPU or dedicated processing unit runs high-level multi-target tracking logic, including trajectory management, cross-camera association, occlusion handling, and target removal. This unit receives re-identification results from the Memory Bank management unit and segmentation information from each TPU inference core to make global tracking decisions.

[0035] In this embodiment, the VPU handles encoding and decoding, the TPU focuses on model inference, and the CPU is responsible for high-level management and memory bank logic. This achieves reasonable allocation and efficient utilization of computing resources, improving the overall processing efficiency and throughput of the system.

[0036] S102. Decode image frames in parallel from at least one associated camera acquired by the initial camera to obtain image frame data corresponding to each associated camera. In one embodiment, when the target object cannot be continuously tracked in the initial camera video (the target leaves the current camera's field of view, is severely occluded, or the tracking confidence decreases, etc.), the TPU computing card utilizes multiple cores to process second video streams acquired by multiple associated cameras linked to the initial camera in parallel. The VPU core in the computing card is specifically responsible for parallel decoding of image frames from the second video streams from all associated cameras, ensuring that there are no bottlenecks in data input.

[0037] Specifically, based on the deployment of the initial camera and other cameras, multiple associated cameras that are geographically adjacent to or logically related to the initial camera are identified. Then, based on factors such as the target's movement direction and speed, several key associated cameras of interest are selected as most likely to capture the target. Multiple VPU cores in the TPU computing card are used to decode image frames in parallel from the second video streams acquired by the multiple associated cameras, obtaining the image frame data corresponding to each associated camera.

[0038] The decoded image frame data is stored in a data buffer, where it undergoes simple pre-screening, selectively transmitting keyframes to reduce redundancy in subsequent processing. The decoded image frame data from each associated camera is then passed to the corresponding image processing model (such as the lightweight SAM2 model) for feature extraction and mask prediction. The entire process is supported by the multi-core architecture of the TPU computing card, enabling efficient parallel processing.

[0039] In one embodiment, before deploying the image processing model, the model needs to be trained according to the actual application requirements. This specifically includes a pre-training stage, a video segmentation and tracking fine-tuning stage, and a target re-identification fine-tuning stage.

[0040] Specifically, in the pre-training phase, the model is pre-trained on a large-scale image segmentation dataset, specifically the SA-1B dataset. In the video segmentation and tracking fine-tuning phase, fine-tuning is performed on video object segmentation and multi-object tracking datasets, specifically the SA-V dataset, DAVIS, MOT17 / 20, and DanceTrack. These datasets provide video sequences with temporal information and multi-object annotations, which are crucial for improving the model's performance in video tracking tasks. In the object re-identification fine-tuning phase (taking pedestrians as the object), further fine-tuning is performed on pedestrian re-identification datasets, specifically Market1501, DukeMTMC-ReID, and MSMT17. These datasets provide a large number of pedestrian images, including different viewpoints, lighting conditions, and occlusion scenarios, used to train the model to learn robust pedestrian feature representations to support cross-camera re-identification.

[0041] During model training, loss functions need to be considered, including segmentation loss, feature embedding loss, and tracking loss. Specifically, segmentation loss mainly includes binary cross-entropy loss and Dice loss, used to supervise the model in generating high-quality target segmentation masks. Feature embedding loss uses triplet loss to learn discriminative target feature embeddings to optimize the re-identification capability of features in the memory bank. Triplet loss ensures that the distance between the anchor point and the positive sample is less than the distance between the anchor point and the negative sample, and has a certain margin, thus ensuring that the feature distance between different targets is large enough, and the feature distance of the same target in different frames or different camera views is small enough. Tracking loss refers to the introduction of correlation loss in multi-target tracking tasks, specifically a matching loss based on the Hungarian algorithm, to optimize the model's ability in target correlation and ID preservation.

[0042] A phased training strategy can be adopted during model training.

[0043] Specifically, in the first stage, basic segmentation and tracking pre-training is performed: pre-training and fine-tuning are carried out on large-scale image segmentation datasets (SA-1B) and video segmentation and tracking datasets (SA-V, DAVIS, MOT17 / 20, DanceTrack) to enable the model to have powerful general segmentation and video tracking capabilities.

[0044] In the second stage, re-identification feature learning can be performed: further fine-tuning can be done on pedestrian (or other target object) re-identification datasets (Market1501, DukeMTMC-ReID, MSMT17). In this stage, the focus is on optimizing the feature embedding loss to learn more discriminative target features, ensuring that the features of the same target are similar under different cameras and different poses, while the features of different targets are different.

[0045] In the third stage, joint fine-tuning of multiple tasks or fine-tuning before deployment is performed: joint fine-tuning is performed on a comprehensive dataset containing annotations for multiple tasks such as segmentation, tracking, and re-identification, or final fine-tuning is performed on a dataset for a specific deployment scenario to adapt to the actual application environment.

[0046] In one embodiment, during model training, the AdamW optimizer can be used to optimize the model, combined with a learning rate scheduling strategy, specifically cosine annealing or multi-step decay, to ensure the stability and convergence of the training process.

[0047] To achieve efficient INT8 inference on the TPU, quantization-aware training (QAT) is integrated into the model during the later stages of training. QAT simulates the low-precision inference process, adjusts the model weights, and minimizes the impact of quantization errors on model performance, thereby ensuring that the quantized model still maintains high accuracy.

[0048] The trained image processing model is deployed to a multi-camera, multi-target tracking application environment to achieve trajectory tracking of multiple cameras and multiple targets in real-world application scenarios. Furthermore, the image processing model can be retrained and optimized periodically and continuously during practical applications.

[0049] S103. Input the image frame data corresponding to each of the associated cameras into the image processing model corresponding to each of the associated cameras for feature extraction and mask prediction, and obtain the current image features of the current image frame of each associated camera. In one embodiment, the image frame data decoded by each associated camera is input in parallel into its respective image processing model (such as the lightweight SAM2 model), and feature extraction and mask prediction are performed on the image frame data corresponding to each associated camera.

[0050] Specifically, the decoded image frames from each associated camera are input in parallel into their respective image processing models, such as the lightweight SAM2 model. The image processing models then extract features from each image frame to obtain the overall image features, including color distribution and texture. Local features, such as shape and size, are extracted from each individual element in the image. Simultaneously, mask prediction is performed to identify target objects within the image.

[0051] For example, image frame data from associated cameras is standardized and adjusted to an input size suitable for the model. A lightweight SAM2 model encoder is used to extract global features from the image. The image is segmented into multiple patches, linearly embedded, and then a simplified Transformer encoding layer (16 layers) is used, combined with local window attention (14x14 windows) and a small number of global attention blocks (4), to extract overall semantic features of the image, such as color distribution and shape contours. Based on prompts (such as the target area clicked by the user), the model's decoder generates a segmentation mask for the target. Using this mask, the target's position in the image is accurately located, focusing on the target itself and reducing background interference. Further mining of local features of the target, such as texture details and key points, is also conducted; these features are highly valuable for target recognition in cross-camera scenes.

[0052] In this embodiment, by making lightweight improvements to SAM2 and using TPU multi-core parallel inference, the computational complexity of the model is significantly reduced, enabling the method to achieve real-time (low-latency) multi-camera multi-target tracking on edge devices (TPUs), meeting the needs of application scenarios such as real-time monitoring and industrial inspection.

[0053] S104. Perform feature comparison and matching between the current image features of the current image frame of each associated camera and the target template features, and output the detection result of the target object in the current image frame of each associated camera; Target template features are stored in a shared memory bank, such as a Memory Bank. The Memory Bank uses a fixed-size FIFO queue to store the unique ID, multi-level feature vectors, and the location information of the latest frame for each tracked target. When new image features are extracted, their cosine similarity with all existing historical features in the Memory Bank is calculated. Based on the similarity results, the four frames with the richest information and greatest differences are intelligently selected and retained, while redundant or overly similar features are discarded. This ensures that the Memory Bank always contains the most representative historical context information while effectively controlling memory usage.

[0054] In one embodiment, each SAM2 component running independently on the TPU core compares the current image features of the current image frame with the target template features cached in the Memory Bank. Through comparison matching, detection, and segmentation, it detects whether a target object exists in the current image frame of each associated camera.

[0055] For example, feature detection can be performed using feature similarity. If the feature similarity between the current image features of the current image frame and the target template features reaches a certain threshold, then the target object can be considered to exist in the current image frame; conversely, if the feature similarity between the current image features of the current image frame and the target template features is low, then the target object can be considered not to exist in the current image frame.

[0056] In one embodiment, the feature similarity between the current image features of the current image frame of each associated camera and the feature of the target template is calculated; when the feature similarity corresponding to any current image frame is greater than or equal to a preset similarity threshold, the detection result is determined to be that the target object exists in the current image frame.

[0057] In one embodiment, feature similarity can be calculated using methods such as cosine similarity, Euclidean distance, and Pearson correlation coefficient. Taking cosine similarity as an example, the formula for calculating feature similarity is:

[0058] Where F is the current image feature vector, and T is the target template feature vector. Indicates feature similarity.

[0059] A similarity threshold (e.g., 0.8) can be preset. If the feature similarity between the current image frame and the target template feature is greater than or equal to the similarity threshold, it is determined that the target object exists in the current image frame; otherwise, it is determined that the target object does not exist in the current image frame.

[0060] S105. When the detection result indicates that the target object exists in the current image frame of the associated camera, extract the target tracking information of the target object in the image frame data corresponding to the associated camera, and generate the target tracking trajectory of the target object.

[0061] In one embodiment, the target tracking information includes target feature information, mask information, and location information of the target object.

[0062] In one embodiment, the target feature information includes global features and local features. Global features include color distribution, shape contours, etc., while local features include texture details, key points, etc.

[0063] When a target object is detected, the target feature information, mask information, and location information of the corresponding target in the Memory Bank are updated to reflect the latest state of the target.

[0064] In one embodiment, based on the image processing model, target tracking information of the target object is extracted in each image frame of the image frame data of the associated camera; based on the mask information, target tracking boxes of the target object in each image frame are extracted; and based on the target center point of the target tracking boxes in at least two image frames, the target tracking trajectory of the target object is generated.

[0065] In one embodiment, when a new target is selected by the user for the first time or automatically detected and identified by the system, a unique tracking ID is assigned to it, and a new tracking trajectory is created to ensure the persistence of the target's identity. Target tracking information is stored in a shared memory bank. The memory bank uses a hash table structure, with the target ID as the key, and stores data structures containing the target's latest features, historical feature sequences, location information, and motion status. The data in the memory bank is stored in the TPU device's DDR memory, but its management logic and operations are executed by the main control CPU.

[0066] When tracking the trajectory of a target object, SAM2's powerful segmentation capabilities are utilized to directly generate the tracking box and target center point from the accurate segmentation mask, significantly reducing the reliance on traditional detection boxes and thus greatly improving tracking accuracy and robustness.

[0067] Specifically, such as Figure 4 As shown, an image processing model, such as a lightweight improved SAM2 model, is used to segment and mask the target object in the image frame, obtaining the mask information of the target object. Simultaneously, based on the mask information, the target feature information of the target object is extracted from the image frame data of the associated camera. This target feature information includes global features of the target object, such as color, shape, and texture, as well as local features, such as key points or corner details on the target object.

[0068] Based on the mask information, the position and shape of the target object in each image frame are determined, thereby extracting the target tracking bounding box. The mask information can accurately identify the region of the target in the image, and by calculating the bounding rectangle of the mask, the specific coordinates of the target tracking bounding box can be obtained.

[0069] The target center points are extracted from the target tracking bounding boxes in at least two consecutive image frames. These center points reflect the position of the target object at different points in time. By connecting these center points in chronological order, a target tracking trajectory can be generated, which visually demonstrates the target's movement path in the scene.

[0070] Generally, in order to improve the accuracy and smoothness of the trajectory, methods such as Kalman filtering can be combined to predict and correct the target's motion in order to cope with temporary occlusion of the target or loss of position information.

[0071] In one embodiment, the target tracking information further includes current position information. After extracting target tracking information for the target object in each image frame of the associated camera's image frame data based on the image processing model, the method further includes: predicting the next position information of the target object based on the current position information and the current motion information of the target object; querying the next position information based on the acquisition area range of each associated camera to determine the target associated camera corresponding to the next position information, so as to perform trajectory tracking of the target object in the second video stream corresponding to the target associated camera.

[0072] In one embodiment, the current position information (such as the coordinates of the target center point) and the current motion information (such as the direction of motion and speed) of the target object in the current image frame are obtained from the extracted target tracking information.

[0073] By combining the motion information of the target object, methods such as Kalman filtering are used to predict the future position of the target object, assisting in the localization of potentially associated cameras. By incorporating historical target features from the Memory Bank, seamless connection of target trajectories across cameras is achieved. When a target moves from one camera's field of view to another, existing tracking IDs can be quickly matched, maintaining tracking continuity.

[0074] Combining the current position and motion information of the target object, a prediction algorithm (such as Kalman filtering) is used to predict the target object's next position at the next moment. Based on the acquisition area of ​​each associated camera, the target associated camera corresponding to the predicted next position information is queried. Trajectory tracking of the target object continues in the video stream corresponding to the determined target associated camera.

[0075] Specifically, the image frame data acquired by the associated camera is processed, and a lightweight SAM2 model is used to extract target feature information (global and local features), mask information, and position information, etc., of the target object. The pixel-level position of the target object in the current image frame can be determined from the mask information, the coordinates of the target's center point can be calculated, and by combining the coordinates of the target's center point in several consecutive image frames, the current motion information of the target object, such as its direction of motion and velocity, can be calculated. The Kalman filter algorithm is used to model and predict the target's motion state, and based on the target's current position and velocity, the position of the target at the next moment is predicted.

[0076] The scene monitored by multiple cameras is divided into a grid of regions corresponding to the camera's acquisition area. Based on the predicted next location information, the corresponding target-associated camera is queried within the region grid. The predicted next location information indicates that the target object will appear within the acquisition area of ​​the target-associated camera at the next moment; that is, the target object may exist in the image frame acquired by the target-associated camera at the next moment. Therefore, within the video stream corresponding to the identified target-associated camera, image processing models can continue to be used to detect and track the target object, achieving cross-camera, continuous updates to the target object's tracking information and trajectory.

[0077] For example, suppose in a shopping mall surveillance scenario, the target is a pedestrian. The pedestrian walks out of the field of view of the initial camera and enters the field of view of an associated camera. The initial camera uses a lightweight SAM2 model to extract the pedestrian's feature information, mask information, and position information, obtaining the pedestrian's last position and motion information in the initial camera. Based on the pedestrian's direction and speed of movement, it predicts that the next position may be within the capture area of ​​associated camera A. At this point, the capture area of ​​associated camera A can be queried to confirm that the predicted next position is indeed within the coverage area of ​​associated camera A. Starting from the next moment, the pedestrian is tracked again in the video stream corresponding to associated camera A, the pedestrian's position information in associated camera A is extracted, and the tracking trajectory of the initial camera is continued, continuously updating the pedestrian's tracking trajectory.

[0078] In one embodiment, the target template features of the target object are stored in a shared memory bank. Similarly, after the target object is determined, the extracted target feature information (including global and local features) as well as the latest mask information and location information are sent to the shared memory bank in real time for caching and updating. This cached information in the memory bank serves as a shared context for cross-camera target re-identification and tracking.

[0079] In one embodiment, the target feature information is cached in the shared memory library, the target template features of the target object in the shared memory library are iteratively updated, and the latest extracted target feature information is used as the target template feature of the target object.

[0080] In one embodiment, the shared memory bank is key to achieving multi-camera target re-identification. The memory bank uses a fixed-size FIFO (First-In-First-Out) queue to store the unique ID of each tracked target object, target feature information (global features, local features), mask information, and the position information of the latest frame, etc.

[0081] To address the issue of redundant information stored in FIFO queues, a dynamic pruning strategy based on cosine similarity is introduced. When new image frame features are extracted by the TPU and transmitted to the CPU, this unit calculates the cosine similarity between the new feature and all existing historical features in the Memory Bank. To maintain the information diversity and non-redundancy of features in the Memory Bank, this unit intelligently selects and retains the four frames with the richest information (lowest cosine similarity to the new feature, indicating the greatest difference), and discards other redundant or excessively similar frames. This ensures that the Memory Bank always contains the most representative historical context information, effectively controlling memory usage.

[0082] In one embodiment, the feature updates in the Memory Bank are real-time. Each frame is dynamically adjusted according to the pruning strategy described above to ensure the timeliness and effectiveness of the content.

[0083] For example, a lightweight SAM2 model is used to perform real-time inference segmentation of the target object, generating a target mask. Global and local features of the target are extracted from the mask region as target feature information. The extracted target feature information is cached in a shared memory bank (such as a memory bank). The memory bank uses a fixed-size FIFO queue to store the unique ID of each tracked target, the latest mask information, target feature information, and the position information of the latest frame.

[0084] Once new image frame features are extracted, their feature similarity with all existing target feature templates in the Memory Bank is calculated. Based on the similarity calculation results, the four frames with the richest information and greatest differences are intelligently selected and retained, while redundant or overly similar features are discarded. If the similarity between the new image frame features and the target template features reaches a certain threshold, the current image frame is considered to contain a target object. The latest extracted target feature information is then used as the target template feature for the target object, and the corresponding record in the Memory Bank is updated. To maintain the information diversity and non-redundancy of features in the Memory Bank, dynamic pruning is performed periodically to ensure that the Memory Bank always contains the most representative historical context information.

[0085] In this embodiment, an optimized Memory Bank management strategy, including temporal feature alignment and multi-level feature fusion, is used to significantly improve the accuracy of cross-camera target re-identification and effectively solve the problem of not being able to continuously track targets that have moved or disappeared briefly between multiple cameras.

[0086] Understandably, in multi-camera monitoring scenarios, due to the complexity of the scene, even deploying multiple cameras may not completely cover every location within the scene. This is especially true for locations that are difficult to move or install, and for scenes with multiple moving objects, which may cause the target object to be occluded or leave the field of view, thus preventing continuous tracking. For a target object whose trajectory is being tracked, if it becomes undetectable due to occlusion or temporarily leaving the field of view during tracking, a time threshold can be set to determine whether to continue tracking the target object.

[0087] In one embodiment, if no target object is detected in either the first video stream corresponding to the initial camera or the second video stream corresponding to each of the associated cameras, and the duration reaches a first duration threshold, the tracking status of the target object is marked as a first state; if no target object is detected in either the first video stream corresponding to the initial camera or the second video stream corresponding to each of the associated cameras, and the duration reaches a second duration threshold, the tracking status of the target object is marked as a second state.

[0088] Wherein, the second duration threshold is greater than the first duration threshold.

[0089] In one embodiment, multiple tracking states can be defined, such as a first state of "pending confirmation" and a second state of "lost", or more tracking states can be defined as needed, such as recovery.

[0090] In one embodiment, when a target cannot be continuously detected due to occlusion or temporary removal from the field of view, its trajectory state is marked as "pending confirmation," and short-term memory is stored using features in the Memory Bank. By analyzing the relative positions, directions of movement, and occlusion relationships between different targets, the system assists in determining whether a target is occluded and how to re-identify it.

[0091] Specifically, the tracking time can be started from the time corresponding to the first image frame in which the target object cannot be detected, and the detection status of the target object in the initial camera and all associated cameras can be continuously monitored, recording the duration for which the target object is not detected. If the target object is not detected in the initial camera and all associated cameras, and the duration reaches a first duration threshold, the target tracking status is marked as a first state ("pending confirmation"); if the target object is not detected in the initial camera and all associated cameras, and the duration reaches a second duration threshold, the target tracking status is marked as a second state ("missing").

[0092] For target objects in the "pending confirmation" state, their feature information in the Memory Bank is retained, and detection and matching continue to be attempted in subsequent frames. Once the target object reappears within the second time threshold, it can be quickly reactivated through feature matching and location prediction mechanisms. For target objects in the "lost" state, they are removed from the tracking list, and their feature information is deleted from the Memory Bank to free up resources.

[0093] In one specific embodiment, assuming a traffic monitoring scenario, the target object is a car. When the car is first detected by the initial camera, its tracking state is initialized to "normal tracking," and the last detection time is recorded. When a car enters a blind spot (e.g., obscured by a building) and is not detected by the initial camera or any associated cameras, a timer begins recording the duration of the undetected car. If the duration of the undetected car reaches a first threshold (e.g., 5 seconds), the system updates the car's tracking status to "Pending Confirmation." If the duration further reaches a second threshold (e.g., 20 seconds), the system updates the car's tracking status to "Lost." For cars in the "Pending Confirmation" state, the system continues to attempt detection and matching in subsequent frames. Once the car is detected again, its tracking status is restored to "Normal Tracking." For cars in the "Lost" state, the system removes them from the tracking list and deletes their feature information from the Memory Bank, freeing up resources for tracking other targets.

[0094] In this embodiment, by combining SAM2's segmentation capabilities, more accurate trajectory updates and correlations are achieved. Simultaneously, intelligent occlusion handling and user interaction mechanisms improve the robustness and user experience of the multi-target tracking system.

[0095] This embodiment provides a multi-camera multi-target tracking method. In multi-camera multi-target tracking scenarios, this method addresses the issue that the target object cannot be continuously tracked in the first video stream acquired by the initial camera. Parallel decoding of the second video stream from associated cameras accelerates the acquisition of image frame data, saves decoding waiting time, and improves image data processing efficiency. Inputting the image frame data into the image processing models corresponding to each associated camera for feature extraction and mask prediction allows for the simultaneous acquisition of image frame feature information from each associated camera, achieving efficient target recognition and localization. Comparing and matching the current image features with target template features quickly filters out image frames containing the target object, improving recognition efficiency. Extracting target tracking information and generating a target tracking trajectory from the associated camera image frame data where the target object is detected enables continuous cross-camera tracking of the target object in multi-camera scenarios, ensuring the integrity and accuracy of the trajectory and effectively improving the tracking efficiency and continuity of multi-camera multi-target tracking.

[0096] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a first embodiment of a multi-camera multi-target tracking device provided in this application. The multi-camera multi-target tracking device is used to perform the aforementioned multi-camera multi-target tracking method.

[0097] like Figure 5 As shown, the multi-camera multi-target tracking device 200 includes: a template feature extraction module 201, an image frame decoding module 202, an image frame feature extraction module 203, a target object detection module 204, and a trajectory generation module 205.

[0098] The template feature extraction module 201 is used to extract the target template features of each target object in the image frames of the first video stream when the target object cannot be continuously tracked in the first video stream initially captured by the camera. The image frame decoding module 202 is used to decode image frames in parallel from at least one associated camera acquired by the initial camera in a second video stream to obtain image frame data corresponding to each associated camera. The image frame feature extraction module 203 is used to input the image frame data corresponding to each of the associated cameras into the image processing model corresponding to each of the associated cameras for feature extraction and mask prediction, so as to obtain the current image features of the current image frame of each associated camera. The target object detection module 204 is used to compare and match the current image features of the current image frame of each associated camera with the target template features, and output the detection result of the target object in the current image frame of each associated camera; The trajectory generation module 205 is used to extract the target tracking information of the target object in the image frame data corresponding to the associated camera when the detection result indicates that the target object exists in the current image frame of the associated camera, and generate the target tracking trajectory of the target object.

[0099] In one embodiment, the target tracking information includes target feature information and mask information of the target object; the trajectory generation module 205 includes: The target tracking information extraction unit is used to extract the target tracking information of the target object in each image frame of the image frame data of the associated camera based on the image processing model. The target tracking box extraction unit is used to extract the target tracking box of the target object in each image frame based on the mask information; The tracking trajectory generation unit is used to generate the target tracking trajectory of the target object based on the target center point of the target tracking box in at least two image frames.

[0100] In one embodiment, the target tracking information further includes current location information; the trajectory generation module 205 further includes: The next position prediction unit is used to predict the next position information of the target object based on the current position information and the current motion information of the target object; The target associated camera determination unit is used to query the next location information based on the acquisition area range of each associated camera, determine the target associated camera corresponding to the next location information, and perform trajectory tracking of the target object in the second video stream corresponding to the target associated camera.

[0101] In one embodiment, the target template features of the target object are stored in a shared memory library; the trajectory generation module 205 further includes: The target template feature update unit is used to cache the target feature information in the shared memory library, iteratively update the target template features of the target object in the shared memory library, and use the latest extracted target feature information as the target template feature of the target object.

[0102] In one embodiment, the target object detection module 204 includes: The feature similarity calculation unit is used to calculate the feature similarity between the current image features of the current image frame of each associated camera and the feature of the target template. The target object detection unit is used to determine that the detection result is that the target object exists in the current image frame when the feature similarity corresponding to any current image frame is greater than or equal to a preset similarity threshold.

[0103] In one embodiment, the multi-camera multi-target tracking device 200 further includes a tracking status marking module, comprising: The first state marking unit is used to mark the tracking state of the target object as a first state if no target object is detected in the first video stream corresponding to the initial camera and the second video stream corresponding to each of the associated cameras, and the duration reaches a first duration threshold. The second state marking unit is used to mark the tracking state of the target object as the second state if no target object is detected in the first video stream corresponding to the initial camera and the second video stream corresponding to each of the associated cameras, and the duration reaches the second duration threshold. Wherein, the second duration threshold is greater than the first duration threshold.

[0104] In one embodiment, the template feature extraction module 201 includes: The target mask generation unit is used to perform real-time inference segmentation on the selected target according to the target selection instruction from the user terminal, and generate the target mask corresponding to the selected target; The feature information extraction unit is used to extract the feature information of the selected target; The target template feature determination unit is used to cache the target mask and feature information corresponding to the selected target as the target template feature.

[0105] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device and each module described above can be referred to the corresponding process in the aforementioned multi-camera multi-target tracking method embodiments, and will not be repeated here.

[0106] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0107] Please see Figure 6 , Figure 6 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0108] See Figure 6 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0109] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any multi-camera, multi-target tracking method.

[0110] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0111] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by the processor, the processor can perform any multi-camera multi-target tracking method.

[0112] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0113] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0114] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: When the target object cannot be continuously tracked in the first video stream initially captured by the camera, the target template feature of each target object is extracted from the image frames of the first video stream. Image frames are decoded in parallel from a second video stream acquired by at least one associated camera that is associated with the initial camera to obtain image frame data corresponding to each associated camera; The image frame data corresponding to each of the associated cameras is input into the image processing model corresponding to each associated camera for feature extraction and mask prediction, so as to obtain the current image features of the current image frame of each associated camera. The current image features of the current image frame of each associated camera are compared and matched with the target template features, and the detection result of the target object in the current image frame of each associated camera is output. When the detection result indicates that the target object exists in the current image frame of the associated camera, the target tracking information of the target object in the image frame data corresponding to the associated camera is extracted, and the target tracking trajectory of the target object is generated.

[0115] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the multi-camera multi-target tracking methods provided in the embodiments of this application.

[0116] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0117] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-camera, multi-target tracking method, characterized in that, The method includes: When the target object cannot be continuously tracked in the first video stream initially captured by the camera, the target template feature of each target object is extracted from the image frames of the first video stream. Image frames are decoded in parallel from a second video stream acquired by at least one associated camera that is associated with the initial camera to obtain image frame data corresponding to each associated camera; The image frame data corresponding to each of the associated cameras is input into the image processing model corresponding to each associated camera for feature extraction and mask prediction, so as to obtain the current image features of the current image frame of each associated camera; The current image features of the current image frame of each associated camera are compared and matched with the target template features, and the detection result of the target object in the current image frame of each associated camera is output. When the detection result indicates that the target object exists in the current image frame of the associated camera, the target tracking information of the target object in the image frame data corresponding to the associated camera is extracted, and the target tracking trajectory of the target object is generated.

2. The multi-camera multi-target tracking method according to claim 1, characterized in that, The target tracking information includes the target object's feature information and mask information; The step of extracting target tracking information of the target object from the image frame data corresponding to the associated camera and generating the target tracking trajectory of the target object includes: Based on the image processing model, target tracking information of the target object is extracted from each image frame in the image frame data of the associated camera; Based on the mask information, extract the target tracking box of the target object in each image frame; The target tracking trajectory of the target object is generated based on the target center point of the target tracking box in at least two image frames.

3. The multi-camera multi-target tracking method according to claim 2, characterized in that, The target tracking information also includes current location information; After extracting the target tracking information of the target object in each image frame of the associated camera's image frame data based on the image processing model, the process further includes: Based on the current location information and the current motion information of the target object, predict the next location information of the target object; Based on the acquisition area range of each associated camera, the next location information is queried to determine the target associated camera corresponding to the next location information, so as to perform trajectory tracking of the target object in the second video stream corresponding to the target associated camera.

4. The multi-camera multi-target tracking method according to claim 2, characterized in that, The target template features of the target object are stored in a shared memory library; After extracting the target tracking information of the target object in each image frame of the associated camera's image frame data based on the image processing model, the process further includes: The target feature information is cached in the shared memory library, and the target template features of the target object in the shared memory library are iteratively updated. The latest extracted target feature information is used as the target template feature of the target object.

5. The multi-camera multi-target tracking method according to claim 1, characterized in that, The step of comparing and matching the current image features of the current image frame of each associated camera with the target template features, and outputting the detection result of the target object in the current image frame of each associated camera, includes: Calculate the feature similarity between the current image features of the current image frame of each associated camera and the feature of the target template; When the feature similarity corresponding to any current image frame is greater than or equal to a preset similarity threshold, the detection result is determined to be that the target object exists in the current image frame.

6. The multi-camera multi-target tracking method according to claim 1, characterized in that, When the detection result indicates that the target object exists in the current image frame of the associated camera, after extracting the target tracking information of the target object from the image frame data corresponding to the associated camera and generating the target tracking trajectory of the target object, the method further includes: If no target object is detected in either the first video stream corresponding to the initial camera or the second video stream corresponding to each of the associated cameras, and the duration reaches a first duration threshold, the tracking status of the target object is marked as the first state. If no target object is detected in either the first video stream corresponding to the initial camera or the second video stream corresponding to each of the associated cameras, and the duration reaches the second duration threshold, the tracking status of the target object is marked as the second state. Wherein, the second duration threshold is greater than the first duration threshold.

7. The multi-camera multi-target tracking method according to claim 1, characterized in that, Extracting the target template features of each target object from the image frames of the first video stream includes: Based on the target selection command from the user terminal, the selected target is subjected to real-time inference and segmentation to generate a target mask corresponding to the selected target. Extract the feature information of the selected target; The target mask and feature information corresponding to the selected target are cached as the target template feature.

8. A multi-camera, multi-target tracking device, characterized in that, The multi-camera multi-target tracking device includes: The template feature extraction module is used to extract the target template features of each target object in the image frames of the first video stream when the target object cannot be continuously tracked in the first video stream initially captured by the camera. An image frame decoding module is used to decode image frames in parallel from a second video stream acquired by at least one associated camera associated with the initial camera, to obtain image frame data corresponding to each associated camera; The image frame feature extraction module is used to input the image frame data corresponding to each of the associated cameras into the image processing model corresponding to each associated camera for feature extraction and mask prediction, so as to obtain the current image features of the current image frame of each associated camera. The target object detection module is used to compare and match the current image features of the current image frame of each associated camera with the target template features, and output the detection result of the target object in the current image frame of each associated camera; The trajectory generation module is used to extract the target tracking information of the target object in the image frame data corresponding to the associated camera when the detection result indicates that the target object exists in the current image frame of the associated camera, and generate the target tracking trajectory of the target object.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the multi-camera multi-target tracking method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the multi-camera multi-target tracking method as described in any one of claims 1 to 7.

Images