Target object tracking methods, systems, storage media, and application products

By employing a multi-camera collaborative visual perception architecture and a two-way mask propagation mechanism, the problem of identity switching and trajectory breakage caused by occlusion, rapid movement, or viewpoint switching in target tracking is solved. This enables accurate target identity recovery and improved 3D spatial positioning accuracy, making it suitable for fields such as intelligent monitoring and autonomous driving.

CN123118702APending Publication Date: 2026-11-17WENZHOU UNIVERSITY ARTIFICIAL INTELLIGENCE & ADVANCED MANUFACTURING INSTITUTE (YONGJIA)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611461107.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing target tracking methods are prone to identity switching or trajectory breakage after the target is occluded, moves rapidly, or the viewpoint changes. Furthermore, re-identification methods based on appearance features suffer from decreased reliability when there are differences in viewpoints from multiple cameras and changes in lighting. They also have high computational overhead and are difficult to apply in resource-constrained embedded platforms or scenarios with high real-time requirements.

Method used

A multi-camera collaborative visual perception architecture is adopted, and a bidirectional complementary verification mechanism based on forward and backward mask propagation is introduced. Combined with 3D geometric fusion and global filtering, the target's accurate identity recovery and stability maintenance are achieved through real-time calibration verification and self-calibration mechanism of depth information.

Benefits of technology

It effectively avoids identity switching and trajectory breakage caused by occlusion, rapid movement or viewpoint switching in target tracking, improves the continuity and robustness of target tracking, reduces the dependence on frame-by-frame detection, and improves the accuracy of 3D spatial positioning and the intelligence level of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118702A_ABST
    Figure CN123118702A_ABST
Patent Text Reader

Abstract

This disclosure provides a method, system, storage medium, and program product for tracking a target object. The method includes: obtaining at least two image sequences simultaneously captured by at least two cameras located at different angles, wherein the target object includes a first target object and a new target object, and each of the at least two image sequences includes a first image frame corresponding to the first target object and a current image frame corresponding to the new target object; based on the at least two image sequences, obtaining first mask information of the first target object at the time of the first image frame and third mask information of the new target object at the time of the current image frame; based on the first mask information, predicting second mask information of the first target object at the time of the current image frame with the first image frame as the start time; and determining whether the new target object matches the first target object at each camera angle based on the similarity between the second mask information and the third mask information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of image technology, and in particular to a method, system, storage medium, and program product for tracking a target object. Background Technology

[0002] In applications such as intelligent monitoring, autonomous driving, and human-computer interaction, continuous and accurate tracking of moving targets is one of the core technological requirements.

[0003] Existing detection-based tracking methods rely excessively on frame-by-frame detection results. When a target is briefly lost due to occlusion, rapid movement, sudden changes in lighting, or a change in viewpoint, the system often marks the target as "lost." When the target reappears, it is difficult to correctly associate the new detection results with the historical trajectory, leading to identity switching or trajectory breakage, which seriously affects the continuity and robustness of tracking.

[0004] On the other hand, re-identification methods based on appearance features suffer a significant drop in reliability when faced with differences in multi-camera perspectives, changes in lighting, and target deformation. Moreover, these methods typically incur high computational costs, making them difficult to apply in resource-constrained embedded platforms or scenarios with high real-time requirements.

[0005] With the development of deep learning technology, especially the emergence of memory-based mask propagation models, new possibilities have been provided for tracking using the shape and spatiotemporal continuity of targets. However, these models have not yet been systematically applied to loss recovery and identity re-identification in multi-target tracking, and there is a lack of a two-way verification mechanism that can fully utilize their spatiotemporal reasoning capabilities. Summary of the Invention

[0006] The purpose of this disclosure is to provide a target object tracking method, system, storage medium, and program product that overcomes the shortcomings of existing target tracking methods, such as relying on frame-by-frame detection after the target is occluded, moves rapidly, or the viewpoint changes, leading to identity switching or trajectory breakage, and lacking an efficient identity recovery mechanism.

[0007] In a first aspect, this disclosure provides a method for tracking a target object, the method comprising:

[0008] At least two image sequences are obtained by simultaneously capturing images from at least two cameras located at different angles, wherein the target object includes a first target object and a new target object, and each image sequence in the at least two image sequences includes a first image frame corresponding to the first target object and a current image frame corresponding to the new target object;

[0009] Based on the at least two image sequences, obtain the first mask information of the first target object at the time of the first image frame and the third mask information of the new target object at the time of the current image frame;

[0010] Based on the first mask information, predict the second mask information of the first target object at the current image frame as the end time, with the first image frame as the start time.

[0011] Based on the similarity between the second mask information and the third mask information, it is determined whether the new target object matches the first target object from at least one camera angle.

[0012] Optionally, the target object tracking method further includes:

[0013] Based on the third mask information, the fourth mask information of the new target object is predicted with the current image frame as the start time and the first image frame as the end time.

[0014] Based on the similarity between the first mask information and the fourth mask information, it is determined whether the new target object matches the first target object at each camera angle.

[0015] Optionally, the prediction step employs a memory-based mask propagation model, wherein the mask propagation model includes at least one of XMem and Cutie.

[0016] Optionally, the step of determining whether the new target object matches the first target object at at least one camera angle includes:

[0017] Obtain the similarity between the second mask information and the third mask information at each camera angle, and use it as the first similarity at each camera angle;

[0018] In response to the first similarity of a first number of camera angles in the at least two cameras exceeding the single-camera matching determination threshold, and the first similarity of a second number of camera angles not exceeding the single-camera matching determination threshold, the first similarity under each camera angle is weighted and fused to obtain a fusion confidence score.

[0019] Based on the comparison result between the fusion confidence score and the global fusion determination threshold, it is determined whether the new target object matches the first target object.

[0020] Optionally, the weighted fusion of the first similarity at each camera angle includes:

[0021] For any camera angle, the weight of the camera angle is determined based on the magnitude of the first similarity at that camera angle, wherein the higher the first similarity, the greater the weight.

[0022] Based on the weights, the first similarity at each camera angle is weighted and summed to obtain the fusion confidence score.

[0023] Optionally, the target object tracking method further includes:

[0024] Based on the at least two image sequences, detect the new target object and obtain the third category, third two-dimensional bounding box and third mask information of the new target object;

[0025] Determine a third 3D bounding box of the new target object at each camera angle, the third 3D bounding box being located in the unified world coordinates of the at least two cameras;

[0026] Based on the intersection-union ratio of the first 3D bounding box and the third 3D bounding box of the first target object, as well as the categories of the first target object and the new target object, it is determined whether the new target object matches the first target object.

[0027] Optionally, the method for tracking the target object includes:

[0028] Based on the at least two image sequences, the first target object is detected and the first object category, first two-dimensional bounding box and first mask information of the first target object are obtained;

[0029] Obtain the depth map of the first target object;

[0030] Based on the depth map and the first two-dimensional bounding box, the local three-dimensional bounding box of the first target object at each camera angle is determined, and the local three-dimensional bounding box is located in the local coordinates of the camera.

[0031] Based on the local 3D bounding box, update the prediction speed and prediction local 3D bounding box of the first target object at the first image frame time obtained by Kalman filtering.

[0032] Optionally, the method for tracking the target object includes:

[0033] Based on the local 3D bounding box, a first 3D bounding box of the first target object is determined at each camera angle, and the first 3D bounding box is located in the unified world coordinates of the at least two cameras;

[0034] Based on the cross-camera intersection-over-union ratio of the first 3D bounding box, it is determined whether the first target object from different camera angles matches.

[0035] Optionally, the method for tracking the target object includes:

[0036] Using the cross-camera intersection-union ratio as a weight, the first three-dimensional bounding box of the first target object at each camera angle is fused to form a fused three-dimensional bounding box.

[0037] Optionally, the method for tracking the target object includes:

[0038] The target object includes a second target object, and each of the at least two image sequences includes a first image frame corresponding to the first target object and a second image frame corresponding to the second target object;

[0039] The method includes:

[0040] Based on the fused 3D bounding box, Kalman filtering is used to predict the velocity of the first target object and the predicted 3D bounding box of the first target object at the second image frame time.

[0041] Determine a second 3D bounding box of the second target object at each camera angle, wherein the second 3D bounding box is located in the unified world coordinates of the at least two cameras;

[0042] Based on the intersection-union ratio of the predicted 3D bounding box and the second 3D bounding box, it is determined whether the first target object matches the second target object.

[0043] Optionally, determining the first 3D bounding box of the first target object at each camera angle includes:

[0044] Obtain the depth map of the first target object and the camera intrinsic parameters of the at least two cameras;

[0045] Based on the depth map and the camera intrinsic parameters, the pixels in the first mask information are back-projected onto the camera coordinate system to generate a three-dimensional point set, so as to construct the local three-dimensional bounding box of the first target object;

[0046] Obtain the camera extrinsic parameters of the at least two cameras;

[0047] Based on the camera extrinsic parameters, the local 3D bounding box is transformed into a unified world coordinate system to obtain the first 3D bounding box.

[0048] Optionally, obtaining the camera extrinsic parameters of the at least two cameras includes:

[0049] Images of the same hand gesture are simultaneously acquired by at least two cameras;

[0050] Using a deep learning-based hand joint detection network, the three-dimensional coordinates of each hand joint in each camera coordinate system are calculated;

[0051] Match the 3D coordinate point pairs of the same hand joint in different cameras to construct the corresponding point set;

[0052] Based on the corresponding point set, the first rotation matrix and the first translation vector between the camera pairs are solved by singular value decomposition and used as the first camera extrinsic parameters.

[0053] Based on the reference camera, the initial camera extrinsic parameters are unified to the unified world coordinates to obtain the first unified camera extrinsic parameters.

[0054] Optionally, obtaining the camera extrinsic parameters of the at least two cameras further includes:

[0055] The random sampling consensus algorithm is used to remove outlier point pairs in the corresponding point set, and the remaining point pairs are used to resolve the first rotation matrix and the first translation vector to obtain the second camera extrinsic parameters. The second camera extrinsic parameters include the second rotation matrix and the second translation vector.

[0056] Based on the reference camera, the second camera extrinsic parameters are unified to the unified world coordinates to obtain the second unified camera extrinsic parameters.

[0057] Optionally, obtaining the camera extrinsic parameters of the at least two cameras further includes:

[0058] Using the three-dimensional joint coordinates of different hand poses and positions in multiple frames, the second rotation matrix and the second translation vector are iteratively optimized to obtain the third camera extrinsic parameters, which include the third rotation matrix and the third translation vector.

[0059] Based on the reference camera, the extrinsic parameters of the third camera are unified to the unified world coordinates to obtain the third unified camera extrinsic parameters.

[0060] Optionally, the method for tracking the target object includes:

[0061] Based on the depth map, at least one of the following is performed to verify the accuracy of the real-time calibration of the at least two cameras:

[0062] Determine whether the cross-camera volume deviation value is within a preset first threshold, wherein the cross-camera volume deviation value is the difference between the volume of the first 3D bounding box at one angle among the different angles and the volume of the first 3D bounding box at another angle among the different angles; and,

[0063] Determine whether the cross-camera position deviation value is within a preset second threshold, wherein the cross-camera position deviation value is the difference between the center position of the first 3D bounding box at one of the different angles and the center position of the first 3D bounding box at another of the different angles; and,

[0064] Determine whether the ratio of the cross-camera projection point exceeds a preset third threshold, wherein the cross-camera projection point is the projection of the first three-dimensional bounding box point of one of the different angles onto the first three-dimensional bounding box of another of the different angles.

[0065] A second aspect of this disclosure provides a target object tracking system, the system comprising:

[0066] A processor for executing the target object tracking system described in the first aspect above.

[0067] Optionally, the system further includes:

[0068] A memory, wherein the memory is used to store at least two image sequences obtained by the at least two cameras simultaneously;

[0069] The processor includes a central processing unit and a graphics processing unit;

[0070] The memory is shared by the central processing unit and the image processor.

[0071] Optionally, the system further includes:

[0072] At least two cameras, wherein the at least two cameras are used to simultaneously capture at least two image sequences;

[0073] The at least two cameras include at least one of a monocular RGB camera, a binocular RGB camera, and a time-of-flight camera.

[0074] A third aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect described above.

[0075] In a fourth aspect, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0076] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0077] The positive and progressive effects of this disclosure are as follows:

[0078] This disclosure addresses the issues of target identity loss, identity switching, or trajectory breakage in existing target tracking technologies due to occlusion, rapid movement, or viewpoint changes. It constructs a multi-camera collaborative visual perception architecture and introduces a bidirectional complementary verification mechanism based on forward and backward mask propagation to achieve accurate target identity recovery after detection interruption, effectively avoiding ID switching and trajectory breakage. Furthermore, it performs 3D geometric fusion and global filtering on multi-camera observations, improving the target's positioning accuracy and robustness in 3D space. Simultaneously, it introduces a real-time calibration verification and self-calibration mechanism based on depth information to maintain the system's stability and reliability during long-term operation. This method significantly reduces reliance on frame-by-frame detection, improves the continuity and intelligence of target tracking, and effectively reduces tracking failures and maintenance costs caused by target loss or identity confusion. Attached Figure Description

[0079] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort.

[0080] Figure 1 This is a first schematic diagram of a target object tracking system provided in some embodiments of this disclosure.

[0081] Figure 2 This is a second schematic diagram of a target object tracking system provided in some embodiments of this disclosure.

[0082] Figure 3 This is a schematic diagram illustrating the tracking state transition of a target object, provided in some embodiments of this disclosure.

[0083] Figure 4 This is a schematic diagram of a forward mask propagation method provided in some embodiments of this disclosure.

[0084] Figure 5 This is a flowchart illustrating a forward mask propagation method provided in some embodiments of this disclosure.

[0085] Figure 6 This is a schematic diagram of a forward mask propagation method provided in some embodiments of this disclosure.

[0086] Figure 7 This is a flowchart illustrating a backward mask propagation method provided in some embodiments of this disclosure.

[0087] Figure 8 This is a schematic diagram of a backward mask propagation method provided in some embodiments of this disclosure.

[0088] Figure 9 This is a flowchart illustrating method 620 of a target object tracking method provided in some embodiments of this disclosure.

[0089] Figure 10 This is a schematic diagram of a forward mask propagation method under multiple cameras, provided in some embodiments of this disclosure.

[0090] Figure 11 This is a flowchart illustrating a target object tracking method provided in some embodiments of this disclosure.

[0091] Figure 12 This is a flowchart illustrating a target object tracking method provided in some embodiments of this disclosure.

[0092] Figure 13 This is a flowchart illustrating a local 3D tracking method provided in some embodiments of this disclosure.

[0093] Figure 14 This is a flowchart illustrating a global 3D tracking method provided in some embodiments of this disclosure.

[0094] Figure 15 This is a flowchart illustrating a target object tracking method provided in some embodiments of this disclosure.

[0095] Figure 16 This is a first flowchart of method 800 in a target object tracking method provided in some embodiments of this disclosure.

[0096] Figure 17 This is a second flowchart of method 800 in a target object tracking method provided in some embodiments of this disclosure.

[0097] Figure 18 This is a schematic diagram of the third process of method 800 in a target object tracking method provided in some embodiments of this disclosure.

[0098] Figure 19 This is a schematic diagram of the fourth process of method 800 in a target object tracking method provided in some embodiments of this disclosure.

[0099] Figure 20 This is a flowchart illustrating a target object tracking method provided in some embodiments of this disclosure. Detailed Implementation

[0100] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0101] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the document does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0102] It should be understood that the terms “device,” “system,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0103] As illustrated herein, unless the context clearly indicates otherwise, the words “a,” “an,” “an,” and / or “the” do not specifically refer to the singular and may also include the plural. Generally speaking, the terms “comprising” and “including” only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0104] The definitions used herein, such as the terms “having,” “may have,” “comprising,” or “may include,” indicate the presence of the corresponding function, operation, element, etc., and do not limit the presence of one or more other functions, operations, elements, etc. Furthermore, it should be understood that the terms “comprising” or “having” as used herein indicate the presence of the features, figures, steps, operations, elements, components, or combinations thereof described in the specification, without excluding the presence or addition of one or more other features, figures, steps, operations, elements, components, or combinations thereof.

[0105] Flowcharts are used in this document to illustrate the operations performed by the system according to the embodiments herein. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0106] In some examples of this disclosure, the method steps described below can be implemented by a program or custom circuitry, or a combination of custom circuitry and a program. For example, the method steps can be executed by a combination of one or more GPUs, one or more CPUs, or any other technically feasible processor such as an ASIC chip. In some examples of this disclosure, one or more processors such as an ASIC chip may include memory. Furthermore, those skilled in the art will understand that the scope of this disclosure can include any system and apparatus capable of performing the method steps described below.

[0107] System Introduction

[0108] like Figure 1 As shown, some embodiments of this disclosure provide a system 100, which may include one or more processors 110. For example, processor 100 may be used to execute the methods provided in this disclosure to track a target object OBJ.

[0109] For example, processor 110 may include one or more of the following: central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), and application-specific integrated circuit (ASIC).

[0110] In some embodiments, processor 100 may execute computer program 120 to implement the target object tracking method provided in this disclosure. For example, computer program 120 may include one or more different computer program modules 122, 124, and 126, which, when executed by processor 100, may implement the target object tracking method provided in this disclosure.

[0111] In some embodiments, the system 100 may further include multiple cameras / vision sensors 130. For example, multiple cameras C1, C2, C3…CN placed at different angles form a camera array. The multiple cameras C1, C2, C3…CN can continuously acquire scene video streams S1, S2, S3…SN of the target object OBJ from different angles. Each video stream is a dynamic sequence composed of a series of consecutive image frames F1, F2, etc., arranged in chronological order, where each frame is an RGB image. These cameras / vision sensors 130 include, but are not limited to, monocular RGB cameras, binocular RGB cameras, or time-of-flight (ToF) cameras.

[0112] In some embodiments, multiple cameras 130 can be synchronized via a hardware synchronization signal SIG to ensure that they capture images of the target object OBJ at the same time, thereby obtaining a temporally aligned RGB image sequence and a depth image sequence S1, S2, S3…SN.

[0113] In some embodiments, system 100 may further include memory 150. For example, memory 150 may contain frame buffer 140.

[0114] Frame buffer

[0115] In some embodiments, real-time video streams captured by multiple synchronized cameras (supporting RGB, stereo, or ToF) can be stored in frame buffer 140. For example, frame buffer 140 can be updated each time a camera acquires a new image frame.

[0116] In some embodiments, the frame buffer 140 can be a dedicated area in computer memory (e.g., a shared memory region on a CPU / GPU) for the orderly storage and management of time-aligned RGB image sequences and depth image sequences S1, S2, S3…SN. The frame buffer 140 can serve as a high-performance data hub and bridge connecting the sensor 130 and the back-end processor 110.

[0117] In some embodiments, the capacity of the frame buffer 140 (i.e., the number of frames it can store) can be dynamically adjusted according to the camera's frame rate and the actual needs of the system. For example, when the camera is streaming at 30 FPS, if the buffer size is set to 60, the system 100 can access the complete historical sequence of 60 frames from the current time t back to t-59, which is equivalent to providing a visual context spanning 2 seconds.

[0118] In some embodiments, frame buffer 140 is created in a shared memory region, which allows multiple deep learning or processing modules to concurrently and simultaneously read data from it with extremely low latency, without time-consuming data copying or waiting. Processing image frames in an image sequence using frame buffer 140 enables high throughput and real-time parallel processing in the system. Figure 2 Another system 200 provided in some embodiments of this disclosure is shown. System 200 is similar to system 100 described above, for example, used to perform the methods provided in this disclosure to track a target object OBJ. In system 200, computer program modules may include, for example, a depth estimation module 210, a bidirectional mask propagation module 220, a target detection and segmentation module 230, a temporal mask matching module 240, a global 3D tracking module 250, a local 3D tracking module 260, a multi-camera calibration module 270, a calibration verification module 280, etc.

[0119] RGB image

[0120] Each camera captures light from a scene through its optical lens and image sensor (such as CMOS or CCD). The sensor surface is covered with an array of color filters, which allows each photosensitive unit to sense only the intensity of light in a specific color band (red, green, or blue). The sensor converts the received continuous light signal into an analog electrical signal, which is then quantized into a digital value by an analog-to-digital converter. Subsequently, the system performs de-mosaic processing on the missing color channel information for each pixel, thereby reconstructing the complete intensity values ​​of the three color channels (red, green, and blue) for each pixel.

[0121] In the video stream, the camera continuously performs the above acquisition and processing at a fixed frame rate (e.g., 30 frames per second), thereby generating a sequence of RGB images ordered by timestamps. Multiple cameras use a hardware synchronization signal SIG to ensure that their respective acquired image sequences are strictly aligned in time, forming synchronized multi-channel RGB video streams, providing a consistent time reference for subsequent multi-view collaborative processing.

[0122] Depth image

[0123] by Figure 2 Taking the depth estimation module 210 shown as an example, for each frame of RGB image acquired, the depth estimation module 210 can be called to generate the corresponding depth image. The depth image can be represented as a two-dimensional matrix with the same resolution as the RGB image. The value of each element in the matrix, i.e., the depth value, represents the distance (Z coordinate) of the corresponding 3D point in the scene from the camera optical center along the camera optical axis.

[0124] Depth images can be obtained through different methods, and the operating mode of the depth estimation module 210 depends on the camera selection. Depending on the depth calculation method, different types of cameras can be selected from cameras C1, C2, C3…CN. For a monocular RGB camera, the depth image is estimated in real time using a deep learning model (e.g., Depth Anything or Metric3D). This model processes RGB images in real time using feedforward inference, learning to infer scene geometry from color, texture, semantics, and perspective cues in a single image, thereby directly outputting an estimated depth map.

[0125] For binocular RGB cameras, stereo vision technology is used to calculate disparity and generate depth maps. This process first performs feature matching on the simultaneously acquired left and right images, finding corresponding points in the right image for pixels in the left image and calculating their horizontal disparity. Then, based on the principle of triangulation, using the known camera baseline distance and focal length, the disparity values ​​are converted into depth values ​​for each pixel, thereby calculating the depth map.

[0126] For ToF sensors, the time of flight of the light signal is directly calculated by analyzing the phase difference between the emitted light and the received reflected light, and then the absolute distance corresponding to each pixel is calculated to generate a depth map.

[0127] When the depth map originates from raw sensor data such as stereo vision computing or ToF sensors, its accuracy is limited by sensor noise, ambient light interference, or computational errors. To overcome this limitation and improve the accuracy and reliability of the depth map, in some embodiments, a deep learning model can be used to receive the raw depth map generated by the sensor and the corresponding RGB image as input. By learning scene priors and noise patterns, the model jointly optimizes and refines the raw depth map, such as filling depth holes, smoothing noise regions, sharpening object edges, and maintaining consistency with the semantic boundaries of the RGB image, ultimately outputting a higher-quality, more accurate optimized depth image. In some embodiments, this deep learning model can be included in the depth estimation module 210 or used as a separate program module.

[0128] Although multiple cameras use independent clocks, in applications requiring high precision, all cameras can achieve frame-level synchronization via hardware synchronization signals, ensuring they capture images at the same point in time. The resulting synchronized RGB color images and corresponding depth values ​​are then fed into the frame buffer described above. The frame buffer can be, for example, a circular queue located in CPU-GPU shared memory, and its size can be flexibly adjusted according to the camera frame rate and system requirements. For example, when the camera transmits images at a frame rate of 30 frames per second and the frame buffer size is set to 60, the historical image sequence from the current time t back to frame t-59 can be accessed. This design facilitates the bidirectional mask tracking described later in this disclosure, allowing access to historical frame information for time mask matching.

[0129] In some embodiments, the frame buffer is updated each time a new image is received, and because it is stored in a shared memory region, multiple deep learning modules can access the data therein simultaneously.

[0130] Object detection and instance segmentation

[0131] The object detection and segmentation module 230 can process each frame of the RGB image. In some embodiments, a novel object detection method based on template matching can be used, which can achieve real-time recognition without retraining the model for new target objects.

[0132] Specifically, the method first generates a template model based on the 3D model of the target to be detected: the 3D model is rendered into 2D images from N different camera angles, and a corresponding binary mask (i.e. instance segmentation mask) is generated for each angle. At the same time, a 3D point map of the corresponding mask can also be generated.

[0133] The selection of N camera angles can be adaptively determined based on the target shape: for symmetrical objects, at least six angles are included: front, back, left, right, top, and bottom; for irregular objects, N can be set to be greater than 100 angles; the angle distribution can be uniformly sampled on the sphere using Fibonacci spherical distribution, geodetic distribution, or spiral point distribution algorithms.

[0134] For each angle, a base model, such as the self-supervised learning model DINOv2, is used to extract the first feature vector from the 2D image and binary mask, and convert it into a first visual descriptor. When detecting new objects, the system acquires a 2D image of the new object in real time via a camera, and uses a segmentation model to segment the target instance from the image, obtaining the binary mask and 2D bounding box of the new object. Simultaneously, a depth map can be obtained through a depth estimation model or a depth camera to obtain a 3D point map.

[0135] Then, using the same base model, a second feature vector is extracted from the 2D image and binary mask of the new object, transforming it into a second visual descriptor. Next, the system calculates the similarity between the first and second visual descriptors for each angle in the template model: the similarity can include the color similarity of the 2D image and the contour similarity of the binary mask. The color similarity is calculated by averaging the normalized dot products of the two visual descriptors, and the contour similarity is also calculated by averaging the dot products of the mask descriptors.

[0136] If a 3D point map is used simultaneously, the intersection-union ratio (IoU) between the template model and the new object's 3D bounding box will also be calculated. The final similarity can be the average of color similarity, contour similarity, and 3D bounding box IoU. When the final similarity exceeds a preset threshold, the currently detected object is determined to be the target object, and a unique identifier for that object is generated to identify the same object across different camera angles and frames.

[0137] This process can be performed offline during the system import phase or online during real-time operation to identify new categories without retraining the detection model, greatly improving the system's flexibility and adaptability.

[0138] Local 3D tracking

[0139] The detected instance mask, along with its category label, 2D bounding box, and depth map, are simultaneously passed to the local 3D tracking module 260. The local 3D tracking module 260 is responsible for performing independent 3D tracking of the target in each camera coordinate system.

[0140] Specifically, for each detected instance mask, the system synchronously retrieves the corresponding depth map from the frame buffer and uses the camera's intrinsic parameter matrix to backproject each foreground pixel (u, v) and its depth value Z within the mask to the camera coordinate system, generating a 3D point set. By analyzing the minimum and maximum values ​​of all 3D points on the X, Y, and Z axes, an axial bounding box closely adhering to the target can be constructed as the local 3D bounding box of the target object.

[0141] The center coordinates of the 3D bounding box are used as observations. In some embodiments, to suppress single-frame depth noise and detection jitter, each camera can independently maintain a Kalman filter whose state vector contains 3D position and velocity. The filter predicts the target state of the current frame based on a uniform motion model, and then updates it using the currently observed 3D bounding box center as the observation, outputting smooth local 3D position and velocity estimates. When the target is lost from detection at a certain viewpoint due to brief occlusion, this local Kalman filter can still continuously provide predictions based on the motion model, enhancing the robustness of single-view tracking.

[0142] In some embodiments, based on multi-camera calibration and global 3D tracking, the independent local 3D observations of each camera can be aligned to the same world coordinate system, and the global positioning accuracy can be improved through multi-view data fusion.

[0143] Multi-camera calibration

[0144] In some embodiments of this disclosure, multi-camera calibration can be achieved using conventional methods, such as calibration boards or calibration patterns (e.g., April Tags, ChArUco tags, or QR codes), or by utilizing hand gestures for markerless automatic calibration. This method will be described in detail in later embodiments of this disclosure. This method eliminates the need for any physical calibration objects; multi-camera calibration can be completed simply by the user demonstrating hand gestures, greatly simplifying the system deployment process.

[0145] Global 3D tracking

[0146] The global 3D tracking module 250 can transform the local 3D bounding boxes obtained by each camera's independent tracking to a unified world coordinate system using camera extrinsic parameters, thus obtaining the 3D bounding box of the target object in world coordinates. Since multiple cameras may simultaneously observe the same target object, matching is performed by calculating the Intersection over Union (IoU) between the 3D bounding boxes reported by different cameras. If the IoU exceeds a preset threshold, these observations are considered to correspond to the same target object.

[0147] In some embodiments, for successfully matched multi-view observations, a weighted average method can be used for fusion. For example, the intersection-union ratio or object detection confidence of each observation can be used as weights to obtain a fused 3D bounding box that is more reliable than any single observation.

[0148] In some embodiments, the fused 3D bounding box is input into a global Kalman filter (Kalman filter) operating in the world coordinate system. The global Kalman filter predicts the target's state in the current frame based on the target's global state (position and velocity) in the previous frame, corrects it with the fused observations, and outputs the optimal global 3D position and velocity estimates, while also predicting the position and velocity in the next frame.

[0149] In some embodiments, this local-global dual-layer filtering and fusion mechanism can overcome the occlusion and depth estimation errors of a single camera, significantly improving the accuracy of 3D positioning and the smoothness of the trajectory.

[0150] Target Re-identification

[0151] In some embodiments, the target re-identification technology achieves accurate identity recovery after detection interruption, solving the problems of ID switching and trajectory breakage caused by brief loss in traditional detection-based tracking methods. It is based on a time mask matching module and a bidirectional mask tracking module, and maintains a global target state machine, a tracking target pool, and a lost object pool.

[0152] The system assigns a globally unique identifier to each tracked target and defines four main states: newly detected, active, temporarily lost, and permanently lost.

[0153] Bidirectional mask propagation

[0154] After detection is interrupted, the bidirectional mask propagation module passes the historical mask of the lost target and the newly detected candidate mask of the current frame to the bidirectional mask propagation module. This module uses a memory-based mask propagation model (such as XMem or Cutie) and can propagate the mask forward or backward along the time axis.

[0155] Specifically, when the target enters a temporarily lost state, the module uses the instance mask of the last frame before its loss as a reference frame, and combines it with the continuous RGB image sequence stored in the frame buffer to iterate frame by frame along the forward time axis to predict the target mask in the current frame and generate a forward prediction mask. If the forward matching fails, the module uses the newly detected candidate mask in the current frame as a reference frame and backtracks frame by frame along the reverse time axis to generate a backtracking mask in the historical frames.

[0156] The results from both propagation directions are sent to the time mask matching module for similarity determination. Through this bidirectional, complementary propagation strategy, the system can recover the identity of a target after it has undergone occlusion or deformation. Details are provided below. Figures 4-12 Detailed explanation.

[0157] Time mask matching

[0158] The temporal mask matching module receives the forward prediction mask or backward backtracking mask from the bidirectional mask propagation module, as well as the current frame detection mask from the target detection and segmentation module.

[0159] This module is responsible for calculating the similarity between the propagation mask and the detection mask. For example, the intersection-union ratio (IUGR) can be used as a metric. For each mask pair, the module treats it as a binary image and calculates the ratio of the number of pixels in the intersection region to the number of pixels in the union region, obtaining an IUGR value between 0 and 1.

[0160] This value serves as the matching score and is compared to a preset threshold (e.g., 0.5). If the score exceeds the threshold, a successful match is determined, and the matching result is used to update the target's tracking status. If a match fails, propagation in another direction is triggered, or the recovery process is terminated, depending on the propagation direction.

[0161] Calibration Verification

[0162] The accuracy of multi-camera calibration parameters is continuously monitored through real-time calibration verification to ensure the geometric accuracy of cross-view fusion. This verification method is based on a core observation: when the fields of view of two cameras overlap and simultaneously observe the same tracked target, the target's 3D geometric information should be consistent between the two cameras. Unlike traditional methods that require additional image acquisition or physical target calibration, this invention directly utilizes the real-time depth information of the actual tracked target to evaluate the effectiveness of the calibration. Details are provided below. Figure 19 Detailed description,

[0163] Tracking state transitions

[0164] In some embodiments, a tracking state transition and pooling management mechanism can be introduced in a multi-camera 3D tracking system to efficiently and robustly manage the entire lifecycle of each detected target. The tracking state is managed by assigning a globally unique identifier to each detected target. Each target can transition between states such as "tracked," "temporarily lost," or "permanently lost," and based on these transitions, the target is placed in different tracking state pools. Subsequently, the target is placed in different management pools according to its state, and the system decides whether to continue tracking, attempt to recover, or ultimately remove the target based on the state. Targets are managed within the system; tracking may be recovered, or the target may be permanently removed. This state management maximizes tracking performance and makes target re-identification easier.

[0165] Figure 3The diagram illustrates a target object tracking state transition as an example. For instance, a cylinder can represent a target object pool, two cylinders can represent the tracked target pool 310 and the lost target pool 320 respectively, and a circle can represent the target's state, which can be "tracked", "temporarily lost", or "permanently lost".

[0166] It should be noted that the term "lost" in this application refers to the target object being undetectable from all camera perspectives, rather than a detection failure from a single camera perspective. Specifically, when a target object is not detected from any camera angle within a preset number of consecutive frames, the system determines that the target object is in a lost state. This determination criterion differs from "lost" in a single-camera scene. In a single-camera scene, the target may be lost from one camera due to a viewpoint switch or occlusion, but other cameras may still be able to observe the target. In this case, the target is not truly "lost" and can still be tracked through cross-viewpoint fusion. Only when all cameras cannot detect the target does the system determine it as lost and trigger the subsequent loss recovery mechanism.

[0167] The example will be explained in detail below:

[0168] Tracking Target Pool 310: Stores objects currently being tracked by the system. Targets in the pool are active and continuously detected and tracked. Once tracking begins, a target becomes active and is added to the tracking target pool. The system tracks targets based on their 3D bounding box and velocity. If a target is not detected or tracking is interrupted, it transitions to a "temporarily lost" state.

[0169] Lost Target Pool 320: Temporarily stores targets whose tracking has been interrupted. If a target is not detected for a period of time after being lost from tracking, it will transition to a "temporarily lost" state 330. If it remains undetected for an even longer period, the target will be moved to the Lost Target Pool 320. If it remains in this state for too long, it will be marked as "permanently lost" and removed from the system 340. Targets in the Lost Target Pool 320 may still be re-identified if they are detected again. Once re-identification is successful, they will be transferred back to the Tracking Target Pool 310.

[0170] Newly detected state 350: This state represents the initial state when the system detects a new target and tracking is about to begin. The system will attempt to match this newly detected target with the target in the "temporarily lost" state 320. In some embodiments, this matching process may be divided into two steps:

[0171] Step 1: Matching based on 3D bounding boxes and velocity

[0172] Step 1 compares the new target's category, 3D bounding box, and the target's category and last known velocity in the "temporarily lost" state 330. If they are of the same category and their 3D bounding box intersection-union ratio exceeds a threshold, the targets are considered a match, and the target is directly transitioned to the active state 360. In some embodiments, Step 1 may be performed only for targets with very short loss times.

[0173] Step 2: If the target did not match in Step 1, it can be matched with targets in the lost target pool 320. This process uses a bidirectional mask-based tracking and recovery method for more accurate matching. The matching is made by comparing the temporal consistency of the target's mask, which is suitable for situations where the loss time is long or the viewpoint changes significantly. The detailed bidirectional mask-based tracking and recovery method will be described below.

[0174] Active 360 ​​State: In this state, the target is being tracked normally by the global 3D tracking module. Its position, velocity, and trajectory history are updated in real time, reflecting the result of multi-camera data fusion.

[0175] Temporarily Lost State 330: This state occurs when a target is temporarily lost but can still be tracked again. This means the target has not completely disappeared, but tracking has temporarily stopped. If a target remains in the "Temporarily Lost" state for 1-2 seconds or longer, it will be timed out and moved to the lost target pool. It can only return to an active state through the recovery process.

[0176] Permanently Lost Status: This status indicates that the target can no longer be tracked. This status occurs when the target permanently moves out of the camera's field of view or is completely obscured for an extended period.

[0177] Re-identification status 370: This status occurs when a lost target is detected again and confirmed to be the same entity as the historical target. After the system completes the verification, it updates the status to "Re-identified" and then immediately returns to the "Active" status.

[0178] Removal Status 340: This status indicates that the object has been completely removed from the tracking module. This can occur when the object no longer needs to be tracked or has been excluded from tracking.

[0179] Figure 4The image sequence S1, obtained by camera C1 at a certain angle, is shown. In image sequence S1, image frame 2 detects target object OBJ1, images 3-6 do not detect target object OBJ1, and image frame 7 detects a new target object OBJ2. The inventors found that current detection-based tracking methods rely on frame-by-frame detection results. When target object OBJ1 is temporarily lost due to occlusion, rapid movement, or sudden changes in lighting, the system often marks the target as "lost." When the new target object OBJ2 reappears, it is difficult to correctly associate it with the historical trajectory of target object OBJ1, leading to a change in target object identity or trajectory breakage. The inventors also found that this problem is more serious in multi-camera, multi-target object environments. For example, when the trajectories of target objects from multiple cameras overlap, due to differences in the perspectives of multiple cameras, lighting, and the similarity of target objects, target object recognition methods based on appearance features become unreliable and computationally expensive, making it difficult to meet real-time requirements.

[0180] The inventors discovered that, compared to volatile appearance features, the shape of a target object deforms more stably over time. Therefore, to track target objects, instead of relying on volatile appearance features, the inventors propose identity matching and association based on the spatiotemporal continuity of the target object's shape deformation, especially in multi-camera, multi-target object environments. For example, mask propagation-based techniques can be used to predict the target object's mask, enabling spatiotemporal tracking of the target object's shape deformation and achieving more accurate re-identification of lost target objects.

[0181] Predicting the mask of a target object can include using at least one of forward masking and backward masking. Furthermore, combining forward and backward masking to predict the target object's mask can form a bidirectional, complementary verification system, thereby more accurately recovering the target object's identity after detection interruption. In some embodiments, to track the fine-grained spatiotemporal correlation of target object shape deformation, a memory-based masking model, such as XMem or Cutie, can be used.

[0182] Example 1

[0183] Accordingly, some embodiments of this disclosure provide a target object tracking method 400. For example... Figure 5 As shown, the method 400 includes:

[0184] S410: Obtain at least two image sequences simultaneously captured by at least two cameras located at different angles.

[0185] like Figure 4As shown, the target object may include a first target object OBJ1 and a new target object OBJ2, and each of the at least two image sequences includes a first image frame 2 corresponding to the first target object and a current image frame 7 corresponding to the new target object.

[0186] For example, the first target object OBJ1 could be a target that was previously tracked by the system but was temporarily lost due to occlusion, rapid movement, or a change in viewpoint. The new target object OBJ2 could be a new target object that has been detected at the current moment but has not yet been assigned any historical identifier. Each image sequence can contain a first image frame 2 corresponding to the first target object OBJ1 and a current image frame 7 corresponding to the new target object OBJ2. For example, the first image frame 2 could be the moment when the first target object was last successfully detected, i.e., the last frame before it was lost, and the current image frame 7 could be the real-time frame 7 when the new target object OBJ2 is detected.

[0187] S420. Based on at least two image sequences, obtain the first mask information of the first target object at the time of the first image frame and the third mask information of the new target object at the time of the current image frame.

[0188] S430. Based on the first mask information, predict the second mask information of the first target object at the current image frame as the end time, with the first image frame as the start time.

[0189] S440. Based on the similarity between the second mask information and the third mask information, determine whether the new target object matches the first target object from at least one camera angle.

[0190] Method 400 employs a forward masking approach to track the target object, thereby determining whether the new target object OBJ2 matches the first target object OBJ1. To intuitively understand this timing relationship and the subsequent forward masking path, as follows... Figure 6 As shown, Figure 6 Eight squares are arranged sequentially to represent the first to eighth image frames of image sequence S1. For example, image sequence S1 can be acquired by camera C1 located at a certain angle. The square in the second frame is marked with the first mask information MSK1, indicating the moment when target OBJ1 was last successfully detected. The squares in frames 3, 4, 5, and 6 are all marked as "missed," indicating that target OBJ1 could not be detected during these four consecutive frames. The square in the seventh frame is marked with the third mask information MSK3, indicating the moment when the new target OBJ2 was re-detected.

[0191] Figure 6Above the image is a horizontal arrow pointing from frame 2 to frame 7, with "Forward Masking Propagation (FMP)" labeled above it. Above frame 7 is also labeled the second mask information, MSK2. This illustrates that FMP starts with the first mask information MSK1 in frame 2, passes through the lost frame interval from frame 3 to frame 6, predicts the second mask information MSK2 in frame 7 (i.e., the predicted mask MSK2), and then matches this predicted mask MSK2 with the third mask information MSK3 actually detected in frame 7.

[0192] Specifically, the first image frame corresponds to frame 2 in the figure, and the stored first mask information MSK1 can be obtained from this frame. The current image frame corresponds to frame 7 in the figure, and the stored third mask information MSK3 can be obtained from this frame. Frames 3 to 6 are a continuous loss interval, during which the target temporarily disappears from the camera's field of view, and no detection results are obtained.

[0193] Based on the aforementioned at least two image sequences, the system can further obtain the first mask information of the first target object at frame 2, and simultaneously obtain the third mask information of the new target object at frame 7. The first mask information is the last clear instance segmentation mask before the target is lost, and it is stored in the lost object pool as a historical reference. The third mask information is the instance mask of the currently detected new object that has not yet been associated with any identity.

[0194] After obtaining these two sets of mask information, the system determines whether bidirectional mask propagation is suitable. It should be noted that in some embodiments, bidirectional mask propagation is not always executed, but is triggered only when normal tracking based on Kalman filtering and multi-camera 3D matching cannot maintain object identity continuity, such as when detection fails for multiple consecutive frames, the target enters the lost object pool, or the identity of a newly detected object is uncertain. When normal tracking is effective, the system prioritizes using Kalman prediction and cross-frame matching to maintain identity continuity, and only enables mask propagation when normal tracking methods are insufficient.

[0195] After determining that mask propagation is required, the system initiates the forward mask propagation process. Specifically, the system can use the first mask information as the spatiotemporal anchor point, take the second frame as the starting time, and use a memory-based mask propagation model to predict the shape that the first target object should present at the end time of the seventh frame along the time axis, thereby generating the second mask information.

[0196] The second mask information is a predicted mask that carries the historical identity assumption of the first target object, representing its pixel-level outline and spatial location in frame 7 if the target had not been lost. To perform this forward prediction, the system needs to retrieve a continuous sequence of RGB images from frame 2 to frame 7 from the frame buffer, including the background images from frames 3, 4, 5, and 6 during the loss period. The frame buffer is a circular queue located in CPU-GPU shared memory, its size pre-configured according to the camera frame rate and system requirements; for example, storing enough frames at a frame rate of 30 frames per second to provide the historical window needed to span the loss interval.

[0197] Mask propagation models (such as XMem or Cutie) can initialize an external memory using the first mask information from frame 2, extracting multi-scale features such as the target's contour, texture, and spatial structure. Then, they process the images in frames 3, 4, 5, and 6 frame by frame. For each frame, a scaled dot product attention mechanism is used to calculate the fine-grained correlation between the current frame's image features and historical features in the memory, generating a predicted mask for that frame, and updating the memory with the newly generated mask features. This process allows the model to gradually adapt to possible pose changes and deformations that may occur during the target's loss. When processing reaches frame 7, the model outputs the final predicted mask, which is the second mask information. The arrow pointing from frame 2 to frame 7 above the hand-drawn illustration, along with the label "Forward Mask Propagation" next to the arrow, visually illustrates this process, while the separate label "Second Mask Information" above frame 7 explicitly indicates that the predicted mask is located at frame 7.

[0198] After obtaining the second and third mask information, the system can determine whether the new target object belongs to the same physical entity as the first target object. To do this, the system calculates the similarity between the second and third mask information, which is usually measured by the intersection-union ratio (IUGR).

[0199] The Intersection over Union (IoU) ratio is calculated by treating each mask as a binary image and calculating the ratio of the number of pixels in the intersection region to the number of pixels in the union region. The value ranges from 0 to 1, with a value closer to 1 indicating that the two masks are more consistent in spatial shape and position. The system can perform this matching calculation independently for each camera angle. If the IoU ratio of the second mask information to the third mask information exceeds a first preset threshold at at least one camera angle, this threshold can be configured according to the actual application, for example, it can be set to 0.5, but is not limited to this, then it is determined that the new target object and the first target object are successfully matched in that camera view.

[0200] Those skilled in the art will understand that, in some embodiments, image sequences S2 obtained from cameras at other angles, such as C2, are also processed as in S1. Mask matching does not need to be successful at all camera angles. When mask matching is successful at one or more camera angles, the system combines the multi-camera 3D matching results, such as 3D position consistency and 3D bounding box overlap, to make a final matching determination. If the final determination is a match, the new target object OBJ2 obtains the historical identity of the first target object OBJ1, and its tracking status is updated from temporarily lost to active, achieving accurate recovery of the lost target.

[0201] Taking forward mask propagation as an example, consider the following scenario: a flying bird is the tracking target. At time t-α, the last frame before the target is lost, the system has successfully detected the bird and saved its instance mask, i.e., the first mask information. Subsequently, the target enters the loss interval until time t+β, when the system detects a new bird instance mask, i.e., the third mask information.

[0202] At this point, the system uses the historical mask at time t-α as a reference frame. Utilizing the continuous image sequence from t-α to t+β stored in the frame buffer, it iterates frame-by-frame along the forward time direction using a memory-based mask propagation model to predict the mask at time t+β, i.e., the second mask information. This predicted mask carries the historical identity assumption of the lost target. Subsequently, the system calculates the intersection-union ratio (IU / U) between this predicted mask and the third mask information actually detected at time t+β. If the IU / U is greater than a first preset threshold, the bird detected at time t+β is determined to be the previously lost target, thus achieving identity recovery.

[0203] In this scheme, to address the detection interruption issue caused by occlusion, rapid movement, or viewpoint switching during target tracking, the current frame mask is predicted frame by frame along the time axis using the historical mask of the last frame before loss as a reference. This mask is then matched with the current detection mask via intersection-union (IoU) comparison. This overcomes the defects caused by frame-by-frame detection failures, such as target identity loss, ID switching, or trajectory breakage. It significantly improves the continuity of target tracking and identity recovery efficiency, reduces the computational overhead and system latency caused by frequent re-identification, and enhances the system's tracking robustness in short-term loss scenarios. This forward mask propagation strategy enables identity recovery in most simple scenarios, such as short-term occlusion and gentle movement, with extremely low latency, significantly reducing the overall computational overhead of the system.

[0204] In a feasible scheme, if the matching result of the forward mask propagation does not reach a preset first threshold, it means that relying solely on forward prediction of the historical mask along time cannot lock the target identity in the current frame. This situation usually occurs when the target undergoes a relatively drastic change in pose, deformation, or a sudden change in ambient lighting during the period of loss, resulting in an excessively low cross-union ratio between the predicted mask and the detection mask. In this case, a backward mask propagation method 600 is proposed.

[0205] Similar to forward propagation, backward mask propagation is also used as a supplementary verification path when normal tracking methods fail and forward propagation cannot achieve a successful match. This method uses a clear and reliable detection mask in the current frame as a reference to backtrack along the timeline to further confirm the identity.

[0206] Accordingly, some embodiments of this disclosure provide a target object tracking method 600. Method 600 uses a backmask propagation-based approach to track the target object, thereby determining whether a new target object OBJ2 matches a first target object OBJ1.

[0207] Specifically, such as Figure 7 As shown, method 600 includes:

[0208] S610. Based on the third mask information, use the current image frame as the start time to predict the fourth mask information of the new target object when the first image frame is the end time.

[0209] S620. Based on the similarity between the first mask information and the fourth mask information, determine whether the new target object matches the first target object from at least one camera angle.

[0210] In the foregoing Figure 6 In the diagram shown, if the cross-intersection ratio (CIP) between the second mask information generated from the second frame forward to the seventh frame and the third mask information actually detected in the seventh frame does not exceed a preset first threshold, then the forward matching fails.

[0211] At this point, the system will not directly mark the target as permanently lost, but will automatically trigger backward mask propagation as a complementary verification path. The core idea of ​​backward mask propagation is to use a clear and reliable detection mask in the current frame as a reference, backtrack along the timeline, predict the target's shape in historical frames, and then match it with the historical mask saved before the target was lost.

[0212] The following is combined Figure 8 This paper describes the method of backmask propagation.

[0213] like Figure 8As shown, the eight squares are arranged sequentially to represent frames 1 to 8 of image sequence S1. For example, image sequence S1 can be obtained by camera C1 located at a certain angle. Frame 2 is marked with the first mask information MSK1, which is the historical mask of the last frame before target OBJ1 was lost; frames 3 to 6 are marked as lost, representing a detection failure interval of four consecutive frames; frame 7 is marked with the third mask information MSK3, representing the instance mask of the new target OBJ2 being detected in the current frame.

[0214] and Figure 6 The difference is, Figure 8 The fourth mask information, MSK4, is marked directly above the second frame, and Figure 8 There is a horizontal arrow pointing from frame 7 to frame 2, with the backward mask propagation (BMP) labeled above the arrow. This design clearly expresses the logic of the backward mask propagation (BMP): taking the third mask information MSK3 of frame 7 as a reference, it iterates backward along the time axis across the lost intervals of frames 6, 5, 4, and 3, and finally generates the fourth mask information MSK4 of frame 2, i.e., the backtracking mask MSK4, by using a memory-based mask propagation model to iterate frame by frame in reverse.

[0215] Backward masking employs the same memory-based masking propagation model as forward propagation, including at least one of XMem and Cutie, but operating along the reverse time axis: the memory is initialized with the third mask information MSK3, each frame in the sequence is processed in reverse, and an attention mechanism is used to calculate the fine-grained correlation between the current frame's features and historical features in the memory, generating a backtracking prediction mask for each frame, and updating the memory with new features. When processing the first image frame, the model outputs the final prediction mask, i.e., the fourth mask information MSK4. This mask carries the identity assumption that if the currently detected target is a historically lost target, then it should have presented its form before being lost.

[0216] The system then calculates the fourth mask information MSK4 and the first mask information MSK1 stored in the lost object pool, which is... Figure 8 The similarity between the historical masks of the second frame is also measured using the intersection-union ratio (IUU). Figure 8 This means that the fourth mask information MSK4, which is traced back to the second frame, is used to calculate the pixel-level overlap with the first mask information MSK1 that was originally stored in the second frame.

[0217] If the crossover ratio exceeds the second preset threshold, which can also be configured according to the actual application, then it is determined that the new target object OBJ2 and the first target object OBJ1 are successfully matched at the current camera angle.

[0218] Those skilled in the art will understand that, in some embodiments, image sequences S2 obtained from cameras at other angles, such as C2, are also processed as in S1. Mask matching does not need to be successful at all camera angles. When mask matching is successful at one or more camera angles, the system combines the multi-camera 3D matching results, such as 3D position consistency and 3D bounding box overlap, to make a final matching determination. If the final determination is a match, the new target object OBJ2 obtains the historical identity of the first target object OBJ1, and its tracking status is updated from temporarily lost to active, achieving accurate recovery of the lost target.

[0219] Let's take a flying bird as an example to further illustrate the application of backward masking. Suppose that during the period when the target is lost, it undergoes a drastic change in attitude, such as turning around, causing the intersection-union ratio of the predicted mask and the detection mask to fall below a preset first threshold, resulting in a failed match in forward propagation.

[0220] At this point, the system triggers backward mask propagation: using the mask of the newly detected bird instance at time t+β, i.e. the third mask information, as the reference frame, and utilizing the continuous image sequence from t+β to t-α stored in the frame buffer, the system iterates frame by frame along the reverse time axis through a memory-based mask propagation model to predict the backward mask at time t-α, i.e. the fourth mask information.

[0221] The backtracking mask carries the assumption that if the currently detected target is the historically lost target, it should have presented its form before being lost. Subsequently, the system calculates the intersection-union ratio (IUR) between this backtracking mask and the historical mask saved at time t-α, i.e., the first mask information. Since both the backtracking mask and the historical mask correspond to the bird's side profile and have the same pose, the IUR usually exceeds a second preset threshold, thus determining a successful match and restoring the target's identity.

[0222] In this scheme, for complex deformation scenarios where forward propagation cannot match, such as turning or sudden changes in posture, even if forward propagation fails due to inaccurate prediction, the identity can still be verified in reverse by using the current clear observation as a reliable anchor point through backward mask propagation. This overcomes the defect in existing technologies where tracking cannot be resumed due to sudden changes in target posture or long-term occlusion, significantly improving the system's recovery success rate in complex scenarios. It forms a complementary closed loop with forward propagation, enhancing the system's tracking robustness and continuity in complex environments.

[0223] like Figure 9 As shown, the multi-camera mask fusion determination method 620 includes:

[0224] S621. Obtain the similarity between the second mask information and the third mask information under each camera angle, and use it as the first similarity under each camera angle;

[0225] S622. In response to the fact that the first similarity of a first number of camera angles in at least two cameras exceeds the single-camera matching judgment threshold, and the first similarity of a second number of camera angles does not exceed the single-camera matching judgment threshold, the first similarity under each camera angle is weighted and fused to obtain the fusion confidence.

[0226] S623. Based on the comparison result between the fusion confidence and the global fusion judgment threshold, determine whether the new target object matches the first target object.

[0227] In this application, "loss" of a target object means that the target cannot be detected from any camera viewpoint. When a target is undetectable from some camera viewpoints but still detectable from others, the system does not classify it as lost, but continues tracking through cross-view fusion. Only when the target is undetectable from all cameras does the system enter the loss recovery process. The mask propagation and fusion determination described below are all based on the premise that the target is in a lost state.

[0228] In practical multi-camera systems, different cameras may have different viewing angles and degrees of occlusion when observing the same target. For example... Figure 10 As shown, the image sequences S1, S2, S3...SN obtained by multiple cameras C1, C2, C3...CN are used as an example.

[0229] In the image sequence S1 obtained by camera C1, the second frame is labeled with the first mask information MSK1 (the historical mask of the last frame before target OBJ1 was lost), frames 3 to 6 are labeled as lost, and the seventh frame is labeled with the third mask information MSK3 (the instance mask of the new target OBJ2 detected in the current frame). The arrow above indicates that the forward mask propagation (FMP) extends from the second frame to the seventh frame, and the seventh frame is labeled with the second mask information MSK2. The system calculates the intersection-union ratio (IUU) of the second mask information MSK2 and the third mask information MSK3, obtaining a first similarity S1 = 0.8. This value exceeds the single-camera matching threshold of 0.5, indicating that the forward matching from the angle of camera C1 is successful.

[0230] Similarly, in the image sequence S3 obtained by camera C3, the second frame is labeled with the first mask information MSK11 (the historical mask of the last frame before target OBJ1 was lost), frames 3 to 6 are labeled with the lost OBJ1, and the seventh frame is labeled with the third mask information MSK31 (the instance mask of the new target OBJ2 detected in the current frame). The arrow above indicates that the forward mask propagation (FMP) extends from the second frame to the seventh frame, and the seventh frame is labeled with the second mask information MSK21. The system performs the same forward mask propagation process and calculates the first similarity S3 = 0.7, which also exceeds the single-camera matching threshold of 0.5, indicating that the forward matching at the angle of camera C3 is successful.

[0231] For the image sequence S2 obtained by camera C2, since the target is severely occluded or in the detection blind zone from the camera's perspective, the overlap between the predicted mask obtained by forward mask propagation and the detection mask of the current frame is low. The first similarity S2 = 0.3 is calculated, which does not exceed the single camera matching judgment threshold of 0.5, indicating that the forward matching at the angle of camera C2 has failed.

[0232] When the first similarity of some camera angles (C1 and C3) exceeds the single-camera matching threshold, while the first similarity of another set of camera angles (C2) does not exceed the threshold, the system does not directly adopt the matching result of any single camera angle as the final judgment basis. Instead, it performs weighted fusion of the first similarity of all camera angles to obtain the fusion confidence score.

[0233] Specifically, for the i-th camera angle, its first similarity is denoted as S_i. The system dynamically determines the weight w_i of the camera angle based on the magnitude of S_i. For example, let w_i = S_i / ΣS_j, meaning that camera angles with higher similarity have a larger weight. Then, the fusion confidence S_fused = Σ(w_i·S_i) is calculated.

[0234] exist Figure 10 In the scene shown, the similarity of the three camera angles is S_1=0.8, S_2=0.3, and S_3=0.7, respectively. The weights of each camera angle are calculated as follows:

[0235] ΣS_j = 0.8 + 0.3 + 0.7 = 1.8;

[0236] w_1 = 0.8 / 1.8 ≈ 0.44;

[0237] w_2 = 0.3 / 1.8 ≈ 0.17;

[0238] w_3 = 0.7 / 1.8 ≈ 0.39;

[0239] The fusion confidence score S_fused = 0.44×0.8 + 0.17×0.3 + 0.39×0.7 = 0.352 + 0.051 + 0.273 = 0.676;

[0240] The fusion confidence score of 0.676 exceeds the preset global fusion judgment threshold of 0.5, indicating that the new target object OBJ2 and the first target object OBJ1 are successfully matched. At this time, the system associates the identity of the newly detected target OBJ2 with the identity identifier of the historically lost target OBJ1, and its tracking status is updated from temporarily lost to active, achieving accurate recovery of the lost target.

[0241] If only the matching result of a single camera C2 (0.3, not exceeding 0.5) is relied upon, the identity cannot be recovered alone; however, by fusing information from all camera angles, the high-confidence matching results of cameras C1 and C3 effectively improve the overall judgment confidence, and the system successfully completes the identity recovery.

[0242] In other embodiments, the fusion weights can be determined in other ways. For example, for any camera angle, a stability weight h_i is pre-set or dynamically updated based on the camera's stability during historical tracking (e.g., target detection success rate, depth estimation accuracy, trajectory smoothness, etc.), and then the fusion confidence S_fused = Σ(h_i·S_i) / Σh_i is calculated. Alternatively, the similarity weight and stability weight can be combined. For the i-th camera angle, the combined weight w_i = α·(S_i / ΣS_j) + β·(h_i / Σh_j), where α and β are preset weight coefficients (e.g., α=β=0.5), and then the fusion confidence S_fused = Σ(w_i·S_i).

[0243] In some embodiments, mask matching at camera angles does not need to be successful at all camera angles. When the first similarity at one or more camera angles exceeds the single-camera matching threshold, the system combines the multi-camera 3D matching results (e.g., 3D position consistency, 3D bounding box overlap) to make a final matching determination. If the final determination is a match, the new target object OBJ2 obtains the historical identity of the first target object OBJ1, and its tracking status is updated from temporarily lost to active, achieving accurate recovery of the lost target. If the fusion confidence of the new target object OBJ2 and the first target object OBJ1 exceeds the global fusion determination threshold at all camera angles, the match can be considered successful directly.

[0244] If the fusion confidence does not exceed the global fusion judgment threshold, the new target object is determined to be mismatched with the first target object. The system continues to track the new target object as a new identity and keeps the first target object in the lost object pool to wait for retry in subsequent frames.

[0245] The above description illustrates multi-camera fusion determination using forward masking as an example. It's important to understand that this multi-camera fusion determination method is also applicable to backward masking scenarios. In backward masking, the system, based on the third masking information, uses the current image frame as the start time to predict the fourth masking information of the new target object, with the first image frame as the end time. Then, it obtains the similarity between the first and fourth masking information at each camera angle as the similarity for that camera angle. Using the same method as in the forward masking scenario, it performs weighted fusion of the similarities at each camera angle to obtain a fusion confidence score. Based on the comparison between the fusion confidence score and the global fusion determination threshold, it determines whether the new target object matches or does not match the first target object.

[0246] In this scheme, through the multi-camera mask fusion judgment mechanism, the system can make more robust identity recovery decisions by comprehensively utilizing the observation information of multiple cameras when the information from a single camera viewpoint is insufficient or the confidence level is low. This effectively avoids identity recovery failure caused by misjudgment or omission from a single viewpoint, and significantly improves the robustness and accuracy of target tracking in a multi-camera environment.

[0247] like Figure 11 As shown in the illustration, some embodiments of this disclosure also provide a target object tracking method 650. Method 650 illustrates, by way of example, a flowchart of a method combining target object state transitions, forward masking, and backward masking. Method 650 is used to determine whether a new target object OBJ2 matches a first target object OBJ1.

[0248] Specifically, the system first checks if there are any targets in the "temporarily lost" state in the lost target pool. If the lost target pool is not empty (i.e., there are lost targets, N>0), then for each newly detected target, the system matches it with targets of the same category in the lost target pool.

[0249] The matching process first attempts forward mask propagation, using the historical mask of the last frame before the lost target was lost as a reference. The mask of the current frame is predicted through forward mask propagation, and the intersection-union ratio (IU / R) between the predicted mask and the actual detected mask of the current frame is calculated. If this IU / R is greater than a preset threshold 'a', where 'a' is a configurable parameter and its specific value can be determined based on camera settings, target type, and application requirements, the matching process proceeds. Figure 11 If IoU > a, the match is successful. The system updates the status of the newly detected target from "temporarily lost" to "active" and ends the matching process.

[0250] If forward matching fails (i.e., the intersection-union ratio ≤ a), the system continues to attempt backward mask propagation. Using the detection mask of the current frame as a reference, it propagates backward to the last frame before the data loss, generating a backtracking mask and calculating the intersection-union ratio with the historical mask. If this intersection-union ratio is greater than a preset threshold b (b is also a configurable parameter, which can be the same as or different from a),... Figure 11 If IoU > b, the match is successful and the target status is updated to "active"; if IoU is still less than or equal to b, the match fails and the system continues to process the next target in the lost target pool or terminates the process.

[0251] It should be noted that both a and b are configurable thresholds and are not limited to any specific value. For example, 0.5 is just an example value. In actual applications, it can be adjusted independently according to factors such as camera resolution, target movement speed, degree of occlusion, and the system's requirements for matching accuracy.

[0252] Furthermore, in some embodiments, bidirectional mask propagation is not always executed, but is triggered only when normal tracking based on Kalman filtering and multi-camera 3D matching cannot maintain the continuity of object identity, such as when detection fails for multiple consecutive frames, the target enters the lost object pool, or the identity of a newly detected object is uncertain. When normal tracking is effective, the system prioritizes using Kalman prediction and cross-frame matching to maintain identity continuity, and only enables mask propagation when normal tracking methods are insufficient.

[0253] In some embodiments, to improve efficiency and reduce computational overhead, a fast matching method 700 based on 3D geometric information is preferentially executed to track the target object during the very short period of time when the target has just entered a temporarily lost state, such as 1-2 seconds. This method 700 can use the cross-camera intersection-over-union ratio and class consistency of the 3D bounding boxes before and after the target's loss to make a judgment, enabling the recovery of the target's identity with extremely low latency even under short-term occlusion or slight movement. Only when fast matching fails, and the target enters the lost object pool, or when detection fails for multiple consecutive frames, will the system sequentially trigger the forward masking and backward masking described above for pixel-level identity recovery.

[0254] Specifically, such as Figure 12 As shown, method 700 includes:

[0255] S710. Based on at least two image sequences, detect a new target object and obtain the third category, third two-dimensional bounding box, and third mask information of the new target object;

[0256] S720. Determine the third 3D bounding box of the new target object at each camera angle. The third 3D bounding box is located in the unified world coordinates of at least two cameras.

[0257] S730. Based on the intersection-union ratio of the first three-dimensional bounding box and the third three-dimensional bounding box of the first target object, and the categories of the first target object and the new target object, determine whether the new target object matches the first target object.

[0258] When the target is lost for a short time and moves smoothly, the system prioritizes fast geometric matching over mask propagation.

[0259] First, as described in step S710, the system detects new target objects based on at least two image sequences and obtains the third category, third 2D bounding box, and third mask information of the new target objects. This step is performed by the target detection and instance segmentation module. For the current frame, i.e. Figure 5 At time t+β, the module identifies all newly emerging targets using cues in the form of text, 2D images, or 3D models.

[0260] Taking the aforementioned bird tracking scenario as an example, the system detects a bird and outputs its category label "bird," a 2D bounding box, and a pixel-level binary mask. The third category is used for subsequent comparison with the category of the first target object, ensuring that only targets of the same category can match; the third 2D bounding box is used to initially locate the target's position in the image; and the third mask information serves as backup data, which can be used for subsequent mask propagation if a fast match fails.

[0261] Next, the system determines the third 3D bounding box of the new target object at each camera angle, and the third 3D bounding box is located in the unified world coordinates of at least two cameras.

[0262] Specifically, the intrinsic parameters of each camera and the corresponding depth map can be used to back-project the pixels in the third mask of the new target object to the camera coordinate system to generate a three-dimensional point set. Then, the minimum and maximum values ​​in each axis can be calculated to obtain the local 3D bounding box from the perspective of each camera.

[0263] Then, using the extrinsic parameters obtained from multi-camera calibration, these local 3D bounding boxes are transformed to a pre-established unified world coordinate system to obtain a third 3D bounding box. Taking the aforementioned bird tracking example, this means that the instance mask of the bird detected at time t+β is used to obtain the 3D bounding box of the bird in the world coordinate system through depth map backprojection and coordinate transformation.

[0264] Simultaneously, the system retrieves the historical first 3D bounding box of the first target object from the lost object pool. This bounding box was recorded at time t-α and is also located in the unified world coordinate system. In this way, the two bounding boxes are in the same spatial reference, allowing for direct comparison of spatial overlap.

[0265] Finally, based on the intersection-union ratio (IU) of the first and third 3D bounding boxes and the categories of the first and new target objects, the system determines whether the new target object matches the first target object. The system first compares whether their category labels are the same: if the lost object is a bird and the newly detected object is also a bird, the categories are the same; if the categories are different, a mismatch is directly determined.

[0266] After classifying the two bounding boxes as the same, the system further calculates the intersection-union ratio (IUR) between them, which is the volume of their intersection divided by their union volume, with a value between 0 and 1. When the IUR exceeds a preset third threshold, the system considers the two bounding boxes to be highly overlapping in spatial location. This third threshold can be set according to application requirements; for example, 0.5 is just an example, and the new target object is very likely the first target object that was previously lost.

[0267] If the bird undergoes only a small displacement during the time interval from t-α to t+β, and its posture does not change significantly, the two 3D bounding boxes will have a large overlap area in space, and the crossover ratio can easily exceed the third threshold, thus achieving rapid recovery.

[0268] At this point, the system directly associates the newly detected target with the historical identity of the first target object, restores it from a temporarily lost state to an active state, and removes the record from the lost object pool. If the intersection-union ratio does not exceed the threshold, the fast matching is deemed to have failed, and the system then proceeds to the forward masking and backward masking processes to handle more complex deformation scenarios.

[0269] In this solution, the hierarchical recovery strategy ensures high efficiency in short-term, simple scenarios while ensuring robustness in complex scenarios through bidirectional mask propagation, achieving the best balance between efficiency and accuracy.

[0270] After detailing the identity recovery mechanism based on fast geometric matching and bidirectional mask propagation in the target re-identification module, it should be noted that in some embodiments, the above method can be further combined with the following methods to achieve target object tracking. For example, the system continuously and stably acquires the precise 3D position and motion state of the target from each camera viewpoint. If the tracking accuracy of a single camera is insufficient or there is severe jitter, continuous real-time cross-view fusion and identity recovery will lose their reliable data foundation.

[0271] Therefore, as Figure 13 As shown, some embodiments of this disclosure provide a target object tracking method 750, which realizes local 3D tracking of the target object in the local coordinate system of each camera. In some embodiments, the method 750 can utilize target detection and instance segmentation results, depth maps, and Kalman filters to provide smooth and continuous three-dimensional position and velocity estimates for each camera.

[0272] like Figure 13 As shown, the local 3D tracking method 750 includes:

[0273] S752. Based on at least two image sequences, detect a first target object and obtain the first object category, first two-dimensional bounding box, and first mask information of the first target object;

[0274] S754. Obtain the depth map of the first target object;

[0275] S756. Based on the depth map and the first two-dimensional bounding box, determine the local three-dimensional bounding box of the first target object at each camera angle. The local three-dimensional bounding box is located in the local coordinates of the camera.

[0276] S758. Based on the local 3D bounding box, update the prediction speed and prediction local 3D bounding box of the first target object predicted using Kalman filtering at the first image frame time.

[0277] First, based on at least two image sequences, the system detects a first target object and obtains its first object category, first 2D bounding box, and first mask information. This step is performed by the object detection and instance segmentation module. For each frame of RGB image acquired by each camera, this module identifies the target of interest using cues in the form of text, 2D images, or 3D models, and outputs the target's category label, 2D bounding box (i.e., the smallest bounding rectangle of the target in the image plane), and pixel-level binary mask.

[0278] The first mask information precisely delineates the pixel area occupied by the target in the image, eliminating interference from background pixels and providing an accurate spatial range for subsequent 3D backprojection. This process can be performed offline during the system import phase or online in real-time, without requiring retraining of the model for new objects.

[0279] Secondly, the system obtains a depth map of the first target object. The depth map is a two-dimensional matrix aligned with the RGB image, where each pixel value represents the distance from the corresponding 3D point to the camera's optical center, i.e., the depth value. The method of acquiring the depth map depends on the camera type: for a monocular RGB camera, the system uses a lightweight deep learning model to estimate the depth image in real time; for a binocular RGB camera, it uses stereo matching and triangulation techniques to calculate disparity and generate the depth map; for a ToF sensor, it directly obtains the depth value of each pixel by analyzing the phase difference between emitted and reflected light. Regardless of the method used, the system also utilizes a depth refinement network to denoise and complete the original depth data to improve the accuracy of the depth map.

[0280] Next, based on the depth map and the first 2D bounding box, the system determines the local 3D bounding box of the first target object at each camera angle, and this local 3D bounding box is located in the camera's local coordinate system, that is, in 3D space with the camera's optical center as the origin. The specific implementation process is as follows:

[0281] For each foreground pixel (u, v) within the detected first mask information, the system reads the corresponding depth value (Z) from its paired depth image.

[0282] To convert pixel coordinates (u, v, Z) to 3D coordinates (X, Y, Z) in the camera coordinate system, it is necessary to rely on the camera's intrinsic parameters. Camera intrinsic parameters are a unique set of parameters that describe the camera's imaging geometry. They define the mathematical mapping between the camera coordinate system and its projection points on the 2D image plane. In digital cameras, intrinsic parameters are typically represented by a 3x3 matrix. express:

[0283]

[0284] Among them, focal length and These represent the focal length along the x and y axes, respectively, in pixels. They determine the camera's magnification and field of view, converting physical distances (meters) into pixel distances in the image. (Main point) and This represents the pixel coordinates of the intersection point of the camera's optical axis and the image plane (i.e., the image center). It corrects for the deviation between the sensor array center and the optical center. Using this intrinsic parameter matrix, coordinate transformation is completed through the inverse transformation (back projection) of the pinhole camera model:

[0285]

[0286]

[0287] Therefore, pixel coordinates (u, v, Z) are converted into 3D coordinates (X, Y, Z) in the camera coordinate system, referred to as "absolute camera coordinates." All pixels belonging to the target are calculated in this way, forming a 3D point cloud of the target surface. To represent the target's position and extent in space more concisely, the system analyzes the distribution of this point cloud along the X, Y, and Z axes. The minimum and maximum values ​​of all points on each axis are calculated, and these six values ​​collectively define a 3D axial bounding box. This bounding box tightly encloses the target's 3D point cloud, and the center coordinates of this bounding box are the extracted 3D position observations of the target in the current frame and from the camera's perspective. Simultaneously, the dimensions (length, width, height) of the bounding box also reflect the target's spatial volume.

[0288] In some embodiments, the system can update the predicted velocity and predicted local 3D bounding box of the first target object at the first image frame time based on the aforementioned local 3D bounding box. Each camera independently maintains a Kalman filter, whose state vector typically contains the target's 3D position and 3D velocity.

[0289] The Kalman filter works in two phases: prediction and update. In the prediction phase, the filter predicts the target position and velocity in the current frame based on the state estimate of the previous frame and a preset state transition model, usually a uniform motion model, while calculating the uncertainty of the predicted state.

[0290] During the update phase, the filter obtains the actual observations of the current frame through backprojection. These observations include at least one of the following: the center coordinates, size, and orientation angle of the local 3D bounding box. The filter then calculates the Kalman gain, corrects the predicted values ​​using the observations, outputs the filtered optimal state estimate, and updates the covariance matrix.

[0291] Through this prediction-update loop, the Kalman filter effectively suppresses measurement noise from the depth sensor and minor jitter in the detection box, outputting smooth and continuous 3D position and velocity estimates. More importantly, when a target is not detected in the current frame due to brief occlusion, the filter can still make pure predictions based on the state and motion model of the previous frame, providing reasonable tracking continuation over several frames, thereby enhancing the robustness of single-view tracking. Furthermore, the predicted states output by the Kalman filters maintained independently by each camera can be transformed into a unified world coordinate system to support subsequent cross-camera multi-target matching and global trajectory fusion.

[0292] Taking the aforementioned bird tracking as an example, when the bird is flying normally within the field of view of camera 1, the local 3D tracking module of camera 1 uses the detection mask and depth map of each frame to continuously output the smooth three-dimensional trajectory of the bird in the coordinate system of camera 1 through back projection and Kalman filtering. Even if the bird briefly flies behind the canopy and is lost in detection, the Kalman filter can predict its position in subsequent frames based on its velocity and position before the loss, providing an initial guess for subsequent recovery.

[0293] The local 3D tracking results provide independent and stable 3D observation data from each camera's perspective for the subsequent cross-view fusion module, which is the foundation for achieving global 3D tracking and identity preservation.

[0294] In this scheme, to address the instability of 3D position estimation caused by depth sensor noise, detection box jitter, and brief occlusion under single-camera perspective, a local 3D bounding box is generated by back-projecting a 2D detection mask onto the camera coordinate system. A Kalman filter is then used to predict and update the observations of consecutive frames. This overcomes the defects of position jumps and velocity noise introduced by directly using single-frame detection results for localization in existing technologies. It significantly improves the smoothness and continuous stability of single-camera tracking, reduces the risk of trajectory interruption caused by detection loss or depth error, and enhances the system's motion prediction capability during short-term occlusion. This provides a reliable 3D observation foundation for subsequent cross-view fusion and identity recovery.

[0295] In some embodiments, after completing local 3D tracking for each camera, each camera independently obtains the target's 3D position and velocity estimate in its own camera coordinate system. Because the different cameras are mounted and oriented differently, these local observations exist in their own independent coordinate systems, making direct cross-view correlation and fusion impossible.

[0296] Furthermore, a single camera may lose sight of the target due to limited field of view or obstruction, leading to tracking interruption. To overcome these shortcomings, such as... Figure 14 As shown, some embodiments of this disclosure provide a target object tracking method 770, which realizes global 3D tracking through a cross-view fusion module, unifies the local observations of each camera into the same world coordinate system, and improves positioning accuracy and tracking continuity by using multi-view data fusion.

[0297] Specifically, such as Figure 14 As shown, the global 3D tracking method 770 includes:

[0298] S772. Based on the local 3D bounding box, determine the first 3D bounding box of the first target object at each camera angle, wherein the first 3D bounding box is located in the unified world coordinates of at least two cameras;

[0299] S774. Based on the intersection-union ratio of the first 3D bounding box, determine whether the first target object matches at each camera angle.

[0300] The system first determines the first three-dimensional bounding box of the first target object at each camera angle based on the local three-dimensional bounding box calculated independently by each camera, and these first three-dimensional bounding boxes are located in the unified world coordinates of at least two cameras.

[0301] Specifically, the cross-view fusion module uses extrinsic parameters obtained from multi-camera calibration to transform the local 3D bounding boxes in each camera coordinate system to a pre-established world coordinate system. Through this transformation, the 3D bounding boxes, originally scattered in different camera coordinate systems, are aligned to the same global reference. At this point, the first 3D bounding box reported by each camera represents the position and volume of the target in the same physical space.

[0302] When multiple cameras simultaneously report their first 3D bounding boxes in the world coordinate system, the system needs to determine which observations correspond to the same physical target. To this end, the system calculates the cross-camera intersection-union ratio (CIU) between the 3D bounding boxes reported by different cameras. The CIU is defined as the intersection volume of two bounding boxes divided by their union volume, and its value ranges from 0 to 1. If the CIU exceeds a preset fourth threshold (which can be configured based on factors such as system calibration accuracy and object volume), then the two observations are determined to originate from the same target.

[0303] For example, when camera 1 and camera 2 simultaneously observe the same bird, their two 3D bounding boxes in the world coordinate system will highly overlap in spatial position, with an intersection-over-union ratio close to 1; conversely, if the two bounding boxes correspond to different objects, the intersection-over-union ratio will be close to 0.

[0304] By employing this spatial overlap-based matching strategy, the system can quickly and accurately associate cross-view observations without relying on appearance features, effectively avoiding feature matching failures caused by changes in lighting and viewing angle.

[0305] For multiple observations that are successfully matched, such as three cameras detecting the same target at the same time, the system uses the cross-camera intersection-union ratio as a weight to fuse the first 3D bounding box of the first target object from each camera angle to form a fused 3D bounding box.

[0306] It should be noted that the cross-camera intersection-over-union ratio here is used for multi-camera multi-object matching between different camera views at the same time step, which is different from the intersection-over-union ratio between predicted boxes and detection boxes used in subsequent inter-frame matching.

[0307] The weights can be set based on the intersection-over-union ratio (IoU) itself or the confidence scores of each camera detection. The weighted average is calculated as: P_fused = (Σ w_i·P_i) / Σ w_i, where P_i is the center coordinate of the 3D bounding box of the i-th camera, and w_i is the corresponding weight.

[0308] The fused 3D bounding box has higher accuracy and stability than any single camera observation because it integrates information from multiple perspectives and can average the random errors in depth estimation from a single camera.

[0309] Subsequently, the fused 3D bounding box is fed into a global Kalman filter running in the world coordinate system to update the target's global 3D position and velocity, and predict its state at the next moment. By fusing data from multiple cameras, the errors introduced by a single camera due to occlusion or inaccurate depth estimation are effectively mitigated, significantly improving the stability and accuracy of 3D velocity estimation.

[0310] In this scheme, to address the issues of limited field of view, occlusion, and incomplete local observations caused by depth noise in single-camera systems, the local 3D bounding boxes of each camera are transformed to a unified world coordinate system. Cross-view matching is performed using the intersection-union ratio (CIRR) of the 3D bounding boxes, and multi-view observations are fused using CIRR or confidence as weights. This overcomes the defects in existing technologies such as target positioning drift and trajectory breakage caused by single-view loss or depth errors, significantly improving the global positioning accuracy and robustness of multi-camera collaborative tracking, and laying a solid data foundation for subsequent identity preservation and continuous tracking.

[0311] like Figure 15 As shown, in some embodiments of this disclosure, a target object tracking method 780 is provided. After completing multi-view weighted fusion to obtain the fused three-dimensional bounding box of the current frame, i.e. the first image frame, the system continues the tracking result to the next frame to achieve continuous tracking of the target.

[0312] The target object includes a second target object, and each of the at least two image sequences includes a first image frame corresponding to the first target object and a second image frame corresponding to the second target object;

[0313] Specifically, such as Figure 15 As shown, method 780 includes:

[0314] S782. Based on the fused 3D bounding box, Kalman filtering is used to predict the velocity of the first target object and the predicted 3D bounding box of the first target object at the second image frame time.

[0315] S784. Determine the second three-dimensional bounding box of the second target object at each camera angle, wherein the second three-dimensional bounding box is located in the unified world coordinates of at least two cameras;

[0316] S786. Based on the intersection-union ratio of the predicted 3D bounding box and the second 3D bounding box, determine whether the first target object matches the second target object.

[0317] The system maintains a global Kalman filter in the world coordinate system, whose state vector contains the target's three-dimensional position and three-dimensional velocity. The prediction phase of this filter is based on the fused observations of the previous time step, i.e., the first image frame, and a preset state transition model, usually a uniform motion model, to calculate the predicted state at the current time step, i.e., the second image frame, including the predicted velocity and the predicted three-dimensional bounding box.

[0318] The center position of the predicted bounding box is determined by the predicted location, and its size can be inherited from the size of the fused bounding box in the previous frame or adjusted appropriately according to the motion model. This prediction mechanism allows the system to provide a reasonable estimated position based on motion inertia even if the target is briefly not detected in a frame, thus maintaining the continuity of tracking.

[0319] Next, upon the arrival of the second image frame, the system detects the newly emerging candidate target, i.e., the second target object, through the target detection and instance segmentation module, and obtains its category, 2D bounding box, and mask information. Subsequently, similar to the aforementioned step S501, the system uses the depth map, camera intrinsic parameters, and extrinsic parameters to back-project the pixels within the mask to the camera coordinate system, and then transforms them to the unified world coordinate system, thereby obtaining the second 3D bounding box at each camera angle. These bounding boxes are under the same global reference as the historical observations of the first target object, allowing for direct spatial comparison.

[0320] The system calculates the 3D intersection-union ratio (IU) of the bounding boxes predicted by the global Kalman filter with each of the detected second 3D bounding boxes.

[0321] Unlike cross-camera matching, this is cross-frame matching: the predicted position of the tracked target in the previous frame is compared with the spatial position of the candidate target in the current frame to determine the overlap. If the intersection-union ratio (IU) of a second 3D bounding box with the predicted bounding box exceeds a preset fifth threshold, which can be adjusted according to motion smoothness and detection noise level (e.g., between 0.3 and 0.7), then the candidate target is considered to be the first target object tracked in the previous frame, and the match is successful. This cross-frame IU is used for temporal matching, ensuring the continuity of identity between adjacent frames in the same camera stream. This is a different matching process from the aforementioned cross-camera IU.

[0322] Subsequently, the successfully matched observation is used to update the global Kalman filter, outputting a corrected 3D position and velocity estimate, which serves as the basis for prediction in the next frame. If no candidate is matched, the target state is temporarily lost, triggering the aforementioned loss recovery mechanism, namely fast geometric matching or bidirectional mask propagation. Through this closed loop of "prediction-detection-matching-update," the system achieves continuous and stable tracking of the target, maintaining identity consistency even during brief occlusion or rapid movement.

[0323] Taking bird tracking as an example: Assume that in the first image frame (time t-α), the bird has been successfully tracked by camera 1 and its 3D bounding box has been fused. Based on this bounding box and the bird's motion velocity, the global Kalman filter predicts the bird's predicted bounding box in the next frame (time t). When the bird actually appears in the next frame, the system calculates the detected bounding box and calculates the 3D intersection-over-union ratio (IoU) with the predicted bounding box.

[0324] If the bird's flight path is smooth and the prediction is accurate, the crossover ratio is usually high, exceeding the fifth threshold, then the match is successful. The filter uses the detected value to correct the predicted value, outputs a more accurate position and velocity, and continues to predict the state in the next frame (time t+β).

[0325] In this scheme, for the problem of continuous tracking from the current frame to the next frame in a multi-camera environment, the velocity of the target in the next frame is predicted by fusing the 3D bounding box based on the previous frame using a global Kalman filter. The second 3D bounding box detected in the next frame is then matched with the predicted bounding box by cross-union ratio (CUB). This overcomes the defects of existing technologies, such as trajectory interruption and identity jump caused by single-frame detection failure or sudden motion changes. It significantly improves the continuity and cross-frame consistency of target tracking, reduces the risk of loss due to brief occlusion or rapid movement, and enhances the long-term tracking stability of the system in dynamic scenes.

[0326] In another feasible option, such as Figure 16 As shown, some embodiments of this disclosure provide a method 800 for transforming a local 3D bounding box to multi-camera global coordinates, which can be used in the target object tracking method described herein, for example, step S772. Figure 16 As shown, method 800 includes:

[0327] S810, Obtain the depth map of the first target object and the camera intrinsics of at least two cameras;

[0328] S812. Based on the depth map and camera intrinsic parameters, back-project the pixels in the first mask information to the camera coordinate system to generate a three-dimensional point set, so as to construct the local three-dimensional bounding box of the first target object.

[0329] S814, Obtain camera extrinsic parameters from at least two cameras;

[0330] S816. Based on the camera extrinsic parameters, transform the local 3D bounding box to a unified world coordinate system to obtain the first 3D bounding box.

[0331] First, depth maps of the target object are acquired according to different camera types. The system also uses a depth refinement network to reduce noise and complete the raw depth data to improve the accuracy of the depth maps. Simultaneously, a camera intrinsic parameter matrix containing focal length and principal point is obtained, which describes the mapping relationship between the camera coordinate system and the image pixel coordinate system. These intrinsic parameters are typically calibrated during production or obtained through offline calibration.

[0332] Next, based on the depth map and camera intrinsic parameters, each pixel within the mask is back-projected into a 3D point set in the camera coordinate system. The axis-aligned bounding box is calculated, and its center coordinates are used as the 3D position observation of the target in the current camera view. This specific method has been described above and will not be elaborated on here.

[0333] The camera extrinsic parameters, including the rotation matrix R and translation vector t, describe the rigid body transformation relationship between the camera coordinate system and the world coordinate system. These extrinsic parameters can be obtained using traditional calibration plate methods or the markerless calibration method based on hand joints proposed in this invention, detailed below. The latter method simultaneously acquires hand pose data from multiple cameras, uses deep learning to detect the 3D coordinates of 21 hand joints, and directly solves the 3D-3D rigid body transformation after matching corresponding points in different cameras. This eliminates the need for physical targets, making it more flexible and convenient.

[0334] Finally, based on the camera extrinsic parameters, the local 3D bounding box is transformed to a unified world coordinate system to obtain the first 3D bounding box. The details are as follows:

[0335] First, the module uses camera extrinsic parameters to determine the local 3D coordinates of the target under each camera. = (X, Y, Z) is transformed to the world coordinate system, following rigid body transformation relations:

[0336]

[0337] For the velocity vector estimated by the local Kalman filter The transformation to the world coordinate system does not consider translation, but is performed only through a rotation matrix:

[0338]

[0339] This step maps all camera observations of the target into the same global 3D space, laying the geometric foundation for cross-view data fusion.

[0340] Through this transformation, the local 3D bounding boxes, originally scattered across different camera coordinate systems, are aligned to the same global reference. At this point, the first 3D bounding box reported by each camera represents the position and volume of the target in the same physical space, providing a unified data foundation for cross-camera IoU matching and fusion of 3D bounding boxes.

[0341] Multi-camera external parameters

[0342] like Figure 17 As shown, some embodiments of this disclosure provide a method 820 for obtaining camera extrinsic parameters of multiple cameras, which can be used in the target object tracking method described herein, for example, step S814. Method 820 is a method for markerless extrinsic parameter calibration using hand pose. This method is essentially a hand pose reconstruction method, which does not rely on a calibration board or specific markers and is simple to operate.

[0343] like Figure 17 As shown, method 820 includes:

[0344] S822. Simultaneously acquire images of the same hand gesture using at least two cameras;

[0345] S824. Using a deep learning-based hand joint detection network, calculate the three-dimensional coordinates of each hand joint in each camera coordinate system.

[0346] S826. Match the 3D coordinate point pairs of the same hand joint in different cameras and construct the corresponding point set;

[0347] S828. Based on the corresponding point set, solve the first rotation matrix and the first translation vector between the camera pairs through singular value decomposition as the first camera extrinsic parameters.

[0348] S830: Based on the reference camera, the initial camera extrinsic parameters are unified to the unified world coordinates to obtain the first unified camera extrinsic parameters.

[0349] Multi-camera extraparameters based on hand joints

[0350] A preferred calibration method is based on automatic calibration of hand joints. This method is essentially a hand pose reconstruction method. First, images of the same hand pose are simultaneously acquired by at least two cameras. Multiple cameras are synchronized via hardware triggering or timestamps, capturing images containing the hand at the same time. This synchronization ensures that each camera captures an image of the hand in the same spatial pose, providing temporal consistency for subsequent corresponding point matching.

[0351] Next, a deep learning-based hand joint detection network is used to calculate the 3D coordinates of each hand joint in each camera coordinate system. For each RGB image obtained from a camera, the system first determines the hand region. To improve efficiency, a dynamic detection strategy can be adopted: when the hand movement is relatively smooth, the hand region of the current frame is predicted by scaling the bounding box of the hand region in adjacent frames, thus avoiding running the detector frame by frame; when the movement is violent or reaches a preset period, an open vocabulary object detection model such as YOLO-World is run for accurate localization.

[0352] After detecting the hand region, a hand correction model, such as MobileCLIP, is used to filter out false detection regions with a probability below a threshold. Then, a single-view hand detection model, such as SimpleHand, is used to estimate the two-dimensional pixel coordinates of the 21 hand joints. These coordinates are then converted into three-dimensional coordinates in each camera's own coordinate system by combining camera intrinsics and depth information. Simultaneously, a chirality detection sub-model identifies whether the hand is left or right, providing semantic labels for subsequent cross-camera matching. This yields the three-dimensional point set of the hand joints observed by each camera.

[0353] After obtaining the 3D coordinates of hand keypoints in each frame of images from each camera using a single-view hand detection model, the system performs spatiotemporal optimization of the hand pose parameters during continuous processing to further improve the accuracy and stability of the hand pose parameters. Specifically, the optimization process includes at least one of the following strategies:

[0354] 1) Temporal Consistency-Based Optimization: For an image sequence captured from the same camera angle, the system calculates the Euclidean distance between the hand pose parameters of the matching hand regions in the i-th frame and the (i-1)-th frame. If this distance is greater than a preset second distance threshold, it indicates a significant jump in the hand state between the current frame and the previous frame, possibly due to detection noise or rapid movement. In this case, the system does not use the original parameters of the current frame, but instead replaces the original parameters of the i-th frame with a weighted result (e.g., a weighted average) of the hand pose parameters of the i-th frame and the corresponding hand pose parameters in the n frames preceding the i-th frame, thereby suppressing abnormal jumps.

[0355] 2) Spatial Consistency-Based Optimization: For images captured at the same time but from different angles, the system calculates the Euclidean distance between the hand pose parameters of the matching hand region in the i-th frame and the image from another angle. If this distance is greater than a preset third distance threshold, it indicates that the reconstruction results of the same physical hand by the two cameras are inconsistent, possibly due to detection errors of one camera. In this case, the system replaces the abnormal parameters in the i-th frame with the hand pose parameters of the corresponding hand region in the (i-1)-th frame captured from the same angle, thereby maintaining cross-view consistency.

[0356] 3) Smoothing for Spatiotemporal Consistency: If the difference in hand pose parameters between frame i and frame i-1 is less than or equal to the second distance threshold (temporal consistency), and the difference in hand pose parameters between frame i and other angle images is also less than or equal to the third distance threshold (spatial consistency), then the hand pose parameters of the current frame are considered reliable. To further reduce random noise, the system smooths the hand pose parameters of frame i against the corresponding hand pose parameters in the previous n frames, using methods such as weighted moving average or exponential smoothing, to make the trajectory smoother and more continuous.

[0357] After the above optimization, the system matches 3D coordinate point pairs of the same hand joint in different cameras and constructs corresponding point sets. Since hand joints have clear correspondences in different cameras—for example, the "wrist" joint in all cameras corresponds to the same physical point—the system matches corresponding point pairs between cameras using the semantic tags of the joints, constructing multiple sets of 3D-to-3D correspondences. These point pairs form the basis for subsequent solutions to rigid body transformations.

[0358] Next, based on the corresponding point set, the first rotation matrix and the first translation vector between the camera pairs are solved by singular value decomposition as the first camera extrinsic parameters.

[0359] Specifically, given two sets of corresponding 3D joint point coordinates, denoted as follows: and They are observed in the camera coordinate system. and In this context, the rigid body transformation between the two cameras is defined as:

[0360]

[0361] Rigid body transformation describes a spatial transformation in which the distance between any two points inside an object (or a coordinate system) remains constant and the angle between any vectors remains constant. The object only undergoes overall rotation and translation, without any deformation such as twisting, stretching, compression or mirror reflection. Table 1 shows the meaning and dimensions of each variable.

[0362] Table 1

[0363]

[0364] Next, the rigid body transformation (rotation matrix R and translation vector t) between camera pairs is estimated using corresponding point pairs. For each camera pair, the optimal rotation matrix and translation vector in the least squares sense are solved using the matched set of corresponding points (e.g., the 3D coordinates of the same joint in camera A and camera B) through methods such as singular value decomposition (SVD), so that the points in camera A coincide as much as possible with the corresponding points in camera B after the transformation. This transformation describes the relative pose between the cameras.

[0365] Specifically, for a pair of cameras i and j that need to be calibrated, an optimal rigid transformation is calculated by solving a least-squares problem using the 21 sets of 3D point pairs matched in step three. This transformation is determined by the rotation matrix. Translation vector Defined, its goal is to transform the set of points. With the target point set To minimize the overall distance error between the two sets of points, the optimal rigid body transformation between them is determined using Singular Value Decomposition (SVD). The calculation steps are as follows:

[0366] The optimal rotation is calculated by minimizing the mean square distance between corresponding 3D points. and :

[0367]

[0368] Computation source set and target point set The geometric center (centroid) of the two point sets, which is also the "average position" of the two point sets:

[0369]

[0370] The center coordinates are obtained by subtracting the centroid:

[0371]

[0372] Now that the two point sets have been translated to the vicinity of the origin, the next step can focus on solving the rotation problem, eliminating the interference of translation, and constructing the covariance matrix:

[0373]

[0374] This matrix contains all the information needed to solve for the optimal rotation. The covariance matrix is ​​then decomposed to extract the optimal rotation matrix.

[0375]

[0376] Therefore, the optimal rigid body transformation can be calculated:

[0377]

[0378] This transformation The relative pose from camera i coordinate system to camera j coordinate system was initially defined.

[0379] Finally, based on the reference camera, the initial camera extrinsics are unified to a unified world coordinate system to obtain the first unified camera extrinsics. One camera is selected as the reference datum, and the transformations of all other cameras relative to this reference camera are calculated, unifying all pairs of transformation relationships into a shared global coordinate system.

[0380] Subsequently, the local 3D coordinates of any object under any camera can be accurately transformed into this global coordinate system using the corresponding extrinsic parameters. These steps enable consistent multi-camera alignment using only the 21 detected hand joints, providing a markerless, pattern-free extrinsic parameter calibration method. For scenarios lacking hand detection capabilities, a traditional calibration board method can be used as an alternative. After obtaining accurate camera extrinsic parameters, the global 3D tracking module can transform the local observations from each camera into a unified world coordinate system, laying the geometric foundation for subsequent cross-camera matching and fusion.

[0381] To further improve the robustness and accuracy of external parameter calibration, such as Figure 18 As shown, some embodiments of this disclosure provide a method 840 for obtaining camera extrinsic parameters of at least two cameras, which can be used in the target object tracking method described herein, for example, step S820. Figure 18 As shown, method 840 includes:

[0382] S842. Using the random sampling consensus algorithm, outlier point pairs in the corresponding point set are removed, and the remaining point pairs are used to re-solve the first rotation matrix and the first translation vector to obtain the second camera extrinsic parameters. The second camera extrinsic parameters include the second rotation matrix and the second translation vector.

[0383] S844. Based on the reference camera, the extrinsic parameters of the second camera are unified to the unified world coordinates to obtain the extrinsic parameters of the second camera.

[0384] Specifically, hand joint detection networks may produce false detections in complex backgrounds, under occlusion, or during rapid movement, causing the 3D coordinates of some joints to deviate from their true positions. Directly using all 21 point pairs for singular value decomposition may introduce significant errors. RANSAC iteratively samples a small number of point pairs, such as 3 pairs, to estimate candidate rigid body transformations. It then counts the number of interior points that satisfy the error tolerance of the transformation, repeats this process multiple times, and retains the candidate transformation with the most interior points. Finally, it recalculates the rotation matrix and translation vector using all interior points.

[0385] This process effectively eliminates erroneous matches, making the calibration results more resistant to interference from external points. Subsequently, based on the reference camera, the extrinsic parameters of the second camera are unified into a unified world coordinate system to obtain the second unified camera extrinsic parameters. Similar to step S50135, one camera is selected as the origin of the world coordinate system, and the second extrinsic parameters of the other cameras are transformed to the same global reference, forming a more consistent set of calibration parameters.

[0386] In this scheme, by introducing RANSAC to remove outliers, the stability of multi-camera extrinsic calibration is significantly improved without increasing the user's operational burden, providing a more reliable coordinate transformation basis for subsequent global 3D tracking.

[0387] like Figure 19 As shown, to further improve the accuracy of extrinsic parameter calibration, some embodiments of this disclosure provide a method 850 for obtaining camera extrinsic parameters of at least two cameras, which can be used in the target object tracking method described herein, for example, step S820 or step S840. Figure 19 As shown, method 850 includes:

[0388] S852. Using the three-dimensional joint coordinates of different hand poses and positions in multiple frames, the second rotation matrix and the second translation vector are iteratively optimized to obtain the third camera extrinsic parameters, which include the third rotation matrix and the third translation vector.

[0389] S854. Based on the reference camera, the extrinsic parameters of the third camera are unified to the unified world coordinates to obtain the extrinsic parameters of the third camera.

[0390] Specifically, a single gesture may lack sufficient constraints in certain degrees of freedom due to the uniformity of hand posture, such as the palm always facing the camera, resulting in significant uncertainty in the calculated extrinsic parameters in a specific direction. The system acquires multiple frames of gesture images containing different hand postures, such as extended, clenched fist, wrist rotation, and gestures in different spatial positions, and independently executes steps S50131 to S50137 for each frame to obtain multiple candidate extrinsic parameter estimates.

[0391] Then, these estimates are fused using nonlinear least squares optimization or weighted average methods: corresponding point pairs from all frames can be merged into a large point set, and a unified rotation matrix and translation vector can be jointly optimized; alternatively, Lie algebra interpolation or averaging can be performed on the extrinsic parameters obtained independently from each frame, and then converted back to the rotation matrix form.

[0392] The third camera extrinsic parameters, obtained after multi-frame iterative optimization, exhibit higher accuracy and consistency across all degrees of freedom in rotation and translation. Subsequently, based on the reference camera, the third camera extrinsic parameters are unified to a unified world coordinate system to obtain the third unified camera extrinsic parameters. This step is identical to the aforementioned unification method, transforming the optimized camera extrinsic parameters to the global coordinate system.

[0393] In this scheme, iterative optimization is performed using multiple frames of hand images in different poses and positions, which further eliminates the random errors and insufficient constraints that may exist in a single sampling. This enables the extrinsic parameter calibration results to achieve accuracy comparable to those using traditional calibration boards, while maintaining the convenience of markerless and rapid deployment, providing a solid and reliable geometric benchmark for global 3D tracking.

[0394] Real-time calibration and verification of multi-camera extrinsic parameters

[0395] like Figure 20As shown, some embodiments of this disclosure provide a calibration and verification method 870 for multi-camera extrinsic parameters. For example, during long-term system operation, due to temperature changes, mechanical vibrations, or the passage of time, the extrinsic parameters of multiple cameras may experience slight drift, thus affecting the accuracy of global 3D tracking. In order to continuously monitor the calibration status without interrupting the task, method 870 provides a real-time calibration and verification method based on depth projection.

[0396] Specifically, such as Figure 20 As shown, method 870 includes:

[0397] S872. Based on the depth map, perform at least one of the following to verify the accuracy of real-time calibration of at least two cameras:

[0398] S874. Determine whether the cross-camera volume deviation value is within a preset first threshold, wherein the cross-camera volume deviation value is the difference between the volume of the first three-dimensional bounding box at one angle in different angles and the volume of the first three-dimensional bounding box at another angle in different angles.

[0399] S876. Determine whether the cross-camera position deviation value is within a preset second threshold, wherein the cross-camera position deviation value is the difference between the center position of the first three-dimensional bounding box at one angle and the center position of the first three-dimensional bounding box at another angle.

[0400] S878. Determine whether the scale value of the cross-camera projection point exceeds the preset third threshold, wherein the cross-camera projection point is the projection of the point in the first three-dimensional bounding box of one angle into the first three-dimensional bounding box of another angle.

[0401] The system of this invention automatically verifies the internal and external parameters of multiple cameras based on the consistency of 3D information observed by different cameras when an object is first detected in a multi-camera environment. This module does not require additional image capture or physical calibration targets. Instead, it uses real-time depth information of the actual tracked object to evaluate the effectiveness of the calibration, distinguishing it from traditional methods. This approach achieves real-time calibration verification solely based on the object's depth data, without relying on external calibration targets.

[0402] First, based on the depth map, at least one of the following validation methods is performed to evaluate the accuracy of real-time calibration for at least two cameras. This validation relies on five fundamental assumptions:

[0403] 1. Time Synchronization Assumption: It is assumed that the two cameras observe the same target at almost the same moment. To this end, each camera needs to achieve frame synchronization through hardware trigger signals or timestamp-based alignment to maintain time consistency.

[0404] 2. Depth Reliability Assumptions: This verification method assumes that the 3D points or depth maps generated by each camera have sufficient accuracy. Therefore, the following conditions must be met: depth data from Time-of-Flight (ToF) or stereo vision-based methods should not have excessive noise; the error of the deep learning-based depth estimation model should be within an acceptable range, for example, approximately ±5%; and the target should not have highly reflective or transparent properties. This ensures that any differences in the 3D bounding box comparisons stem from actual structural differences, rather than sensor noise.

[0405] 3. Target Visibility Assumption: The same target should be fully observable in both cameras. Validation will be delayed if the camera fields of view overlap is insufficient or the target visibility is inadequate. This includes: sufficient overlap between the fields of view of the two cameras; the 2D masks of partially visible targets should not differ significantly; and the same target must have been identified as a matching target during the multi-camera target detection phase.

[0406] 4. Initial Calibration Validity Assumption: This verification module is not responsible for recalibrating the system; instead, it assumes that the initial calibration is accurate. Its function is to detect drift or abnormal errors. If there are significant errors in the initial extrinsic parameter calibration, such as errors of tens of centimeters, the verification module will mark the camera as unverifiable.

[0407] 5. Non-planar structure assumption: To achieve higher verification accuracy, it is expected that the target has a 3D structure, such as a person, box, or other object. The system assumes that the target has at least a minimum 3D volume, rather than a planar object such as a screen or wall.

[0408] When the above assumptions are met and both cameras observe the same target, the calibration accuracy is evaluated based on the following three criteria:

[0409] 1. Determine whether the cross-camera volume deviation value is within a preset first threshold. This deviation value is the difference between the volume of the first 3D bounding box at one angle and the volume of the first 3D bounding box at another angle. The volume deviation reflects the consistency of camera intrinsic parameters and depth reconstruction. If the relative difference is within the threshold, such as 10%, it is considered that the two cameras reconstruct the same physical target with similar dimensions.

[0410] 2. Determine whether the cross-camera position deviation value is within a preset second threshold. This deviation value is the difference between the center position of the first 3D bounding box at one angle and the center position of the first 3D bounding box at another angle. The smaller this deviation value, the more accurate the multi-camera extrinsic parameter calibration, and consequently, the more reliable the cross-camera intersection-union ratio calculation. By transforming the center of the bounding box under one camera to the coordinate system of another camera through extrinsic parameters, the Euclidean distance to the directly observed center is calculated. If this distance is within the threshold (e.g., 3 cm), it indicates that the extrinsic parameter translation has not drifted.

[0411] 3. Determine whether the ratio of the cross-camera projection points exceeds a preset third threshold. This projection point is calculated by projecting 3D points from one angle's first 3D bounding box onto another angle's first 3D bounding box, and then counting the proportion of points falling within the bounding box. If this ratio exceeds the threshold, for example, 80%, then the rotation relationship between the two cameras is considered consistent, and the target structure observations are considered to match.

[0412] Using the above three criteria, the judgment logic is as follows: if all criteria are within the set limits, the calibration is deemed valid and the system continues to operate normally; if one or two criteria exceed the threshold, a calibration drift warning is triggered and an exception log is recorded; if all three criteria exceed the threshold, the calibration is deemed invalid, and the system will exclude the affected camera, for example, by suspending the use of the camera or automatically triggering a recalibration process, such as prompting the user to show their hand posture for recalibration.

[0413] In this scheme, the accuracy of multi-camera calibration parameters is continuously monitored through a real-time calibration verification function to ensure the geometric accuracy of cross-view fusion. This verification method is based on a core observation: when the fields of view of two cameras overlap and they simultaneously observe the same tracked target, the three-dimensional geometric information of the target should be consistent between the two cameras.

[0414] Unlike traditional methods that require additional image acquisition or physical calibration targets, this invention eliminates the need for additional image acquisition or physical targets. By utilizing the tracking target itself as a natural calibration target, it achieves zero-cost, continuous monitoring of external parameter drift, thereby ensuring the geometric reliability and tracking accuracy of the entire multi-camera system during long-term operation.

[0415] Example 2

[0416] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target object tracking method provided in any of the above embodiments.

[0417] Example 3

[0418] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described target object tracking method.

[0419] In some embodiments of this disclosure, the readable storage medium and computer program product may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0420] Example 4

[0421] This disclosure also provides a computer program that, when executed by one or more processors, can implement the above-described target object tracking method.

[0422] The program code for executing the computer program product disclosed herein can be written in any combination of one or more programming languages. The program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0423] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A tracking method of a target object, characterized by, The method includes: At least two image sequences are obtained by simultaneously capturing images from at least two cameras located at different angles, wherein the target object includes a first target object and a new target object, and each image sequence in the at least two image sequences includes a first image frame corresponding to the first target object and a current image frame corresponding to the new target object; Based on the at least two image sequences, obtain the first mask information of the first target object at the time of the first image frame and the third mask information of the new target object at the time of the current image frame; Based on the first mask information, predict the second mask information of the first target object at the current image frame as the end time, with the first image frame as the start time. Based on the similarity between the second mask information and the third mask information, it is determined whether the new target object matches the first target object from at least one camera angle.

2. The tracking method of a target object according to claim 1, wherein The method further includes: Based on the third mask information, the fourth mask information of the new target object is predicted with the current image frame as the start time and the first image frame as the end time. Based on the similarity between the first mask information and the fourth mask information, it is determined whether the new target object matches the first target object from at least one camera angle.

3. The tracking method of a target object according to claim 1 or 2, characterized by, The prediction step employs a memory-based mask propagation model, wherein the mask propagation model includes at least one of XMem and Cutie.

4. The tracking method of a target object according to Claim 1, wherein The step of determining whether the new target object matches the first target object at at least one camera angle includes: Obtain the similarity between the second mask information and the third mask information at each camera angle, and use it as the first similarity at each camera angle; In response to the first similarity of a first number of camera angles in the at least two cameras exceeding the single-camera matching determination threshold, and the first similarity of a second number of camera angles not exceeding the single-camera matching determination threshold, the first similarity under each camera angle is weighted and fused to obtain a fusion confidence score. Based on the comparison result between the fusion confidence score and the global fusion determination threshold, it is determined whether the new target object matches the first target object.

5. The target object tracking method as described in claim 4, characterized in that, The weighted fusion of the first similarity at each camera angle includes: For any camera angle, the weight of the camera angle is determined based on the magnitude of the first similarity at that camera angle, wherein the higher the first similarity, the greater the weight. Based on the weights, the first similarity at each camera angle is weighted and summed to obtain the fusion confidence score.

6. The target object tracking method as described in claim 1, characterized in that, The method includes: Based on the at least two image sequences, detect the new target object and obtain the third category, third two-dimensional bounding box and third mask information of the new target object; Determine a third 3D bounding box of the new target object at each camera angle, the third 3D bounding box being located in the unified world coordinates of the at least two cameras; Based on the intersection-union ratio of the first 3D bounding box and the third 3D bounding box of the first target object, as well as the categories of the first target object and the new target object, it is determined whether the new target object matches the first target object.

7. The target object tracking method as described in claim 1, characterized in that, The method includes: Based on the at least two image sequences, the first target object is detected and the first object category, first two-dimensional bounding box and first mask information of the first target object are obtained; Obtain the depth map of the first target object; Based on the depth map and the first two-dimensional bounding box, the local three-dimensional bounding box of the first target object at each camera angle is determined, and the local three-dimensional bounding box is located in the local coordinates of the camera. Based on the local 3D bounding box, update the prediction speed and prediction local 3D bounding box of the first target object at the first image frame time obtained by Kalman filtering.

8. The target object tracking method as described in claim 7, characterized in that, The method includes: Based on the local 3D bounding box, a first 3D bounding box of the first target object is determined at each camera angle, and the first 3D bounding box is located in the unified world coordinates of the at least two cameras; Based on the cross-camera intersection-over-union ratio of the first 3D bounding box, it is determined whether the first target object from different camera angles matches.

9. The target object tracking method as described in claim 8, characterized in that, The method includes: Using the cross-camera intersection-union ratio as a weight, the first three-dimensional bounding box of the first target object at each camera angle is fused to form a fused three-dimensional bounding box.

10. The target object tracking method as described in claim 9, characterized in that, The method includes: The target object includes a second target object, and each of the at least two image sequences includes a first image frame corresponding to the first target object and a second image frame corresponding to the second target object; The method includes: Based on the fused 3D bounding box, Kalman filtering is used to predict the velocity of the first target object and the predicted 3D bounding box of the first target object at the second image frame time. Determine a second 3D bounding box of the second target object at each camera angle, wherein the second 3D bounding box is located in the unified world coordinates of the at least two cameras; Based on the intersection-union ratio of the predicted 3D bounding box and the second 3D bounding box, it is determined whether the first target object matches the second target object.

11. The target object tracking method as described in claim 8, characterized in that, Determining the first three-dimensional bounding box of the first target object at each camera angle includes: Obtain the depth map of the first target object and the camera intrinsic parameters of the at least two cameras; Based on the depth map and the camera intrinsic parameters, the pixels in the first mask information are back-projected onto the camera coordinate system to generate a three-dimensional point set, so as to construct the local three-dimensional bounding box of the first target object; Obtain the camera extrinsic parameters of the at least two cameras; Based on the camera extrinsic parameters, the local 3D bounding box is transformed into a unified world coordinate system to obtain the first 3D bounding box.

12. The target object tracking method as described in claim 11, characterized in that, The process of obtaining the camera extrinsic parameters of the at least two cameras includes: Images of the same hand gesture are simultaneously acquired by at least two cameras; Using a deep learning-based hand joint detection network, the three-dimensional coordinates of each hand joint in each camera coordinate system are calculated; Match the 3D coordinate point pairs of the same hand joint in different cameras to construct the corresponding point set; Based on the corresponding point set, the first rotation matrix and the first translation vector between the camera pairs are solved by singular value decomposition and used as the first camera extrinsic parameters. Based on the reference camera, the initial camera extrinsic parameters are unified to the unified world coordinates to obtain the first unified camera extrinsic parameters.

13. The target object tracking method as described in claim 12, characterized in that, The process of obtaining the camera extrinsic parameters of the at least two cameras also includes: The random sampling consensus algorithm is used to remove outlier point pairs in the corresponding point set, and the remaining point pairs are used to resolve the first rotation matrix and the first translation vector to obtain the second camera extrinsic parameters. The second camera extrinsic parameters include the second rotation matrix and the second translation vector. Based on the reference camera, the second camera extrinsic parameters are unified to the unified world coordinates to obtain the second unified camera extrinsic parameters.

14. The target object tracking method as described in claim 13, characterized in that, The process of obtaining the camera extrinsic parameters of the at least two cameras also includes: Using the three-dimensional joint coordinates of different hand poses and positions in multiple frames, the second rotation matrix and the second translation vector are iteratively optimized to obtain the third camera extrinsic parameters, which include the third rotation matrix and the third translation vector. Based on the reference camera, the extrinsic parameters of the third camera are unified to the unified world coordinates to obtain the third unified camera extrinsic parameters.

15. The target object tracking method as described in claim 11, characterized in that, The method further includes: Based on the depth map, at least one of the following is performed to verify the accuracy of the real-time calibration of the at least two cameras: Determine whether the cross-camera volume deviation value is within a preset first threshold, wherein the cross-camera volume deviation value is the difference between the volume of the first 3D bounding box at one angle among the different angles and the volume of the first 3D bounding box at another angle among the different angles; and, Determine whether the cross-camera position deviation value is within a preset second threshold, wherein the cross-camera position deviation value is the difference between the center position of the first 3D bounding box at one of the different angles and the center position of the first 3D bounding box at another of the different angles; and, Determine whether the ratio of the cross-camera projection point exceeds a preset third threshold, wherein the cross-camera projection point is the projection of the first three-dimensional bounding box point of one of the different angles onto the first three-dimensional bounding box of another of the different angles.

16. A target object tracking system, characterized in that, The system includes a processor for executing the target object tracking method as described in any one of claims 1 to 15.

17. The target object tracking system as described in claim 16, characterized in that, The system also includes: A memory for storing at least two image sequences captured simultaneously by the at least two cameras.

18. The target object tracking system as described in claim 17, characterized in that, in, The processor includes a central processing unit or a graphics processing unit; The memory includes a shared memory region of the central processing unit or the image processor, and the shared memory region is used to store at least two image sequences obtained by the at least two cameras simultaneously.

19. The target object tracking system as described in claim 17, characterized in that, The system also includes: At least two cameras, wherein the at least two cameras are used to simultaneously capture at least two image sequences; The at least two cameras include at least one of a monocular RGB camera, a binocular RGB camera, and a time-of-flight camera.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the target object tracking method according to any one of claims 1 to 15.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the target object tracking method as described in any one of claims 1 to 15.