A dual-camera gimbal camera cooperative multi-target tracking method and device

Abstract

The application discloses a dual-camera gimbal camera cooperative multi-target tracking method and device. The method comprises the following steps: acquiring a video image collected by a dual-camera gimbal camera, and identifying a moving target based on a motion entropy driven adaptive region optical flow processing; determining the rotation and zoom of each gimbal camera by using a minimum enclosing circle field of view control method containing a dynamic safety boundary and a motion smoothing constraint according to the target position, size and motion state; performing cross-node target association based on target information and appearance features to maintain the identity consistency of the target during switching; modeling the dual-camera gimbal cooperative tracking process as a decentralized partially observable Markov decision process, and training a cooperative scheduling strategy. The method comprehensively optimizes target tracking coverage, tracking stability, node load balancing, system energy consumption and decision-making delay, and can realize continuous, stable and low-latency cooperative tracking of multiple targets in a complex dynamic scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent video surveillance and multi-target tracking technology, and in particular to a method and device for multi-target tracking using a dual-camera PTZ camera. Background Technology

[0002] Pan-tilt cameras, also known as PTZ cameras, are camera devices capable of horizontal rotation, pitch rotation, and focal length adjustment. Because they can flexibly adjust the observation direction and field of view over a large spatial range, they are widely used in scenarios such as public safety, smart cities, park management, traffic monitoring, industrial production, and elderly care. Compared to fixed cameras, PTZ cameras can actively track and magnify details of targets of interest, thereby improving the monitoring system's ability to perceive key targets.

[0003] Existing single-gimbal camera tracking solutions typically control the gimbal's rotation and zoom based on the target's position in the image to keep the target within the field of view. However, a single gimbal camera has limitations in its field of view, rotation speed, zoom capability, and computing resources: when multiple targets appear simultaneously in the field of view, the camera struggles to simultaneously monitor the global scope and local target details; when the camera increases its focal length to obtain target details, other targets may easily leave the field of view; and when different targets move in opposite directions, a single gimbal cannot effectively track them simultaneously. Therefore, single-node solutions are prone to problems such as missed target detection, target loss, insufficient tracking time, and frequent field-of-view jitter.

[0004] To expand the monitoring range, existing technologies typically employ multiple cameras to form a collaborative monitoring system. For example, wide-angle cameras are used to provide a global view, while PTZ cameras are used for detailed local tracking, or multiple PTZ cameras are used to assign tasks to different areas and targets. However, existing multi-camera collaborative solutions are often based on preset rules, fixed priorities, or centralized scheduling. When the number of targets, direction of movement, occlusion, and node load change dynamically, fixed rules are difficult to adapt to complex scenarios; while centralized scheduling requires continuous collection of video data and status information from multiple nodes, which can easily lead to increased communication bandwidth consumption, increased decision-making delays, and reduced real-time performance and reliability when the system is deployed at the edge.

[0005] Furthermore, multi-gimbal collaborative tracking also faces the technical challenge of ensuring target identity consistency across nodes. The same target may have different imaging angles, resolutions, lighting conditions, and scales on different cameras, and there are often issues such as insufficient field of view overlap, short-term occlusion, or time asynchrony during target handover. Relying solely on proximity or a single appearance feature for target matching can easily lead to identity switching, repeated tracking, or handover failures, thereby affecting subsequent task allocation and gimbal control.

[0006] In terms of PTZ field-of-view control, existing technologies are mostly designed for single targets, placing the target in the center of the image or adjusting the focal length according to a fixed ratio. For multi-target scenarios, simply adjusting the field of view based on the farthest boundary of all targets can easily lead to problems such as excessive zooming, frequent zooming, or targets being too close to the edge; if only focusing on a few high-priority targets, other targets may be caused to fall out of the monitoring range. Therefore, there is an urgent need for a PTZ field-of-view control solution that can ensure multi-target coverage while also maintaining a compact field of view and smooth control.

[0007] At the level of collaboration among multiple dual-camera PTZ cameras, existing distributed solutions typically use traditional single-lens PTZ cameras as each node. This inevitably leads to inherent problems with single-lens PTZ cameras, such as narrow field of view, the trade-off between field of view and detail, and high energy consumption. These issues, to some extent, reduce the overall effectiveness of the distributed PTZ camera system. Although dual-camera PTZ cameras, at the individual device level, resolve the trade-off between global field of view and local detail through heterogeneous collaboration between wide-angle / panoramic cameras and high-precision PTZ zoom cameras, forming an intelligent, distributed cluster of multiple dual-camera PTZ cameras still faces three core challenges: First, collaborative decision-making under partially observable conditions. Each node can only acquire target information within its local field of view through its own wide-angle lens, resulting in a partial and incomplete understanding of the global situation. How to make globally optimal tracking resource allocation decisions under conditions of localized information is the primary theoretical challenge of distributed collaboration; secondly... It is a resource optimization allocation under multiple objective constraints, that is, the system needs to find a balance among multiple conflicting objectives such as tracking coverage, system energy consumption, load balancing and tracking stability. This is essentially a dynamic optimization problem with multiple objectives and constraints. Finally, the dual-camera gimbal distributed collaborative system will face the problem of suppressing the transmission of perception uncertainty to the control link. The target detection of the wide-angle lens has noise, missed detection and positioning error. These uncertainties will be directly transmitted to the decision layer and control layer. How to suppress the impact of perception noise and ensure the robustness of system behavior in the decision and control process is the key to realizing a high-efficiency and highly available distributed system.

[0008] In summary, existing technologies still lack a dual-camera gimbal collaborative multi-target tracking solution for complex dynamic scenarios that can comprehensively address issues such as lightweight target perception, multi-target field-of-view adaptive control, cross-node target identity association, noise recognition and suppression, dynamic task allocation, and low-latency collaborative decision-making under conditions of partial observability and limited communication and computing resources. Summary of the Invention

[0009] The purpose of this invention is to provide a method and device for multi-target tracking using dual-camera gimbals, which solves the problem in existing gimbal camera multi-target tracking technologies where a single gimbal cannot simultaneously handle large-scale global monitoring and local target detail tracking; and where target loss easily occurs when multiple targets move in different directions.

[0010] In a first aspect, the present invention provides a method for collaborative multi-target tracking using dual-camera PTZ cameras, applicable to a monitoring system with at least two dual-camera PTZ cameras. Each dual-camera PTZ camera includes a fixed-direction wide-angle lens and a rotatable zoom lens. The method includes the following steps:

[0011] S1. Obtain the difference image between the current video frame and the previous video frame captured by each wide-angle lens, calculate the motion entropy based on the difference image, divide the difference image into grids, perform optical flow calculation on the foreground pixels within each grid to obtain the target motion vector, and dynamically adjust the grid division granularity and detection threshold to continuously identify the target and output target information; the target information includes: target position, scale and motion state.

[0012] S2. Determine the set of targets to be tracked based on the target information and decision instructions, solve for the minimum enclosing circle covering all targets to be tracked, and superimpose a dynamic safety margin on the radius of the minimum enclosing circle to obtain the target field of view circle. Generate control instructions for each gimbal camera based on the center offset and radius of the target field of view circle; the control instructions include: rotation and zoom of the gimbal camera.

[0013] S3. After triggering the target handover, the source node sends a handover request to the target node. The target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features. Both the source node and the target node include dual-camera gimbal cameras.

[0014] S4. Model the collaborative tracking process of multiple dual-camera gimbal cameras as a distributed partially observable Markov decision process, and output the decision instructions to step S2 to execute target tracking; the decision instructions include at least one of the following: target allocation, continue tracking, release target, hand over target, and priority adjustment.

[0015] The process of calculating motion entropy based on the difference image and dividing the difference image into a grid includes:

[0016] In the initialization phase, motion statistical features of each region are collected using a uniform grid division to establish an initial estimate of the motion entropy of each region. Based on the distribution of motion pixels in the differential image, the differential image is divided into multiple initial grid regions. The motion statistical features include: target pixel density, motion intensity standard deviation, and velocity variation coefficient.

[0017] For each region, the region motion entropy is calculated, which is obtained by weighting the motion intensity standard deviation, velocity variation coefficient, and target pixel density.

[0018] During the adaptive operation phase, the motion vector of the foreground target pixels in each region is calculated using the pyramid Lucas-Kanade optical flow algorithm to dynamically obtain the target contour.

[0019] The method further includes: using a first granularity grid when the motion entropy is higher than a high threshold, using a second granularity grid when it is between the high threshold and a low threshold, and using a third granularity grid when it is lower than the low threshold, wherein the first granularity grid is smaller than the second granularity grid, and the second granularity grid is smaller than the third granularity grid.

[0020] The method further includes: dynamically adjusting the region division granularity and detection threshold through a two-layer feedback adjustment mechanism based on optical flow quality index; wherein the two-layer feedback adjustment mechanism includes a short-term performance monitoring layer and a long-term pattern learning layer; the short-term performance monitoring layer calculates the tracking quality index of each region in real time, and triggers the mesh granularity refinement operation of the region when the tracking quality index of a certain region is lower than the preset quality threshold; the long-term pattern learning layer detects whether the region motion pattern is consistent with the historical record, and performs a region reorganization operation when the region motion pattern changes.

[0021] The dynamic safety margin is obtained by weighted summation of components in four dimensions: Δr t = β1Δr dyn + β2Δr trend + β3Δr err + β4Δr scene .

[0022] Where: β1, β2, and β3 are non-negative weights and their sum is 1, Δr dyn Δr is the dynamic prediction component calculated based on the target maximum velocity and system time delay. trend Let Δr be the component representing the relative motion trend between targets. err For the error component, Δr scene For scene components.

[0023] The target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features, including:

[0024] The target node uses a target motion model to predict its expected position at the current moment, based on the target's last observation time and position contained in the handover request, and to compensate for communication delays.

[0025] Traverse all target observations in the current wide-angle field of view of the target node, calculate the spatial distance between each target observation and the predicted position, and retain candidate targets whose spatial distance is not greater than the maximum distance and whose motion direction difference is not greater than a preset direction threshold.

[0026] Calculate the overall matching score for each candidate target;

[0027] If the highest comprehensive matching score is not lower than the high score threshold, the corresponding candidate target is determined to be the transfer target, and the handover is confirmed.

[0028] If the highest overall matching score is between the high score threshold and the low score threshold, then the transient confirmation stage is entered, and the overall matching score is continuously tracked and recalculated within the subsequent preset number of frames.

[0029] If the highest overall matching score is lower than the low score threshold, the handover request information will be stored in the observation queue and matching will be continuously attempted within the preset future frames. If the matching is still unsuccessful after the maximum waiting time, a handover failure notification will be sent to the source node.

[0030] The Markov decision process includes:

[0031] Each dual-camera gimbal camera uploads local observation information to its respective edge node. Each edge node runs the trained policy network to perform batch inference and outputs decision instructions. Each dual-camera gimbal camera then calls step S2 to perform specific tracking based on the decision instructions.

[0032] The method further includes:

[0033] When communication between the dual-camera gimbal and the edge node is interrupted, it automatically switches to a degraded operation mode, pauses and waits for decision instructions from the edge node, and continues to track the current target using preset rules.

[0034] Secondly, the present invention provides a dual-camera PTZ camera collaborative multi-target tracking device, applied to a monitoring system with at least two dual-camera PTZ cameras. Each dual-camera PTZ camera includes a fixed-direction wide-angle lens and a rotatable zoom lens, comprising:

[0035] The moving target detection module is used to obtain the difference image between the current video frame and the previous video frame captured by each wide-angle lens, calculate the motion entropy based on the difference image, divide the difference image into a grid, perform optical flow calculation on the foreground pixels within each grid to obtain the target motion vector, and dynamically adjust the grid division granularity and detection threshold to continuously identify targets and output target information; the target information includes: target position, scale, and motion state;

[0036] The control command generation module is used to determine the set of targets to be tracked based on the target information and decision commands, solve the minimum enclosing circle covering all targets to be tracked, superimpose a dynamic safety margin on the radius of the minimum enclosing circle to obtain the target field of view circle, and generate control commands for each gimbal camera based on the center offset and radius of the target field of view circle; the control commands include: rotation and zoom of the gimbal camera;

[0037] The tracking module is used to send a handover request to the target node after the source node triggers the target handover. The target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features. Both the source node and the target node include dual-camera gimbal cameras.

[0038] The decision instruction generation module is used to model the collaborative tracking process of multiple dual-camera gimbal cameras as a distributed partially observable Markov decision process, and output the decision instructions to step S2 to execute target tracking; the decision instructions include at least one of the following: target allocation, continue tracking, release target, hand over target, and priority adjustment.

[0039] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the dual-camera gimbal-based multi-target tracking method as described in any of the preceding claims. The method comprehensively optimizes target tracking coverage, tracking stability, node load balancing, system energy consumption, and decision latency, enabling continuous, stable, and low-latency collaborative tracking of multiple targets in complex dynamic scenarios. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating a dual-camera gimbal-based multi-target tracking method according to an embodiment of the present invention.

[0041] Figure 2 This is a strategy network structure diagram according to an embodiment of the present invention;

[0042] Figure 3 This is a diagram of a distributed execution architecture according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of a dual-camera gimbal-based multi-target tracking device according to an embodiment of the present invention. Detailed Implementation

[0044] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0045] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0046] The embodiment of this invention takes two dual-camera gimbal camera nodes as an example, denoted as node A and node B respectively. Each node includes a dual-camera gimbal camera, an edge computing unit, a communication unit, and a gimbal control unit. The dual-camera gimbal camera is used to acquire video data; the edge computing unit is used to perform target detection, tracking, association, and strategy reasoning; the communication unit is used to exchange structured information; and the gimbal control unit is used to drive the dual-camera gimbal camera to perform horizontal rotation, pitch rotation, and focus adjustment.

[0047] The monitoring system does not require continuous transmission of raw video frames. Each node prioritizes sending only structured information such as the target task list, local target identifiers, position and velocity, compact appearance features, global identifiers, node load, and handover requests. Each node can be configured with a maximum simultaneous tracking capacity K. max For example, 3; each decision cycle can process D sel There are 5 candidate targets, for example.

[0048] Specifically, this invention provides a method for collaborative multi-target tracking using dual-camera PTZ cameras, applied to a monitoring system with at least two dual-camera PTZ cameras. Each dual-camera PTZ camera includes a fixed-direction wide-angle lens and a rotatable zoom lens, such as... Figure 1 As shown, the method includes the following steps:

[0049] S1. Obtain the difference image between the current video frame and the previous video frame captured by each wide-angle lens. Calculate the motion entropy based on the difference image, and divide the difference image into a grid. Perform optical flow calculation on the foreground pixels within each grid to obtain the target motion vector. Dynamically adjust the grid division granularity and detection threshold to continuously identify the target and output target information. The target information includes: target position, scale, and motion state. Specifically, the wide-angle lenses of each dual-camera gimbal continuously capture video images, and the current video frame F... t Compared to the previous video frame F t-1 Difference is performed to obtain the difference image D. t D t = |F t - F t-1The motion entropy of each region is calculated based on the distribution of moving pixels in the differential image. The differential image is then adaptively divided into non-uniform grids based on the motion entropy. Optical flow is calculated on the foreground pixels in each divided region to obtain the target motion vector. The region division granularity and detection threshold are dynamically adjusted through a two-layer feedback adjustment mechanism based on optical flow quality index, thereby continuously identifying moving targets within the field of view with low computational overhead.

[0050] S2. Based on the target information and decision instructions, determine the set of targets to be tracked, solve for the minimum bounding circle covering all targets to be tracked, and superimpose a dynamic safety margin on the radius of the minimum bounding circle to obtain the target field of view circle. Based on the center offset and radius of the target field of view circle, generate control instructions for each gimbal camera; the control instructions include: rotation and zoom of the gimbal camera. Specifically, based on the target position, scale, and motion state of multiple targets identified in step S1, and according to the target allocation result output in step S4, determine the set of targets to be tracked for this node, solve for the minimum bounding circle covering all targets to be tracked for this node, and superimpose a dynamic safety margin determined by target dynamics prediction, relative motion trend between targets, sensor and positioning errors, and scene priority on the radius of the minimum bounding circle to obtain the target field of view circle. Convert the center offset and radius of the target field of view circle into the horizontal rotation angle, pitch angle, and zoom parameters of the gimbal, respectively, and apply motion smoothing constraints of update threshold and low-pass filtering to generate rotation and zoom control instructions for each gimbal camera.

[0051] S3. After triggering target handover, the source node sends a handover request to the target node. The target node performs spatiotemporal filtering based on motion prediction to filter candidate targets, and then performs precise matching based on compact appearance features. Both the source node and the target node include dual-camera gimbal cameras. Specifically, when a target is about to leave the current node's field of view, the current node is overloaded, or an adjacent node has a better viewpoint, target handover is triggered. At this time, the source node sends a handover request to the target node containing the target's observation time, position, speed, direction of motion, compact appearance features, and global identifier. The target node performs spatiotemporal filtering based on motion prediction to filter candidate targets, then performs precise matching based on compact appearance features, and binds a globally unique identifier to the matched targets to maintain the consistency of the target's identity when switching between different gimbal cameras.

[0052] S4. The collaborative tracking process of multiple dual-camera gimbals is modeled as a distributed partially observable Markov decision process, and the decision instructions are output to step S2 to execute target tracking. The decision instructions include at least one of the following: target allocation, continue tracking, release the target, hand over the target, and priority adjustment. Specifically, based on the distributed partially observable Markov decision process, a multi-objective team reward function integrating tracking coverage, system energy consumption, load balancing, and tracking stability is designed. A multi-agent proximal policy optimization algorithm is used to train the collaborative scheduling strategy in a centralized training and distributed execution mode. In actual operation, each dual-camera gimbal camera node uploads its local observation information to its corresponding edge node. Each edge node runs the trained policy network to perform batch inference and outputs decision instructions for target allocation, continue tracking, release the target, hand over the target, and priority adjustment. Each node executes specific tracking by calling the field-of-view adaptive control in step S2 according to the decision instructions. It should be noted that step S4 is not strictly executed sequentially with steps S1 to S3, but rather runs in parallel and periodically: the identification result of step S1 and the association state of step S3 serve as the decision input for step S4, and the decision command output by step S4 drives step S2 to perform vision adaptive control. The steps are thus cyclically connected to form a closed loop of perception-association-decision-control, completing the continuous collaborative tracking of multiple targets.

[0053] In a preferred embodiment, calculating motion entropy based on the difference image and dividing the difference image into a grid includes:

[0054] During the initialization phase, i.e., the first N... init Frame, N init To determine the preset initialization frame count, motion statistical features of each region are collected using a uniform grid, an initial estimate of the motion entropy of each region is established, and the differential image is divided into multiple initial grid regions based on the distribution of moving pixels in the differential image. The motion statistical features include: target pixel density ρ. θ σ, the standard deviation of exercise intensity t and velocity variation coefficient C v Based on this, an initial estimate of the motion entropy of each region is established; after initialization, the system enters the adaptive operation phase.

[0055] For each region, the region motion entropy is calculated, which is obtained by weighting the motion intensity standard deviation, velocity variation coefficient, and target pixel density: E m (t) = λ1σ t + λ2C v + λ3ρ θ , where σ t C represents the standard deviation of exercise intensity within the region. v ρ is the velocity variation coefficient. θTo satisfy the target pixel density under the preset motion conditions, λ1, λ2, and λ3 are non-negative weights and their sum is 1.

[0056] During the adaptive operation phase, the motion vector of the foreground target pixel (x, y) in each region is calculated using the pyramid Lucas-Kanade optical flow algorithm. =(ⅾ x , ⅾ y This allows for the dynamic acquisition of the target outline, avoiding redundant calculations of background pixels.

[0057] According to the target contour completeness S c Effective feature retention level R f and the continuity of the target boundary F b Calculate the tracking quality index Q for each region r Q r = ω1S c + ω2R f + ω3F b ω1, ω2, and ω3 are the non-negative weights of the three, and ω1+ω2+ω3=1. The values ​​can be determined according to the focus of the scenario, with ω1=0.4, ω2=0.35, and ω3=0.25 being preferred.

[0058] In a preferred embodiment, the method further includes: using a first granularity grid when the motion entropy is higher than a high threshold, using a second granularity grid when it is between the high and low thresholds, and using a third granularity grid when it is lower than the low threshold, wherein the first granularity grid is smaller than the second granularity grid, and the second granularity grid is smaller than the third granularity grid. Specifically, a smaller grid, such as 8×8 pixels, is used when the motion entropy is higher than the high threshold; a 16×16 to 32×32 pixel grid is used when it is between the high and low thresholds; and a larger grid, such as no more than 64×64 pixels, is used when it is lower than the low threshold.

[0059] In a preferred embodiment, the method further includes: dynamically adjusting the region division granularity and detection threshold through a two-layer feedback adjustment mechanism based on optical flow quality indicators; wherein the two-layer feedback adjustment mechanism includes a short-term performance monitoring layer and a long-term pattern learning layer; the short-term performance monitoring layer calculates the tracking quality indicators of each region in real time, and when the tracking quality indicator of a certain region is lower than a preset quality threshold, it triggers a mesh granularity refinement operation for that region to improve the tracking accuracy of the problem region; the long-term pattern learning layer detects whether the region motion pattern is consistent with the historical record, and performs a region reorganization operation when the region motion pattern changes.

[0060] Specifically, the system saves the historical records of quality indicators for the most recent K moments and calculates the quality fluctuation ΔQ. vol = 1 / (K-1) · ∑ k=1K-1 |Q r (t-k) - Q r (t-k-1) | and dynamically adjust the detection threshold: T a = T base · (1 - η·ΔQ vol ), where t is the current time, k is the number of moments since the current time, and Q r To track quality metrics, T base To adjust the baseline interval, η is the mass fluctuation sensitivity coefficient (η≥0), used to control the mass fluctuation amount ΔQ. vol Regarding the adjustment strength of the adjustment interval: the larger the η, the more sensitive the adjustment interval is to the tracking quality fluctuations; when η=0, the adjustment interval degenerates into a fixed value T. base To avoid excessively frequent regional recombination due to excessively small adjustment intervals when quality fluctuates drastically, it is preferable to adjust η·ΔQ. vol The amplitude is limited to the range of [0, 0.8], meaning the minimum adjustment interval is no less than 0.2T. base The long-term pattern learning layer maintains the historical record of regional motion patterns H={E}. m (t -k) Q r (t-k)} (k=0,…,K-1), where E m The region's motion entropy. When a systematic change in the region's motion pattern is detected, such as a continuous increase in motion entropy or a continuous decrease in tracking quality, a region reorganization operation is performed. The region reorganization follows the principle of minimum disturbance, prioritizing the adjustment of the problem region and its neighborhood, while maintaining the spatiotemporal continuity of the overall partition structure.

[0061] When the tracking quality is stable, ΔQ vol Approaching 0, the adjustment interval is close to T. base When the mass fluctuates drastically, ΔQ vol The interval between adjustments is shortened accordingly to increase the reorganization frequency and respond quickly to scene changes. An adaptive partitioning mechanism driven by motion entropy concentrates computational resources in high motion complexity regions, reducing edge perception overhead while ensuring target contour integrity. A dual-layer feedback adjustment mechanism of short-term performance monitoring and long-term pattern learning layers adjusts region partitioning and detection thresholds, forming a closed-loop, lightweight continuous perception process. This ensures target contour integrity while suppressing false detections and missed detections caused by factors such as sudden changes in illumination, achieving an adaptive balance between computational accuracy and resource consumption.

[0062] In a preferred embodiment, for m targets to be tracked within the current node's field of view, the center point, boundary point, or detection box vertex of each target is obtained to form a target point set Q. tAnd find the smallest enclosing circle C(c t , r t The minimum enclosing circle satisfies: q i (t) Q t , ||q i (t) - c t || ≤ r t Among them, q i (t) For Q t For any node in c, t and r t C(c) t ,r t The center and radius of the circle.

[0063] The preferred method is the stochastic incremental method to solve for the minimum enclosing circle. Considering target motion, detection error, gimbal control error, and processing delay, a dynamic safety margin Δr is constructed. t The target field of view circle, with radius r *t = r t + Δr t .

[0064] In a preferred embodiment, the dynamic safety margin Δr t Δr is obtained by weighted summation of the components in the four dimensions. t =β1Δr dyn + β2Δr trend + β3Δr err + β4Δr scene , where β1, β2, and β3 are non-negative weights and their sum is 1.

[0065] Δr dyn The predicted displacement is δ, which is the dynamic prediction component calculated based on the target maximum velocity and system time delay. i = α · v max i · τ, where v max i Let τ be the target's maximum velocity, τ be the total system time delay, and α be a safety factor (taken as α > 1) to compensate for uncertainties in the target's sudden acceleration, turning, and other motions, as well as model prediction errors, ensuring that the predicted displacement covers the target's actual displacement within the system time delay. The predicted displacement δ for all targets is taken as... i The maximum value is used as the dynamic prediction component.

[0066] Δr trendThis represents the relative motion trend component between targets, reflecting the relative motion trend of the targets, such as moving away or closer. Calculate the target point set Q. t The convex hull is used to monitor the movement pattern of its vertices in recent frames. If vertices generally move outward, the margin is increased; if they move inward, the margin can be reduced appropriately.

[0067] Δr err This is the error component, composed of detection error, gimbal pointing error, and zoom error. Let 2σ be the value. total (95% confidence interval) is used as the error component, where the overall uncertainty is: ,in, To detect error, This refers to the gimbal pointing error. This is the zoom error.

[0068] Δr scene The scene components reflect the importance of the target, edge risk, or occlusion risk. The horizontal and vertical components, whose center is off-center from the image, are converted into the pan / tilt and tilt angles of the gimbal, respectively, and the target's field of view radius is converted into the required focal length or field of view. Update thresholds and low-pass filtering constraints can be applied to the control values ​​of the scene components to avoid frequent fine-tuning and achieve a motion smoothing mechanism.

[0069] This invention employs a field-of-view control method that combines the minimum enclosing circle with dynamic safety margin, taking into account target coverage, image compactness, zoom stability, and anti-latency capability. It formalizes the multi-target tracking problem into a minimum enclosing circle solution in computational geometry, and combines it with the gimbal kinematics model to achieve automatic field-of-view wrapping and smooth tracking of multiple assigned targets by a single zoom lens.

[0070] In a preferred embodiment, node A is the source node and node B is the target node. When node A determines that the following situations occur, it triggers a handover request: the target is about to leave the field of view, the current node is overloaded, or node B has a better viewpoint, node A sends a handover request to node B. The handover request includes at least one of the following: the target observation time, position, velocity, direction of movement, compact appearance features, and original global identifier. Specifically, the triggering conditions for triggering the handover request include: (1) the target's predicted position exceeds the zoom field of view boundary of this node; (2) the number of targets currently tracked by this node reaches the capacity limit and the new target has higher value; (3) the adjacent node has a better viewpoint or better tracking conditions; (4) the policy network explicitly outputs the target action type of the handover request.

[0071] Preferably, the target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features, including:

[0072] The target node is based on the target's last observation time t included in the handover request. obsand position p obs and the speed of motion v obs Predict its current time t using the target motion model. now Expected position now = p obs + v obs ∙(t now - t obs ).

[0073] Iterate through all target observations in the current wide-angle field of view of the target node and calculate τ for each target observation. k Position p k obs With predicted location k now Spatial distance: d space (τ k ) = ||p k obs - k now ||.

[0074] Set the maximum distance threshold D max = α ∙ ||v obs || ∙ Δt + σ nos , where Δt = t now - t obs α is the safety factor, taken as α > 1, used to compensate for uncertainties in the target's motion such as sudden acceleration and turning, as well as model prediction errors, ensuring that the predicted displacement covers the actual displacement of the target within the system time delay. σ nos This is the standard deviation for location noise. Specifically, it applies to spatial distance d. space Not greater than the maximum distance D max And the difference in direction of motion θ diff Not greater than the preset direction threshold Θ max The candidate targets are retained.

[0075] Preferably, the target image is scaled to 64×64 pixels, and 128-dimensional, L2-normalized appearance features are extracted using MobileNet V2. The appearance similarity is: s k = (f k handover ∙ f k now ) / (||f k handover || ∙ ||f k now ||), where fk handover The target observation τ included in the handover request k The appearance feature vector, f k now For τ k The appearance feature vector in the current wide-angle view of the target node.

[0076] In a preferred embodiment, a comprehensive matching score is calculated for each candidate target. Specifically, appearance similarity, normalized spatial distance, and normalized orientation difference are fused into a comprehensive matching score. k = w s ·s̃ k + w d ·(1−d̃ space ) +w θ ·(1−θ̃ diff ), where s̃ k To normalize appearance similarity, d̃ space For normalized spatial distance, θ̃ diff To normalize the difference in motion direction, w s w d w θ The weights are non-negative and their sum is 1; w is typically set to... s The higher the level, the more prominent the dominant role in highlighting the appearance features.

[0077] The matching decision employs a three-tiered threshold mechanism. If the highest comprehensive matching score is not lower than the high score threshold, the corresponding candidate target is determined as the handover target, and the handover is confirmed. If the highest comprehensive matching score is between the high and low score thresholds, a transient confirmation phase is initiated, continuously tracking and recalculating the comprehensive matching score within a preset number of frames. If the highest comprehensive matching score is lower than the low threshold, the handover request information is stored in an observation queue, and matching attempts continue within a preset number of future frames. If no match is found after the maximum waiting time, a handover failure notification is sent to the source node. Specifically, if the highest score is not lower than the high score threshold... high If the corresponding candidate target is determined to be the handover target, its local temporary identifier is bound to the global identifier of the handover target, and the handover is confirmed; if the highest score is between the high score threshold and the low score threshold... low If the target is between these two points, a transient confirmation phase begins. No global identifier is assigned at this time; the candidate target is marked as a target awaiting confirmation. Tracking continues for the next 3 to 5 frames, and the matching score is recalculated. If the score stabilizes, the match is confirmed. If the highest score is below a low threshold score... lowIf the target to be handed over has not yet appeared in the field of vision of this node, the handover request information is stored in the observation queue and matching is continuously attempted in the next few frames. If the matching is still unsuccessful after the maximum waiting time, it is determined that the target may have been lost and a handover failure notification is sent to the source node.

[0078] If the match is successful, target node B becomes the globally unique identifier in the handover request for that target. After confirmation by the policy network, the target information is added to the tracking task list of this node, and a handover confirmation message containing the confirmation timestamp and the current status of the target is sent to the source node A. After receiving the handover confirmation message, the source node A releases the target and completes the handover.

[0079] In a preferred embodiment, for a target that appears in the system for the first time and cannot be associated with any existing global identifier, a new target identification and identifier allocation process is performed: (1) Local candidate generation: After the node's wide-angle lens detects a new target, it performs a continuous test, requiring the target to appear continuously for no less than 5 frames and the movement speed to exceed a preset threshold in order to filter out instantaneous noise; (2) Local feature library query: The node queries the local short-term feature library that stores the features of targets that have appeared in the most recent preset time period. If a highly similar target is found, its global identifier is used to avoid duplicate allocation; (3) Global identifier application: When there is no match in the local feature library, the node sends an identifier application containing the target's appearance features, the first appearance location and timestamp to the global identifier manager deployed on the edge node through the edge node. The global identifier manager returns a new globally unique identifier and records it in the global registry; (4) Feature library update: The newly allocated identifier and its features are added to the local feature library and the global registry for subsequent association.

[0080] Regarding the lifecycle management of global identifiers, each node maintains an active target table locally, recording the global identifiers of all targets within its current field of view and their latest observations, and periodically reports a summary of the active target status to edge nodes. When a target disappears from the field of view of all nodes for more than a preset time threshold, it is determined that the target has disappeared, and its global identifier is reclaimed by the global identifier manager. If two nodes simultaneously report the discovery of the same target but assign different global identifiers, the edge node initiates a conflict resolution process, compares the historical trajectories and appearance characteristics of the two targets, merges the identifiers when they are determined to be the same target, and notifies the relevant nodes to update.

[0081] To address the uncertainties in perception, such as missed detections and false alarms, when a target briefly disappears from the field of view of a node where it is expected to appear, it is not immediately determined that the target is lost. The receiving node continues to search near the predicted location while retaining the target's global identifier and appearance features, waiting for the target to reappear. The target loss process is only initiated when the target has not appeared for several consecutive frames. For candidate targets with low matching scores but close spatial locations, they are not easily assigned labels but are marked as suspicious targets and their movement trajectories are observed. When multiple candidate targets have similar scores to the handed-over target, no decision is made until the target's movement causes its location to separate or a more unique perspective emerges and more evidence is accumulated before matching is performed.

[0082] In a preferred embodiment, the collaborative tracking process of multiple dual-camera gimbal nodes is modeled as a distributed partially observable Markov decision process (Dec-POMDP): each node acts as an agent, making independent decisions based solely on its local observations, and learning a globally optimized collaborative tracking strategy within a framework of centralized training and distributed execution. The formal definitions of global state, local observations, hierarchical action space, and team reward function are as follows.

[0083] (1) Global State and Local Observation: The global state st is a description of the complete physical reality of the system at time t, which includes three parts: the target state set, the agent state set, and the environment state. Among them, the target state set includes the target identifier, the true 3D position in the world coordinate system, the true instantaneous velocity vector, the target category, and the true appearance features; the agent state set includes the gimbal's current true horizontal angle, pitch angle, focal length, instantaneous power, and the number of targets currently being tracked; the environment state includes variables that affect perception, such as lighting conditions. The global state cannot be directly observed by any single agent in the real system, and is only exposed to the learner by the simulation environment during the centralized training phase to guide policy optimization. Each agent's local observation consists of four information channels: First, target observation within the field of view, each target observation includes the local temporary identifier assigned by the node, observation position, observation speed, compact appearance features, detection confidence, and the number of frames in which the target continuously appears; second, the node's own state, including gimbal attitude, busy / idle status, current load, and instantaneous energy consumption rate; third, historical association information, recording the global identifier of each target currently being tracked by the node, the duration of tracking, and the number of times the target has switched between this node and other nodes, used to maintain tracking stability and avoid unnecessary handovers; and fourth, a summary of the states of neighboring nodes, recording the load and relative position of neighboring nodes, used to support inter-node collaboration.

[0084] (2) Hierarchical Action Space: The action space adopts a fixed-dimensional, hierarchical, and decoupled structure. The agent's decision output is defined as a combination of five components: action type, target selection vector, target release vector, target handover vector, and priority weight vector. The action type consists of six discrete action categories, including keeping idle, requesting a new target, continuing to track, releasing the target, handing over the target, and adjusting the priority. These are obtained by sampling the 6-dimensional probability distribution output by the Softmax function. The target selection vector is D. sel A binary vector of dimension K indicates which candidate targets within the field of view are added to the tracking list; each dimension is sampled by binary classification using an independent sigmoid function; the target release vector is K. max A binary vector indicating which targets in the current tracking list are being released; the target handover vector contains K... max The selection of one-hot handover targets and N-dimensional one-hot handover objects is defined, where N is the total number of dual-camera gimbal camera nodes participating in the collaboration; the priority weight vector is K. max A continuous vector, output by Sigmoid and then normalized by Softmax, indicates the relative priority of each target in the current tracking list, affecting the field-of-view bias of the zoom lens. In this embodiment, K is taken as... max =3、D sel =5; When the actual number of targets is less than the preset maximum capacity, placeholder filling and masking mechanisms ensure that invalid outputs do not participate in environmental interaction. After the action determines the tracking target and its priority, the specific gimbal rotation and zoom control is automatically completed by the field-of-view control module based on the minimum bounding circle, forming a hierarchical control structure that separates high-level scheduling decisions from low-level servo control.

[0085] (3) Team Reward Function: The design of the reward function follows three principles: multi-objective fusion (weighted balancing among conflicting objectives such as tracking coverage, energy consumption, load balancing, and tracking stability), a combination of sparse guidance and dense shaping (routine behaviors are fed back frame by frame, while event-driven behaviors such as handover are only fed back when the event occurs), and team reward as the primary factor (all agents share the same reward value, encouraging cooperation and suppressing lazy agents). The team reward R at time t is... t The weighted sum of the following five components: R t = w1R track – w2R energy – w3R switch – w4R overload + w5R handover Among them, w1 to w5 are non-negative weight coefficients, which can be adjusted according to different scenario preferences. The definitions of each component are as follows:

[0086] 1. Tracking Coverage Bonus R track: Target tracking is considered effective only when the target is effectively locked by the zoom lens of at least one node and the target's image resolution in the zoomed view is higher than a preset threshold. Let there be M targets in the scene at time t. t There are 1 real targets, and the set of targets that are effectively tracked is M. tracked Then the tracking coverage C t = |M tracked | / M t The coverage tracking bonus is C. t Subtract the lazy penalty: An additional penalty is applied when there are untracked targets and nodes with remaining tracking capacity, preventing the agent from choosing inaction when it has the ability to track more targets. This reward is a dense reward calculated per frame.

[0087] 2. System energy consumption penalty R energy The instantaneous energy consumption rate of node i is e i = μ1· (|Δ i pan | + |Δ i tilt |)+ μ2· |Δ i zoom |, where Δ i pan Δ i tilt These represent the horizontal and pitch angle displacements of the gimbal per unit time, Δ i zoom Let μ1 and μ2 be the change in focal length per unit time, respectively. μ1 and μ2 are energy consumption coefficients calibrated by hardware, i.e., the energy consumption values ​​corresponding to unit angular displacement and unit change in focal length measured by hardware calibration experiments. The unit of μ1 is joules / degree, and the unit of μ2 is joules / focal length step. The total energy consumption is the sum of the instantaneous energy consumption rates of each node. The system energy consumption penalty is defined as the sum of the total system energy consumption and the maximum possible instantaneous energy consumption E. max The normalized energy consumption of the ratio is the dense reward calculated per frame;

[0088] 3. Tracking switch penalty R switch Assume each tracked target has a home node at time t. When a target's home node changes in two consecutive non-empty time points, it is considered a valid switch and a penalty is applied. The penalty amount is positively correlated with the value of the switched target; the switch penalty for high-value targets is higher than that for low-value targets. This penalty is an event-driven sparse penalty, calculated only when a switch occurs.

[0089] 4. Load imbalance penalty R overloadThe load factor of node i is defined as the ratio of its current number of tracked targets to its maximum tracking capacity. The variance of the load factors of all nodes is used to measure the system load imbalance, promoting load balancing among nodes and preventing some nodes from becoming overloaded and causing a decrease in tracking quality. This penalty is a density penalty calculated per frame.

[0090] 5. Successful handover reward R handover A successful relay tracking is defined as one where the target remains effectively tracked before and after the handover, and its owner node completes the transfer from the source node to the receiving node, and a reward is awarded. This reward is an event-driven sparse reward, calculated only when a successful handover occurs.

[0091] Different tracking modes can be configured by setting different weights for each component: The basic balanced mode allows setting w1=1.0, w2=0.5, w3=0.3, w4=0.2, and w5=0.5, resulting in a relatively balanced distribution of components; the energy-saving priority mode increases w2 to 1.0 and decreases w1 to 0.8 to enhance the impact of energy consumption in the reward function; the high-reliability mode increases w1 to 1.2 and decreases w2 to 0.2 to improve tracking coverage; and the smooth handover priority mode increases w3 and w5 to 0.8 and 1.0 respectively to strengthen the guidance of handover behavior. In actual training, adaptive weight adjustment can be adopted: the initial training focuses on exploring how to cover the target, the middle stage gradually introduces energy consumption and load constraints, and the later stage strengthens handover stability.

[0092] (4) MAPPO Network Structure and Training: During the training phase, the MAPPO framework of centralized training and distributed execution (CTDE) is adopted. The policy network receives local observations, the value network receives the global state, and all agents share the same set of policy network parameters. The structure of the policy network is as follows: Figure 2As shown, the architecture includes a feature encoding layer, a multi-head attention aggregation layer, a feature concatenation layer, a shared fully connected layer, and a multi-branch output layer. The feature encoding layer extracts target-level features from the feature vector of each target slot through a two-layer fully connected network. The multi-head attention aggregation layer combines effective masks to aggregate dynamically changing target features, generating a fixed-dimensional scene context vector, enabling the network to automatically focus on important targets and remain robust to changes in the number of targets. The feature concatenation layer embeds the scene context vector with the node's own state, historical association information, neighboring node state summaries, and learnable agent identifiers to form a complete observation encoding vector. The shared fully connected layer extracts high-level features through LayerNorm normalization and ReLU activation. The output layer is divided into six branches, outputting the action type probability distribution (Softmax activation), target selection probability (Sigmoid activation), target release probability (Sigmoid activation), one-hot selection of the handover target (Gumbel-Softmax activation), one-hot selection of the handover object (Gumbel-Softmax activation), and normalized priority weights (Sigmoid+Softmax activation). During actual reasoning, the output of irrelevant branches is masked according to the action type obtained from sampling: the target release vector is ignored when the action type is not the release target, the handover vector is ignored when the action type is not the handover target, and the priority weight does not participate in the actual control when the action type is not the priority adjustment.

[0093] The value network is used to estimate the state value function during the centralized training phase, providing an advantage function estimate for policy updates. Its input is the global state, which is first aggregated through an attention-based module to gather information from all targets in the scene. This information is then concatenated with the agent's state features and output as a scalar state value estimate via a three-layer fully connected network. The value network is only used during the training phase; during the distributed execution phase, nodes do not need to access the global state. To avoid convergent behavior among agents due to parameter sharing, the policy network introduces learnable agent identifier embeddings and relative position encodings of neighboring nodes. Combined with load balancing and switching penalty terms in the reward function, this guides agents to form implicit specialization. In actual operation, each edge node only runs the trained policy network and calls the multi-target adaptive vision control module by tracking the task list.

[0094] The total loss function for MAPPO training consists of policy loss, value loss, and policy entropy regularization term: L total =L policy + c1L value - c2H(π). Where, the policy loss L policy The target is optimized using a near-end strategy with importance sampling pruning, and performance collapse is avoided by limiting the policy update step size; value loss L valueThe mean squared error of value estimation and reward objectives is used; the policy entropy regularization term H(π) is used to encourage exploration and prevent the policy from converging prematurely to a suboptimal solution; c1 and c2 are the weight coefficients of value loss and entropy regularization, respectively. Training is carried out in a high-fidelity 3D simulation environment. The simulation scene includes elements that form natural field of view occlusion, such as roads, buildings, and green belts. Each dual-camera node is modeled as a combination of a fixed-direction wide-angle camera and a rotatable zoom camera, and can be configured with targets such as pedestrians and vehicles with different motion behaviors.

[0095] The training process employs a course-based learning strategy, organized into four progressively more challenging stages: Stage 1 focuses on basic single-target tracking, enabling the agent to learn basic target locking and tracking; Stage 2 focuses on independent multi-target tracking, teaching the agent to track multiple targets simultaneously while avoiding resource redundancy; Stage 3 focuses on dense scene tracking, enabling the agent to maintain tracking in dense scenes and learn load balancing; Stage 4 focuses on target handover training, teaching the agent to initiate and accept target handovers. The agent automatically progresses to the next stage once the average reward for each stage reaches a preset threshold. A domain randomization strategy is also employed, randomly sampling target motion parameters, sensor parameters, gimbal parameters, communication parameters, and scene layout from a preset distribution at the beginning of each training round. The randomization range is gradually expanded as training progresses, ensuring the strategy is robust to differences between the simulation environment and the real world.

[0096] The system adopts a layered master-slave collaborative architecture with edge-node layers, containing three logical layers: The first layer is the dual-camera node layer (slave layer), which is the perception and execution layer. Each dual-camera node is responsible for local perception and execution, maintaining communication with its corresponding edge node through the local area network, uploading local observations and receiving tracking instructions; the second layer is the edge node layer (master layer), which is the decision layer, such as... Figure 3 As shown, the architecture includes a policy network that has completed training, aggregates local observations from its subordinate nodes, and performs independent decision-making and inference for each node. Edge nodes are deployed on edge servers or smart gateways close to the cameras. The third layer is the cloud center layer, an optional component, responsible for long-term storage of the global situation, macro-level scheduling across edge nodes, and model updates and distribution, but does not participate in real-time decision-making. This layered architecture enables reasonable allocation of computing load and controllable communication overhead. Edge nodes and their subordinate nodes are located on the same local area network, and communication latency can be controlled within ten milliseconds. Furthermore, adding a new node only requires connecting to the edge node without modifying the global configuration.

[0097] Real-time decision-making follows the process of "collection-decision-issuance-execution": (1) Local observation and collection at each node: each dual-camera node collects information at a fixed frequency to construct local observations, including a list of local target observations from the wide-angle perception module, the node's own status from the zoom control module, historical association information from the local coordinator, and optional state summaries of adjacent nodes; (2) Observation upload: the node packages the local observations into a lightweight structured message and sends it to its respective edge node; (3) Observation aggregation and preprocessing: the edge node performs time alignment of the observations of each node based on the current decision time, uses the target motion model to predict the target position to the current time, and encodes the observations of each node into a batch of observation vectors with fixed dimensions; (4) Batch inference of the policy network: the edge node inputs the batch of observations into the policy network to perform forward inference and outputs the distance of each node. The probability distribution of scattered actions and continuous action parameters are sampled to obtain the specific actions of each node. Batch inference makes full use of the parallel computing capabilities of edge nodes, so that the total inference time of multiple nodes is basically equivalent to that of a single node; (5) Action parsing and instruction issuance: the edge node parses the inference results into instruction messages containing node identifier, timestamp, discrete action type, continuous action parameters and optional handover information and issues them; (6) Node execution: the local coordinator of the dual-camera node parses the instruction: if it is to continue tracking, the current tracking list is maintained; if it is to release the target, the zoom control module is notified to stop tracking the specified target; if it is to request a new target, a high-value target is selected from the candidate pool to start tracking; if it is to adjust the priority, the new priority weight is passed to the zoom control module to affect the calculation of the minimum enclosing circle; if it is to hand over the target, the cross-node target handover process is triggered.

[0098] In a preferred embodiment, the method further includes: when communication between the dual-camera gimbal and the edge node is interrupted, automatically switching to a degraded operation mode, pausing and waiting for decision instructions from the edge node, and continuing to track the current target using preset rules. To address abnormal situations in actual deployment, the system has a robust protection mechanism: when communication between the node and the edge node is temporarily interrupted, the node automatically switches to a degraded operation mode, pausing and waiting for edge instructions, and continuing to track the current target using preset rules. Simultaneously, it caches timestamped local observations to be uploaded after communication is restored. When the communication interruption exceeds a preset threshold, non-critical targets are gradually released to reduce system risk. The edge node outputs decision confidence along with the decision. When the confidence level is lower than a preset threshold, low-confidence decision cases are recorded for model improvement, and alarms are triggered in critical scenarios, allowing for remote manual intervention. To address the unavoidable delays during observation uploading and instruction issuance, the edge node records the observation time when receiving observations and performs motion prediction compensation for the target position. The issued instructions include the expected execution time, and the node executes the instructions at the expected execution time to achieve multi-node synchronization.

[0099] This invention also provides a dual-camera PTZ camera collaborative multi-target tracking device, applied to a monitoring system with at least two dual-camera PTZ cameras. Each dual-camera PTZ camera includes a fixed-direction wide-angle lens and a rotatable zoom lens, such as... Figure 4 As shown, it includes:

[0100] The moving target detection module 401 is used to obtain the difference image between the current video frame and the previous video frame captured by each wide-angle lens, calculate the motion entropy based on the difference image, divide the difference image into a grid, perform optical flow calculation on the foreground pixels within each grid to obtain the target motion vector, and dynamically adjust the grid division granularity and detection threshold to continuously identify targets and output target information; the target information includes: target position, scale, and motion state;

[0101] The control command generation module 402 is used to determine the set of targets to be tracked based on the target information and decision commands, solve the minimum enclosing circle covering all targets to be tracked, superimpose a dynamic safety margin on the radius of the minimum enclosing circle to obtain the target field of view circle, and generate control commands for each gimbal camera based on the center offset and radius of the target field of view circle; the control commands include: rotation and zoom of the gimbal camera;

[0102] The tracking module 403 is used to send a handover request to the target node after the source node triggers the target handover. The target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features. Both the source node and the target node include dual-camera gimbal cameras.

[0103] The decision instruction generation module 404 is used to model the collaborative tracking process of multiple dual-camera gimbal cameras as a distributed partially observable Markov decision process, and output the decision instructions to step S2 to execute target tracking; the decision instructions include at least one of the following: target allocation, continue tracking, release target, hand over target, and priority adjustment.

[0104] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the dual-camera gimbal-based multi-target tracking method as described in any of the above embodiments.

[0105] The method described in this invention comprehensively optimizes target tracking coverage, tracking stability, node load balancing, system energy consumption, and decision latency, enabling continuous, stable, and low-latency collaborative tracking of multiple targets in complex dynamic scenes. Through motion entropy-driven adaptive regional optical flow calculation, computational resources are concentrated in regions of high motion complexity, reducing edge perception overhead while ensuring target contour integrity. A field-of-view control method combining minimum bounding circle and dynamic safety margin balances target coverage, image compactness, zoom stability, and latency resistance. A dual association mechanism combining motion prediction filtering and appearance feature matching improves identity consistency during cross-camera handovers; and a multi-agent framework with centralized training and distributed execution allows nodes to complete collaborative scheduling without transmitting the original video stream.

[0106] Experimental verification on high-fidelity 3D simulation environment and real dataset shows that the method of the present invention achieves a tracking coverage of 84.4% in dense scenes, with an average tracking time of 23.3 seconds and good load balancing among nodes. The edge-node hierarchical architecture enables nodes to complete collaborative scheduling without transmitting raw video streams, only exchanging structured information, which significantly reduces communication overhead and end-to-end decision latency, meets the deployment requirements of real-time monitoring scenarios, and provides a complete theoretical framework and algorithmic support for the design and implementation of distributed intelligent monitoring systems.

[0107] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for collaborative multi-target tracking using dual-camera PTZ cameras, applied to a monitoring system with at least two dual-camera PTZ cameras, each dual-camera PTZ camera comprising a fixed-direction wide-angle lens and a rotatable zoom lens, characterized in that, The method includes the following steps: S1. Obtain the difference image between the current video frame and the previous video frame captured by each wide-angle lens, calculate the motion entropy based on the difference image, divide the difference image into grids, perform optical flow calculation on the foreground pixels within each grid to obtain the target motion vector, and dynamically adjust the grid division granularity and detection threshold to continuously identify the target and output target information; the target information includes: target position, scale and motion state. S2. Determine the set of targets to be tracked based on the target information and decision instructions, solve for the minimum enclosing circle covering all targets to be tracked, and superimpose a dynamic safety margin on the radius of the minimum enclosing circle to obtain the target field of view circle. Generate control instructions for each gimbal camera based on the center offset and radius of the target field of view circle; the control instructions include: rotation and zoom of the gimbal camera. S3. After triggering the target handover, the source node sends a handover request to the target node. The target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features. Both the source node and the target node include dual-camera gimbal cameras. S4. Model the collaborative tracking process of multiple dual-camera gimbal cameras as a distributed partially observable Markov decision process, and output the decision instructions to step S2 to execute target tracking; the decision instructions include at least one of the following: target allocation, continue tracking, release target, hand over target, and priority adjustment.

2. The method according to claim 1, characterized in that, Calculating motion entropy based on the difference image and dividing the difference image into a grid includes: In the initialization phase, motion statistical features of each region are collected using a uniform grid division to establish an initial estimate of the motion entropy of each region. Based on the distribution of motion pixels in the differential image, the differential image is divided into multiple initial grid regions. The motion statistical features include: target pixel density, motion intensity standard deviation, and velocity variation coefficient. For each region, the region motion entropy is calculated, which is obtained by weighting the motion intensity standard deviation, velocity variation coefficient, and target pixel density. During the adaptive operation phase, the motion vector of the foreground target pixels in each region is calculated using the pyramid Lucas-Kanade optical flow algorithm to dynamically obtain the target contour.

3. The method according to claim 2, characterized in that, The method further includes: using a first granularity grid when the motion entropy is higher than a high threshold, using a second granularity grid when it is between the high threshold and a low threshold, and using a third granularity grid when it is lower than the low threshold, wherein the first granularity grid is smaller than the second granularity grid, and the second granularity grid is smaller than the third granularity grid.

4. The method according to claim 1, characterized in that, The method further includes: dynamically adjusting the region division granularity and detection threshold through a two-layer feedback adjustment mechanism based on optical flow quality index; wherein, the two-layer feedback adjustment mechanism includes a short-term performance monitoring layer and a long-term pattern learning layer; the short-term performance monitoring layer calculates the tracking quality index of each region in real time, and when the tracking quality index of a certain region is lower than a preset quality threshold, it triggers the mesh granularity refinement operation of that region; the long-term pattern learning layer detects whether the region motion pattern is consistent with the historical record, and performs a region reorganization operation when the region motion pattern changes.

5. The method according to claim 1, characterized in that, The dynamic safety margin is obtained by weighted summation of components in four dimensions: Δr t = β1Δr dyn + β2Δr trend + β3Δr err + β4Δr scene ; Where: β1, β2, and β3 are non-negative weights and their sum is 1, Δr dyn Δr is the dynamic prediction component calculated based on the target maximum velocity and system time delay. trend Let Δr be the component representing the relative motion trend between targets. err For the error component, Δr scene For scene components.

6. The method according to claim 1, characterized in that, The target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features, including: The target node uses a target motion model to predict its expected position at the current moment, based on the target's last observation time and position contained in the handover request, and to compensate for communication delays. Traverse all target observations in the current wide-angle field of view of the target node, calculate the spatial distance between each target observation and the predicted position, and retain candidate targets whose spatial distance is not greater than the maximum distance and whose motion direction difference is not greater than a preset direction threshold. Calculate the overall matching score for each candidate target; If the highest comprehensive matching score is not lower than the high score threshold, the corresponding candidate target is determined to be the transfer target, and the handover is confirmed. If the highest overall matching score is between the high score threshold and the low score threshold, then the transient confirmation stage is entered, and the overall matching score is continuously tracked and recalculated within the subsequent preset number of frames. If the highest overall matching score is lower than the low score threshold, the handover request information will be stored in the observation queue and matching will be continuously attempted within the preset future frames. If the matching is still unsuccessful after the maximum waiting time, a handover failure notification will be sent to the source node.

7. The method according to claim 1, characterized in that, The Markov decision process includes: Each dual-camera gimbal camera uploads local observation information to its respective edge node. Each edge node runs the trained policy network to perform batch inference and outputs decision instructions. Each dual-camera gimbal camera then calls step S2 to perform specific tracking based on the decision instructions.

8. The method according to claim 7, characterized in that, The method further includes: When communication between the dual-camera gimbal and the edge node is interrupted, it automatically switches to a degraded operation mode, pauses and waits for decision instructions from the edge node, and continues to track the current target using preset rules.

9. A dual-camera PTZ camera collaborative multi-target tracking device, applied to a monitoring system with at least two dual-camera PTZ cameras, each dual-camera PTZ camera comprising a fixed-direction wide-angle lens and a rotatable zoom lens, characterized in that, include: The moving target detection module is used to obtain the difference image between the current video frame and the previous video frame captured by each wide-angle lens, calculate the motion entropy based on the difference image, divide the difference image into grids, perform optical flow calculation on the foreground pixels in each grid to obtain the target motion vector, and dynamically adjust the grid division granularity and detection threshold to continuously identify targets and output target information. The target information includes: target location, scale, and motion state; The control command generation module is used to determine the set of targets to be tracked based on the target information and decision commands, solve the minimum enclosing circle covering all targets to be tracked, superimpose a dynamic safety margin on the radius of the minimum enclosing circle to obtain the target field of view circle, and generate control commands for each gimbal camera based on the center offset and radius of the target field of view circle; the control commands include: rotation and zoom of the gimbal camera; The tracking module is used by the source node to send a handover request to the target node after triggering the target handover. The target node performs spatiotemporal filtering based on motion prediction to screen candidate targets, and then performs precise matching based on compact appearance features. Both the source node and the target node include dual-camera gimbal cameras. The decision instruction generation module is used to model the collaborative tracking process of multiple dual-camera gimbal cameras as a distributed partially observable Markov decision process, and output the decision instructions to step S2 to execute target tracking; the decision instructions include at least one of the following: target allocation, continue tracking, release target, hand over target, and priority adjustment.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the dual-camera gimbal camera collaborative multi-target tracking method as described in any one of claims 1 to 8.

Images