A ball camera cooperative target tracking method and device, electronic equipment and storage medium
By working together with panoramic cameras and local PTZ cameras, multi-dimensional information is acquired to make consistent judgments across different viewpoints and achieve stable closed-loop control. This solves the problems of misjudgment and instability in complex scenarios of traditional PTZ camera collaborative tracking solutions, and achieves safe and reliable target tracking.
Patent Information
- Application Number
- CN202611620205.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-20
- Publication Date
- 2026-11-17
AI Technical Summary
Traditional PTZ camera collaborative tracking solutions are prone to misjudgment in complex scenarios, have difficulty adapting to fixed parameters, and are unstable in direct model control, resulting in unstable target association and insufficient security.
By working in concert with panoramic cameras and local PTZ cameras, multi-dimensional information of panoramic targets and local candidate targets is acquired, cross-view consistency judgment is performed, semantic judgment results are generated, and stable steering and zoom closed-loop control is achieved through parameter boundary constraints and deterministic control.
It improves the reliability of cross-view target association and the tracking stability in complex scenarios, ensuring safe and stable completion of tracking tasks under conditions of target intersection, short-term occlusion and rapid scale changes.
Smart Images

Figure CN123123727A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video surveillance and intelligent tracking technology, and in particular to a PTZ camera-assisted target tracking method, device, electronic equipment, and storage medium. Background Technology
[0002] With the widespread application of video surveillance technology in large-scale scenarios, the collaborative tracking of panoramic cameras and local PTZ cameras has become an important means for airports, ports, industrial parks, and other locations to achieve global situational awareness and local detail observation. However, traditional collaborative tracking solutions face many challenges when dealing with complex scenarios.
[0003] Traditional solutions typically rely on target detection, trajectory tracking, geometric mapping, and fixed control parameters to control the PTZ camera's rotation or zoom by calculating the target's positional deviation within the frame. This approach works well when the number of targets is small and scene changes are minimal. However, when there are intersecting targets, short-term occlusion, rapid changes in target scale, or multiple similar targets in a localized area, relying solely on geometric position or a single identifier can easily lead to false associations and tracking loss. Furthermore, fixed control parameters are difficult to adaptively adjust based on target region, movement stage, and image quality, resulting in insufficient tracking stability and adaptability.
[0004] Furthermore, multi-camera linkage schemes lack semantic-level confirmation of whether panoramic targets and local candidate targets are the same target, and cross-view correlation is easily affected by detection errors, viewpoint changes, and occlusion; while schemes involving language models in camera control are mostly oriented towards natural language interaction, and if continuous control commands are directly output, it is difficult to guarantee real-time performance and security.
[0005] Therefore, there is an urgent need for a collaborative target tracking method that can improve the reliability of target association between panoramic and local PTZ cameras, ensure tracking stability and security in complex scenarios, and is intelligent, real-time and scalable. Summary of the Invention
[0006] The present invention provides a PTZ camera-assisted target tracking method to address the problems of easy misjudgment in cross-view correlation, difficulty in adaptive fixed parameters, and instability in direct model control in existing technologies. The technical solution is as follows: According to one aspect of the present invention, a PTZ camera-coordinated target tracking method is provided. The method includes: performing target detection on a panoramic video using a panoramic camera to obtain panoramic information of the panoramic target; the panoramic information includes a panoramic target identifier, a panoramic target position, a panoramic target region, a panoramic target scale, a panoramic target motion trend, and historical trajectory information; performing candidate target detection on a local frame using a local PTZ camera to obtain a local candidate target identifier, a local candidate target position, and a local candidate target scale, and reading the current turning angle and zoom state of the PTZ camera to construct a unified target state with the panoramic information; performing cross-view consistency judgment on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary in the unified target state, and outputting a semantic judgment result; the semantic judgment result includes the same target confidence level, target state category, occlusion risk level, image quality level, and candidate target ranking; generating a tracking strategy and control parameter suggestions based on the semantic judgment result, and performing closed-loop control of turning and zoom on the local PTZ camera after parameter boundary constraints; when tracking is unstable, performing search control on the local PTZ camera in conjunction with the panoramic target position and historical motion trend, and resuming closed-loop tracking after confirming the target.
[0007] In one embodiment, target detection is performed on panoramic video using a panoramic camera to obtain panoramic information of the panoramic target. This is achieved through the following steps: the panoramic camera reads the target perception result of the current frame in the panoramic video, and parses it to obtain the panoramic information of the panoramic target. If the target perception result is abnormal or there is no target to be tracked, the status is updated and the system waits for the next cycle. The identifier is used to refer to the target; the position is used to determine the coordinates of the target in the panoramic image; the region is used to determine the scene area where the target is located; the scale is used to describe the size of the target in the panoramic image; the motion trend is used to describe the direction and state of movement of the target; and the historical trajectory is used to record the position changes of the target in historical cycles. In one embodiment, candidate target detection is performed on a local PTZ camera to obtain local candidate target identifiers, local candidate target positions, and local candidate target scales. The current turning angle and zoom state of the PTZ camera are then read to construct a unified target state with the panoramic information. This is achieved through the following steps: Local candidate targets in the current frame of the local image are read, and their identifiers, positions, and scales are obtained. The current turning angle and zoom state of the PTZ camera are also read. The local candidate target identifier is used to distinguish different candidate targets. The local candidate target position is used to determine the coordinates of the local candidate target in the local image. The local candidate target scale is used to describe the size of the local candidate target in the local image. The current turning angle of the PTZ camera is used to determine the current pointing direction of the local PTZ camera. The current zoom state of the PTZ camera is used to determine the current magnification of the local PTZ camera. The local candidate target identifier, position, scale, current turning angle, and current zoom state of the PTZ camera are integrated with the panoramic information, scene area information, and service identification information to form a unified target state. If there is no target to be tracked, the state is updated only, and the system waits for the next cycle.
[0008] In one embodiment, cross-view consistency judgment is performed on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary in the unified target state, and the semantic judgment result is output through the following steps: A panoramic target screenshot is obtained based on the panoramic target location; a local candidate screenshot is obtained based on the local candidate target location; a target motion summary is generated based on the panoramic target motion trend and the panoramic target historical trajectory; a historical tracking summary is generated based on the panoramic target historical trajectory and the local candidate target identifier; a cross-view consistency judgment is performed on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary to determine whether the panoramic target in the panoramic video and each local candidate target in the local PTZ camera image are the same target, and the current state of the target, occlusion risk, and image quality are evaluated to output the semantic judgment result; the same target confidence level is used to indicate the probability that the panoramic target and the local candidate targets belong to the same target; the target state category is used to indicate the target state; the occlusion risk level is used to indicate the degree of risk of the target being occluded; the image quality level is used to indicate the clarity of the local image; and the candidate target ranking is used to indicate the ranking of the consistency between each local candidate target and the panoramic target.
[0009] In one embodiment, generating tracking strategies and control parameter suggestions based on the semantic determination results is achieved through the following steps: generating tracking strategies and control parameter suggestions for the current period based on the semantic determination results; the tracking strategies include continuing tracking, holding and waiting, reducing zoom, enhancing search, switching candidate targets, entering the recapture process, or returning to the default observation state; the control parameter suggestions include response intensity levels, target retention thresholds, target loss tolerance time, and zoom compensation intensity; the response intensity levels are used to indicate the speed of response of local PTZ camera turning and zoom control; the target retention thresholds are used to indicate the determination threshold required for the target to remain within a preset area of the local image; the target loss tolerance time is used to indicate the length of time allowed to wait and continue searching after the target disappears briefly; the zoom compensation intensity is used to indicate the magnitude of compensation for zoom control when the target scale changes.
[0010] In one embodiment, the closed-loop control of steering and zoom of the local PTZ camera after parameter boundary constraints is achieved through the following steps: boundary clipping and legality verification are performed on the response intensity level, target retention threshold, target loss tolerance time, and zoom compensation intensity; the value range is limited; abnormal outputs are filtered; and when the results are unreliable, the system reverts to the preset default strategy and parameters; proportional-integral-derivative control, fuzzy control, or model predictive control are used to calculate the steering control quantity based on the positional deviation between the local candidate target position and the preset area; the zoom control quantity is calculated based on the scale deviation between the local candidate target scale and the desired scale; and when the local candidate target deviates from the preset area when the local image is magnified, the steering control and zoom control are coupled and compensated.
[0011] In one embodiment, when tracking is unstable, the local PTZ camera is searched and controlled by combining the panoramic target position and historical motion trend. After the target is confirmed, closed-loop tracking is restored through the following steps: If the confidence level of the same target meets the threshold for continuing tracking and the occlusion risk level is within an acceptable range, the current tracking is determined to be stable, the state is updated, and the next cycle is entered; otherwise, the current tracking is determined to be unstable, and the search is continued or the default observation state is returned based on the panoramic target position, panoramic target motion trend, panoramic target historical trajectory, PTZ camera current turning angle, PTZ camera current zoom state, and historical observation records; the continued search includes waiting for the target to reappear, predicting the direction according to the historical trajectory, searching a small area, and expanding the field of view by reducing zoom; cross-view consistency judgment is performed on newly discovered local candidate targets. If it is confirmed as the original target, local closed-loop control is restored. If the target loss tolerance time is exceeded and it is still not confirmed, the current tracking is stopped or a new candidate target is switched.
[0012] According to one aspect of the present invention, a PTZ camera-coordinated target tracking device includes: a panoramic target detection module, configured to perform target detection on panoramic video using a panoramic camera to acquire panoramic information of the panoramic target; the panoramic information includes panoramic target identifier, panoramic target position, panoramic target region, panoramic target scale, panoramic target motion trend, and historical trajectory information; a target state construction module, configured to perform candidate target detection on a local scene using a local PTZ camera to acquire local candidate target identifier, local candidate target position, and local candidate target scale, and to read the current turning angle and zoom state of the PTZ camera to construct a unified target state with the panoramic information; and a cross-viewpoint language... The semantic determination module is used to perform cross-view consistency judgment on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary in the unified target state, and output the semantic determination result. The semantic determination result includes the confidence of the same target, the target state category, the occlusion risk level, the image quality level, and the candidate target ranking. The control and re-acquisition module is used to generate tracking strategies and control parameter suggestions based on the semantic determination result, and perform steering and zoom closed-loop control on the local PTZ camera after parameter boundary constraints. When the tracking is unstable, the local PTZ camera is searched and controlled in combination with the panoramic target position and historical motion trend. After the target is confirmed, the closed-loop tracking is resumed.
[0013] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored in the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the PTZ-camera cooperative target tracking method as described above.
[0014] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, which are executed by one or more processors to implement the PTZ camera cooperative target tracking method as described above.
[0015] The beneficial effects of the technical solution provided by this invention are: In the above technical solution, this invention first acquires panoramic information of the panoramic target in the panoramic video and local information of local candidate targets in the local PTZ camera footage. It then constructs a unified target state including panoramic target identifier, panoramic target location, panoramic target region, panoramic target scale, panoramic target motion trend, panoramic target historical trajectory, local candidate target identifier, local candidate target location, local candidate target scale, current PTZ camera turning angle, current PTZ camera zoom state, scene area information, and service identifier information. This accurately describes the target association basis between the panoramic and local views. Based on this, a multimodal semantic judgment module performs cross-view consistency judgment on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary in the unified target state. It outputs the same target confidence, target state category, occlusion risk level, image quality level, and candidate target ranking, obtaining the semantic judgment result. Simultaneously, based on the semantic judgment result, it generates tracking strategies and control parameter suggestions, and adjusts the response strength... The system employs parameter boundary constraints, including zoom level, target retention threshold, target loss tolerance time, and zoom compensation intensity. Then, after these constraints, a deterministic controller performs closed-loop steering and zoom control on the local PTZ camera based on the positional deviation between the local candidate target's location and the preset area, as well as the scale deviation between the local candidate target's scale and the desired scale. Finally, when tracking becomes unstable, the local PTZ camera is searched and controlled by combining the panoramic target's position, motion trend, and historical trajectory. Once the target is confirmed, closed-loop tracking resumes. This achieves semantic confirmation of the target association between the panoramic and local PTZ cameras and stable control of the local PTZ camera. This not only improves the reliability of cross-view target association but also significantly enhances tracking stability in complex scenes. The system can more safely and stably complete tracking tasks under conditions such as target intersection, short-term occlusion, and rapid scale changes. This effectively solves the problems of easy misjudgment in cross-view association, difficulty in adapting fixed parameters, and instability in direct model control when existing technologies face complex scenes. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0017] Figure 1 This is a flowchart illustrating a PTZ camera-coordinated target tracking method according to an exemplary embodiment; Figure 2 This is a schematic flowchart illustrating a PTZ camera-coordinated target tracking method in an exemplary embodiment; Figure 3 yes Figure 2The software implementation flowchart of the PTZ camera-coordinated target tracking method in the corresponding embodiment; Figure 4 This is a block diagram illustrating a PTZ camera-assisted target tracking device according to an exemplary embodiment; Figure 5 This is a hardware structure diagram of an electronic device according to an exemplary embodiment; Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0019] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.
[0020] This invention provides a PTZ camera cooperative target tracking method. By constructing a unified state of panoramic and local candidate targets and judging multimodal semantic consistency, it achieves reliable association of cross-view targets and stable closed-loop control of local PTZ cameras. This solves the problems of easy misjudgment in cross-view association, difficulty in adapting fixed parameters, and instability in direct model control in existing technologies. This PTZ camera cooperative target tracking method is applicable to PTZ camera cooperative target tracking devices, which can be electronic devices. The PTZ camera cooperative target tracking method in this invention can be applied to various scenarios, such as panoramic PTZ camera cooperative target tracking, etc.
[0021] Please see Figure 1 This invention provides a PTZ camera-assisted target tracking method, which is applicable to electronic devices.
[0022] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.
[0023] like Figure 1 As shown, the method may include the following steps: Step 110: Target detection is performed on the panoramic video using a panoramic camera to obtain panoramic information of the panoramic target.
[0024] In one possible implementation, the target perception result of the current frame in the panoramic video is read by a panoramic camera, and the panoramic information of the panoramic target is obtained by parsing. If the target perception result is abnormal or there is no target to be tracked at present, the status is updated and the system waits for the next cycle.
[0025] The panoramic information includes panoramic target identification, panoramic target location, panoramic target area, panoramic target scale, panoramic target movement trend and historical trajectory information, etc., which are not limited here.
[0026] Among them, the identifier is used to refer to the target; the location is used to determine the target's coordinates in the panoramic view; the region is used to determine the scene area where the target is located; the scale is used to describe the size of the target in the panoramic view; the motion trend is used to describe the target's direction of movement and motion state; and the historical trajectory is used to record the target's position changes in a historical period.
[0027] Specifically, the target perception results of the current frame are read from the panoramic camera or panoramic video processing module, and the panoramic target identifier, panoramic target position, panoramic target region, panoramic target scale, panoramic target motion trend, and panoramic target historical trajectory are obtained. If the target perception results are abnormal or there is no target to be tracked, the status is only updated and the system waits for the next cycle to avoid unnecessary calculations that consume system resources.
[0028] Furthermore, the panoramic target identifier is used to uniquely identify the panoramic target in the subsequent unified target status, avoiding identity confusion in multi-target scenarios; the panoramic target location is used to determine the coordinates of the panoramic target in the panoramic image, providing a basis for subsequent screenshot acquisition and region judgment; the panoramic target region is used to determine the scene area where the panoramic target is located, providing context for scene risk identification; and the panoramic target scale is used to describe the size of the panoramic target in the panoramic image, providing a basis for subsequent scale change judgment.
[0029] Furthermore, the panoramic target motion trend describes the target's direction of movement and motion state, providing a basis for subsequent target motion summary generation and recapture prediction; the panoramic target historical trajectory records the target's positional changes over historical periods, providing data support for historical tracking summaries in cross-view consistency judgment and trajectory prediction in abnormal recapture. All of the above panoramic information serves as panoramic-side constraints, inputting into the subsequent unified target state construction.
[0030] In the above process, the embodiments of the present invention use a panoramic camera to perform target detection on panoramic video and analyze multi-dimensional panoramic information, enabling the system to accurately discover targets from a global perspective and grasp their large-scale position and movement trend, providing a target constraint basis on the panoramic side, and realizing data support for subsequent cross-view association and recapture prediction.
[0031] Step 120: Detect candidate targets in a local area using a local PTZ camera, obtain local candidate target identifiers, local candidate target positions, and local candidate target scales, and read the current turning angle and zoom status of the PTZ camera to construct a unified target state with the panoramic information.
[0032] In one possible implementation, local candidate targets in the current frame of a local scene are read to obtain their identifiers, locations, and scales. The current turning angle and zoom state of the PTZ camera are also read. These local candidate target identifiers, locations, scales, current turning angle, and zoom state are then integrated with panoramic information, scene area information, and service identifiers to form a unified target state. If there is no target to track at present, the state is updated only, and the system waits for the next cycle.
[0033] Among them, the local candidate target identifier is used to distinguish different candidate targets; the local candidate target position is used to determine the coordinates of the local candidate target in the local image; the local candidate target scale is used to describe the size of the local candidate target in the local image; the PTZ camera's current turning angle is used to determine the current pointing direction of the local PTZ camera; and the PTZ camera's current zoom state is used to determine the current magnification of the local PTZ camera.
[0034] Specifically, local candidate targets are read from the local PTZ camera view of the current frame, and the local candidate target identifier, local candidate target position, and local candidate target scale are obtained. The local candidate target identifier is used to distinguish different candidate targets in the subsequent unified target state; the local candidate target position is used to determine the coordinates of the local candidate target in the local view; and the local candidate target scale is used to describe the size of the local candidate target in the local view.
[0035] Furthermore, the current turning angle and zoom status of the PTZ camera are read. The current turning angle is used to determine the current pointing direction of the local PTZ camera, providing a reference for subsequent turning control and re-acquisition search; the current zoom status is used to determine the current magnification of the local PTZ camera, providing a basis for zoom control and scale compensation.
[0036] Furthermore, the local candidate target identifier, local candidate target location, local candidate target scale, current PTZ camera turning angle, and current zoom state are integrated with the panoramic target identifier, panoramic target location, panoramic target area, panoramic target scale, panoramic target motion trend, and panoramic target historical trajectory, as well as scene area information and business identification information, to form a unified target status. If there is no target to be tracked at present, only the status is updated and the process waits for the next cycle. This step can be executed in parallel with step one.
[0037] In the above process, the embodiments of the present invention use a local PTZ camera to detect candidate targets in a local scene and read the device status, so that the system can unify the panoramic constraints and local observations into a structured target state, providing the complete input required for cross-view semantic judgment and strategy generation, and realizing the spatiotemporal alignment and unified expression of panoramic and local information.
[0038] Step 130: Perform cross-view consistency judgment on the panoramic target screenshot, local candidate screenshot, target motion summary and historical tracking summary in the unified target state, and output the semantic judgment result.
[0039] In one possible implementation, a panoramic target screenshot is obtained based on the panoramic target location, a local candidate target screenshot is obtained based on the local candidate target location, a target motion summary is generated based on the panoramic target motion trend and the panoramic target historical trajectory, and a historical tracking summary is generated based on the panoramic target historical trajectory and the local candidate target identifiers.
[0040] In one possible implementation, cross-view consistency judgment is performed on panoramic target screenshots, local candidate screenshots, target motion summaries, and historical tracking summaries to determine whether the panoramic target in the panoramic video and each local candidate target in the local PTZ camera image are the same target, and the target's current state, occlusion risk, and image quality are evaluated to output semantic judgment results.
[0041] The semantic determination results include the confidence level of the same target, the target state category, the occlusion risk level, the image quality level, and the ranking of candidate targets, etc., which are not specified here.
[0042] Among them, the same target confidence score is used to indicate the probability that the panoramic target and the local candidate target belong to the same target; the target state category is used to indicate the target state; the occlusion risk level is used to indicate the degree of risk of the target being occluded; the image quality level is used to indicate the clarity of the local image; and the candidate target ranking is used to indicate the ranking of the consistency between each local candidate target and the panoramic target.
[0043] Specifically, a panoramic target screenshot is obtained based on the panoramic target's location, and a local candidate target screenshot is obtained based on the local candidate target's location. A target motion summary is generated based on the panoramic target's motion trend and historical trajectory, and a historical tracking summary is generated based on the panoramic target's historical trajectory and local candidate target identifiers. These materials describe the target's appearance, motion, and tracking history from different dimensions, providing multi-source evidence for cross-view consistency judgment.
[0044] Furthermore, cross-view consistency judgment is performed on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary to determine whether the panoramic target in the panoramic video and each local candidate target in the local PTZ camera image are the same target, and to evaluate the target's current state, occlusion risk, and image quality. This module can be implemented by a visual language model, a target re-identification model, a trajectory prediction model, a target classification model, or a combination thereof. Further, the semantic judgment results include the same target confidence score, target state category, occlusion risk level, image quality level, and candidate target ranking. The same target confidence score indicates the probability that the panoramic target and local candidate targets belong to the same target; the target state category indicates whether the target is stationary, moving, occluded, or invisible; the occlusion risk level indicates the degree of risk of the target being occluded; the image quality level indicates the clarity of the local image; and the candidate target ranking indicates the order of consistency between each local candidate target and the panoramic target. This semantic judgment result is not used as a continuous control command but is used for subsequent tracking strategy generation.
[0045] In the above process, the embodiments of the present invention use a multimodal semantic determination module to make cross-view consistency judgments on panoramic targets and local candidates, enabling the system to confirm the identity of the target by comprehensively considering multi-dimensional information such as appearance, motion, region and image quality. This provides a semantic-level basis for cross-view association and reduces the probability of false association under target intersection, occlusion and interference from similar targets.
[0046] Step 140: Generate tracking strategy and control parameter suggestions based on semantic judgment results. After parameter boundary constraints, perform steering and zoom closed-loop control on the local PTZ camera. When tracking is unstable, combine the panoramic target position and historical motion trend to perform search control on the local PTZ camera. After confirming the target, resume closed-loop tracking.
[0047] In one possible implementation, a tracking strategy and control parameter suggestion for the current cycle are generated based on the semantic judgment result; boundary clipping and legality verification are performed on the response intensity level, target retention threshold, target loss tolerance time and zoom compensation intensity, the value range is limited, abnormal outputs are filtered, and the system reverts to the preset default strategy and parameters when the result is unreliable.
[0048] In one possible implementation, proportional-integral-derivative control, fuzzy control, or model predictive control are used to calculate the steering control quantity based on the positional deviation between the local candidate target position and the preset area, and the zoom control quantity is calculated based on the scale deviation between the local candidate target scale and the desired scale. When the local candidate target deviates from the preset area when the local image is magnified, the steering control and zoom control are coupled and compensated.
[0049] In one possible implementation, if the confidence level of the same target meets the threshold for continuing tracking and the occlusion risk level is within an acceptable range, then the current tracking is determined to be stable, the status is updated, and the next cycle begins; otherwise, the current tracking is determined to be unstable, and the search is continued or the default observation state is returned based on the panoramic target position, panoramic target movement trend, panoramic target historical trajectory, PTZ camera current turning angle, PTZ camera current zoom state, and historical observation records; continuing the search includes waiting for the target to reappear, predicting the direction based on the historical trajectory, searching within a small area, and expanding the field of view by reducing zoom.
[0050] In one possible implementation, a cross-view consistency judgment is made on newly discovered local candidate targets. If the target is confirmed as the original target, local closed-loop control is restored. If the target loss tolerance time is exceeded and the target is still not confirmed, the current tracking is stopped or the target is switched to a new candidate target.
[0051] The tracking strategies include continuing to track, waiting, reducing zoom, enhancing search, switching candidate targets, entering the recapture process, or returning to the default observation state, etc. The control parameters suggested include response intensity level, target retention threshold, target loss tolerance time, zoom compensation intensity, etc., none of which are specified here.
[0052] Among them, the response intensity level is used to indicate the speed of response of local PTZ camera turning and zoom control; the target retention threshold is used to indicate the judgment threshold required for the target to remain within the preset area of the local image; the target loss tolerance time is used to indicate the length of time allowed to wait and continue searching after the target disappears for a short time; and the zoom compensation intensity is used to indicate the extent of compensation for zoom control when the target scale changes.
[0053] Specifically, based on the same target confidence level, target state category, occlusion risk level, image quality level, and candidate target ranking, the system generates tracking strategies and control parameter suggestions for the current period. Tracking strategies include continuing tracking, holding, reducing zoom, enhancing search, switching candidate targets, entering the recapture process, or returning to the default observation state. Control parameter suggestions include response intensity levels, target retention thresholds, target loss tolerance time, and zoom compensation intensity. Response intensity levels indicate the speed of response to local PTZ camera turning and zoom control; the target retention threshold indicates the threshold required to keep the target within a preset area of the local image; the target loss tolerance time indicates the length of time allowed to wait and continue searching after a short-term disappearance of the target; and the zoom compensation intensity indicates the magnitude of compensation for zoom control when the target scale changes.
[0054] Furthermore, the tracking strategy and control parameter recommendations must be processed by the parameter boundary constraint module. This module performs boundary clipping and validity checks on the response intensity level, target retention threshold, target loss tolerance time, and zoom compensation intensity, limiting the value range, filtering abnormal outputs, and reverting to preset default strategies and parameters when model results are unreliable, ensuring the system's safe and stable operation.
[0055] Furthermore, after parameter boundary constraints, a deterministic controller performs local closed-loop control. The deterministic controller employs proportional-integral-derivative (PID), fuzzy control, or model predictive control. It calculates the steering control quantity based on the positional deviation between the local candidate target's position and the preset area, and the zoom control quantity based on the scale deviation between the local candidate target's scale and the desired scale. When the local candidate target deviates from the preset area during local image magnification, the steering control and zoom control are coupled and compensated to reduce the probability of the target falling out of the frame. Control commands are sent to the local PTZ camera for execution in a specified format.
[0056] Furthermore, by comprehensively considering the confidence level of the same target, the occlusion risk level, the target state category, and the closed-loop control execution status, it is determined whether the current tracking is stable. If the confidence level of the same target meets the threshold for continuing tracking and the occlusion risk level is within an acceptable range, the current tracking is determined to be stable, the status is updated, and the next cycle begins; otherwise, the current tracking is determined to be unstable, and abnormal recapture processing begins. Abnormal recapture processing selects from waiting for the target to reappear, predicting the direction based on the panoramic target position, panoramic target movement trend, panoramic target historical trajectory, PTZ camera current turning angle, PTZ camera current zoom status, and historical observation records, choosing between waiting for the target to reappear, predicting the direction based on the historical trajectory, conducting a small-range search, expanding the field of view search after reducing zoom, or returning to the default observation state. During the recapture process, the multimodal semantic judgment module is called again to perform cross-view consistency judgment on newly discovered local candidate targets. If it is confirmed to be the original target, local closed-loop control is restored; if it is still not confirmed after the target loss tolerance time has expired, the current tracking is stopped or a new candidate target is switched to.
[0057] In the above process, the embodiments of the present invention enable the system to form a hierarchical structure between semantic decision-making and real-time control through the coordinated cooperation of strategy generation, parameter constraints, deterministic closed-loop control and anomaly recapture, providing safe, controllable and adaptive tracking control capabilities, and realizing continuous and stable target tracking and reliable recovery after short-term loss in complex scenarios.
[0058] Through the above process, this embodiment of the invention first acquires panoramic information of the panoramic target and local information of local candidate targets, constructing a unified target state that includes location, scale, motion trend, historical trajectory, device status, and scene area information, thereby establishing a data foundation for cross-view association. Based on this, a multimodal semantic judgment module performs cross-view consistency judgment on panoramic target screenshots, local candidate screenshots, target motion summaries, and historical tracking summaries, outputting the same target confidence level, target state category, occlusion risk level, image quality level, and candidate target ranking. Simultaneously, based on the semantic judgment results, tracking strategies and control parameter suggestions are generated, and parameter boundary constraints are applied to response intensity levels, target retention thresholds, target loss tolerance time, and zoom compensation intensity. Then, a deterministic controller, based on the local... The system performs steering and zoom closed-loop control on the local PTZ camera by considering the positional deviation between the candidate target location and the preset area, as well as the scale deviation between the local candidate target scale and the desired scale. Finally, when tracking is unstable, the local PTZ camera is searched and controlled by combining the panoramic target position, panoramic target motion trend, and panoramic target historical trajectory. After the target is confirmed, closed-loop tracking is resumed. This achieves hierarchical collaborative tracking with semantic decision-making at the top, constraint control at the bottom, and anomaly recapture as a fallback. This not only improves the reliability of cross-view target association but also significantly enhances the tracking stability in complex scenes. It enables the system to complete the tracking task more safely and stably in situations such as target intersection, short-term occlusion, and rapid scale changes. This effectively solves the problems of easy misjudgment in cross-view association, difficulty in adapting fixed parameters, and instability of direct model control in existing technologies.
[0059] In one exemplary embodiment, a flowchart illustrating the construction of a corresponding airport panoramic PTZ camera cooperative target tracking system using the PTZ camera cooperative target tracking method of the present invention is provided. Taking an airport surface surveillance scenario as an example, the panoramic PTZ camera cooperative target tracking system and method based on multimodal semantic constraints of the present invention are described in detail.
[0060] The panoramic PTZ camera collaborative target tracking system provided in this embodiment includes a multi-source perception module, a target state construction module, a multimodal semantic determination module, a strategy generation and parameter constraint module, a local closed-loop control module, and an anomaly recapture module.
[0061] The system comprises several modules: a multi-source perception module for acquiring panoramic video and local PTZ camera information, and a target state construction module for integrating scene and historical state information to provide perceptual input for subsequent processing; a multimodal semantic determination module for cross-view consistency assessment of panoramic and local candidate targets, and evaluation of target state, occlusion risk, and image quality; a strategy generation and parameter constraint module for generating tracking strategies and control parameter suggestions based on semantic determination results, and for imposing boundary constraints on the parameters; a local closed-loop control module for executing local PTZ camera steering and zoom closed-loop control based on the constrained strategy and parameters; and an anomaly re-acquisition module for searching and re-acquiring targets when tracking is unstable or targets are temporarily lost, by combining panoramic video and historical state information.
[0062] From an overall architecture perspective, the system forms a collaborative structure of "semantic decision-making at the top and constraint control at the bottom": the multimodal semantic judgment module and the policy generation and parameter constraint module are responsible for high-level semantic judgment and policy suggestions, the local closed-loop control module is responsible for real-time execution, and the anomaly recapture module is responsible for fallback handling in abnormal situations. Each module works collaboratively according to data flow and control flow to jointly complete the entire tracking chain of panoramic discovery, semantic verification, policy constraint, local closed-loop control, and anomaly recapture.
[0063] like Figure 2 As shown in the diagram, the collaborative tracking system block diagram of this embodiment is executed sequentially from left to right, including six stages: multi-source perception, target state construction, semantic understanding, policy constraint, closed-loop execution, and execution feedback.
[0064] The first step is multi-source sensing.
[0065] The system acquires panoramic video information, local PTZ camera information, and scene and historical state information through a multi-source sensing module. Panoramic video information is used to detect targets, determine the target's location, describe the target's movement trend, and provide a panoramic view of the target. Local PTZ camera information is used to obtain local images of the target, local candidate targets, the target's position within the local image, and its local tracking status. Scene and historical state information is used to provide the scene area where the target is located, its historical trajectory, and service identifier. All of the above information serves as input for subsequent target state construction.
[0066] The second step is to construct the target state.
[0067] The system integrates panoramic target identifiers, locations, regions, scales, motion trends, and historical trajectories from panoramic video information with local candidate target identifiers, locations, scales, current camera angles, and zoom states from local PTZ camera information, along with scene and historical state information, through a target state construction module. This unified target state is then used for rule-based logic processing and can also serve as input for a multimodal semantic determination module.
[0068] The third step is semantic understanding.
[0069] The system utilizes a multimodal semantic determination module to perform cross-view consistency assessments on panoramic target screenshots, local candidate screenshots, target motion summaries, and historical tracking summaries within a unified target state, thus completing the identification of the same target and scene risk assessment. The multimodal semantic determination module determines whether the panoramic target in the panoramic video and each local candidate target in the local PTZ camera footage are the same target, and evaluates the target's current state, occlusion risk, image quality, and risk of continued tracking. The multimodal semantic determination module can be implemented using a visual language model, a target re-identification model, a trajectory prediction model, a target classification model, or a combination thereof. Its output includes the confidence level of the same target, the target state category, the occlusion risk level, the image quality level, and the candidate target ranking. This output is not used as continuous control instructions but rather as a high-level semantic result for subsequent strategy generation.
[0070] The fourth step is policy constraints.
[0071] The system generates tracking strategies and sets parameter boundaries based on semantic judgment results through a strategy generation and parameter constraint module. Tracking strategies include continuing tracking, holding and waiting, reducing zoom, enhancing search, switching candidate targets, entering the recapture process, or returning to the default observation state. Suggested control parameters include response intensity levels, target hold thresholds, target loss tolerance time, and zoom compensation intensity. Strategy generation can be implemented using a large language model, rule engine, state machine, or lightweight strategy model. Its role is to convert semantic judgments into executable tracking modes, rather than directly outputting continuous gimbal control quantities. To ensure safety and stability, the strategy generation results must be processed by the parameter constraint module. This module performs boundary clipping and validity checks on the response intensity levels, target hold thresholds, target loss tolerance time, and zoom compensation intensity, filters out abnormal outputs, and reverts to a preset strategy when the model results are unreliable.
[0072] Fifth step: Closed-loop execution.
[0073] The system employs a local closed-loop control module to execute local tracking control, steering, and zoom based on the target and control parameters constrained by the strategy. This module utilizes proportional-integral-derivative (PID), fuzzy control, or model predictive control. It calculates the steering control quantity based on the positional deviation between the local candidate target's location and the preset area, and the zoom control quantity based on the scale deviation between the local candidate target's scale and the desired scale. When a local candidate target deviates from the preset area during local image magnification, the system couples and compensates the steering and zoom controls to reduce the risk of the target going out of frame. Control commands are sent to the local PTZ camera for execution in a specified format.
[0074] Step 6: Execution feedback.
[0075] The system completes image feedback and status updates through the execution feedback loop, writing the target status, tracking status, control output, and semantic judgment results of the current cycle into the status cache and historical trajectory for use by each stage in the next cycle. The system then waits for the next frame or the next control cycle to trigger before restarting execution from the multi-source perception stage.
[0076] like Figure 3 As shown, the collaborative tracking software process of this embodiment includes steps S1 to S9: a stable path enters the next cycle, and an abnormal path returns to S4 for multimodal semantic determination after recapture.
[0077] S1, acquire panoramic video target information.
[0078] The system reads the target perception results of the current frame from the panoramic camera or panoramic video processing module, and analyzes the target position, target identifier, target category, target scale, motion state annotation, and scene area information to obtain panoramic target identifier, panoramic target position, panoramic target area, panoramic target scale, panoramic target motion trend, and panoramic target historical trajectory. This step provides a global perspective input for subsequent target state construction. If the panoramic perception results are abnormal or there is currently no target to be tracked, the system jumps to S9 to update the state and waits for the next cycle.
[0079] S2, obtain local candidate information.
[0080] The system reads local candidate targets from the target detection module of the local PTZ camera image, obtains the identifier, position, and scale of each local candidate target, and reads the current turning angle and zoom state of the PTZ camera. This step can be executed in parallel with S1; if there is no target to be tracked, it jumps to S9.
[0081] S3, Construct a unified target state.
[0082] The system integrates the outputs of S1 and S2 with historical trajectory information, scene area information, and business identification information to form a unified target state, which serves as the input for semantic judgment and strategy generation. The target state description includes, but is not limited to: panoramic target identification and location, local candidate target identification and location, historical target movement trend, target scale changes, local image observation quality, and current device status.
[0083] S4, Multimodal semantic determination.
[0084] The system inputs a unified target state into the multimodal semantic determination module to determine whether the panoramic target in the panoramic video and the various local candidate targets in the local PTZ camera footage are the same target, and evaluates the target's current state, occlusion risk, and image quality. The multimodal semantic determination module can be implemented using a visual language model, a target re-identification model, or a combination of both. Its inputs are panoramic target screenshots, local candidate screenshots, target motion summaries, and historical tracking summaries; the outputs are the same target confidence score, target state category (e.g., stationary, moving, occluded, invisible), occlusion risk level, image quality level, and candidate target ranking. This output is not used as continuous control commands but is used for S5 strategy generation.
[0085] S5 generates and constrains the tracking strategy.
[0086] Based on the semantic judgment results of S4, the system generates tracking strategies and control parameter suggestions for the current period using a strategy generation module (which can be a large language model, a rule engine, or a combination of both). Typical strategies include: continue tracking, hold and wait, reduce zoom to expand the field of view, switch candidate targets, enter the recapture process, or return to the default observation state. Control parameter suggestions include: response intensity level, target retention threshold, target loss tolerance time, and zoom compensation intensity. The above suggestions must be subject to boundary clipping and legality verification by the parameter constraint module; if the model output is abnormal or the confidence level is insufficient, it will automatically fall back to the preset default strategy and parameters to ensure stable system operation.
[0087] S6 executes local closed-loop control.
[0088] Based on the target and control parameters determined in S5, the system executes local closed-loop control via a deterministic control module. The control module calculates steering control based on the positional deviation between the local candidate target's position and the preset area, and calculates zoom control based on the scale deviation between the local candidate target's scale and the desired scale. When a local candidate target deviates from the preset area and needs to be magnified, coupling compensation is applied to steering and zoom to reduce the probability of the target falling out of frame. Control commands are sent to the local PTZ camera in a specified format for execution; this step continues until the next frame arrives or the control cycle ends.
[0089] S7 determines whether the tracking is stable.
[0090] The system combines the target confidence level, occlusion risk level, target state category output from S4 with the control execution status from S6 to determine whether the current tracking is in a stable state. If the target confidence level meets the threshold for continuing tracking and the occlusion risk level is within an acceptable range, it is determined to be stable, and the system enters the S8 update state to continue to the next cycle; otherwise, it is determined to be unstable, and the system enters the low confidence or abnormal branch, executing the S9 recapture processing.
[0091] S8, update the status and enter the next cycle.
[0092] The system writes the target state, tracking state, control output, and semantic determination results of the current cycle into the state cache and historical trajectory for use in the steps of the next cycle. The system then waits for the next frame or the next control cycle to trigger before restarting execution from S2.
[0093] S9, recapture processing.
[0094] The system integrates the panoramic target's position, movement trend, historical trajectory, current camera angle, zoom level, historical observation records, and semantic judgment results from S4 in the panoramic video to select an appropriate re-acquisition strategy: waiting for the target to reappear, predicting the direction based on the historical trajectory, conducting a small-scale search, expanding the field of view by reducing zoom, or returning to the default observation state. During re-acquisition, S4 is invoked again to perform multimodal semantic judgment on newly discovered local candidate targets; if confirmed as the original target, local closed-loop control is restored via S5, and candidate target tracking resumes; if confirmation is not achieved even after exceeding preset tolerance conditions, current tracking is stopped or target switching is triggered. After re-acquisition of abnormal paths, the system returns to S4 for multimodal semantic judgment.
[0095] Through the above process, this embodiment of the invention acquires panoramic information of the panoramic target and local candidate target information through collaborative perception of the panoramic camera and local PTZ camera, and constructs a unified target state; it uses a multimodal semantic judgment module to perform cross-view consistency judgment on the panoramic target and local candidate targets, and outputs the same target confidence, target state category, occlusion risk level, image quality level and candidate target ranking; it generates tracking strategy and control parameter suggestions based on the semantic judgment results, and after parameter boundary constraints, the deterministic controller performs steering and zoom closed-loop control on the local PTZ camera; when the tracking is unstable, it performs search control on the local PTZ camera in combination with the panoramic target position and historical motion trend, and resumes closed-loop tracking after the target is confirmed. It realizes hierarchical collaborative tracking with semantic decision-making at the top, constraint control at the bottom, and abnormal recapture as a fallback, which solves the problems of easy misjudgment of cross-view association, difficulty in adaptive fixed parameters, and instability of direct model control in the prior art.
[0096] The following are embodiments of the apparatus of the present invention, which can be used to execute the PTZ camera cooperative target tracking method involved in the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the method embodiments of the PTZ camera cooperative target tracking method involved in the present invention.
[0097] Please see Figure 4 This invention provides a PTZ camera-assisted target tracking device 800.
[0098] The PTZ camera-assisted target tracking device 800 includes, but is not limited to: a panoramic target detection module 810, a target state construction module 830, a cross-view semantic determination module 850, and a control and re-capture module 870.
[0099] Among them, the panoramic target detection module 810 is used to detect targets in panoramic video through a panoramic camera and obtain panoramic information of the panoramic target; the panoramic information includes panoramic target identifier, panoramic target location, panoramic target area, panoramic target scale, panoramic target motion trend and historical trajectory information.
[0100] The target state construction module 830 is used to detect candidate targets in a local area of the image using a local PTZ camera, obtain the local candidate target identifier, local candidate target position and local candidate target scale, and read the current turning angle and zoom status of the PTZ camera to construct a unified target state with the panoramic information.
[0101] The cross-view semantic determination module 850 is used to perform cross-view consistency judgment on panoramic target screenshots, local candidate screenshots, target motion summaries and historical tracking summaries in a unified target state, and output semantic determination results. The semantic determination results include the confidence of the same target, target state category, occlusion risk level, image quality level and candidate target ranking.
[0102] The control and reacquisition module 870 is used to generate tracking strategies and control parameter suggestions based on semantic judgment results. After parameter boundary constraints, it performs steering and zoom closed-loop control on the local PTZ camera. When tracking is unstable, it combines the panoramic target position and historical motion trend to perform search control on the local PTZ camera. After the target is confirmed, closed-loop tracking is resumed.
[0103] It should be noted that the above-described PTZ camera-coordinated target tracking is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the PTZ camera-coordinated target tracking device will be divided into different functional modules to complete all or part of the functions described above.
[0104] Furthermore, the embodiments of the PTZ camera-coordinated target tracking device and the PTZ camera-coordinated target tracking method provided in the above embodiments belong to the same concept. The specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.
[0105] Figure 5 A schematic diagram of the structure of an electronic device according to an exemplary embodiment is shown.
[0106] It should be noted that this electronic device is merely an example adapted to the present invention and should not be construed as providing any limitation on the scope of use of the present invention. Furthermore, this electronic device should not be interpreted as requiring or depending on having... Figure 5 One or more components of the exemplary electronic device 2000 shown.
[0107] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 5 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.
[0108] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.
[0109] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this invention, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 5 As shown, this does not constitute a specific limitation.
[0110] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.
[0111] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0112] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 5 (Not shown), each module may contain computer-readable instructions for the electronic device 2000. For example, the PTZ camera-coordinated target tracking device can be considered as application program 253 deployed on the electronic device 2000.
[0113] Data 255 may be signal information, etc., and is stored in memory 250.
[0114] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby enabling the computation and processing of massive amounts of data 255 in the memory 250. For example, a PTZ-based cooperative target tracking method may be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.
[0115] Furthermore, the present invention can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, or combination thereof.
[0116] Please see Figure 6 This invention provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc., with sensor recognition capabilities.
[0117] exist Figure 6 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.
[0118] The data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0119] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.
[0120] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0121] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 4000, but not limited thereto.
[0122] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.
[0123] The computer-readable instructions are executed by one or more processors 4001 to implement the PTZ camera cooperative target tracking method in the above embodiments.
[0124] Furthermore, this embodiment of the invention provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the PTZ camera cooperative target tracking method described above.
[0125] This invention provides a computer program product, which includes computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, thereby enabling the electronic device to implement the PTZ camera cooperative target tracking method as described above.
[0126] Compared with related technologies, the beneficial effects of the present invention are: 1. This invention can improve the reliability of cross-view target association; by using a multimodal semantic judgment module to perform cross-view consistency judgment on panoramic target screenshots, local candidate screenshots, target motion summaries, and historical tracking summaries in a unified target state, it integrates multi-dimensional information such as panoramic target identifier, panoramic target position, panoramic target region, panoramic target scale, panoramic target motion trend, panoramic target historical trajectory, local candidate target identifier, local candidate target position, local candidate target scale, PTZ camera current turning angle, and PTZ camera current zoom state, and outputs the same target confidence, target state category, occlusion risk level, image quality level, and candidate target ranking, thereby reducing false associations caused by relying solely on geometric position or a single identifier, enabling the system to stably track the same target even when targets intersect, there is short-term occlusion, scale changes, or similar targets exist in local images.
[0127] 2. This invention combines the advantages of intelligent judgment and real-time control. By configuring the multimodal semantic judgment module and strategy generation module to be responsible for high-level semantic judgment and strategy suggestions, and configuring the deterministic controller to be responsible for real-time closed-loop execution, the multimodal model does not directly output continuous control quantities, but only outputs control parameter suggestions such as the confidence level of the same target, the target state category, the occlusion risk level, the image quality level, the candidate target ranking, the response intensity level, the target retention threshold, the target loss tolerance time, and the zoom compensation intensity. After the parameter boundary constraint module performs boundary clipping and legality verification, the parameters are handed over to the deterministic controller for steering and zoom control. This avoids the instability risk caused by the large model directly outputting continuous control commands, and meets the requirements of real-time performance and security while ensuring semantic understanding capabilities.
[0128] 3. This invention can improve tracking stability in complex scenarios. By comprehensively judging whether the current tracking is stable based on the same target confidence level, occlusion risk level, target state category, and closed-loop control execution state, when the same target confidence level meets the threshold for continuing tracking and the occlusion risk level is within an acceptable range, it is determined to be stable and enters the next cycle; otherwise, it enters abnormal re-acquisition processing. Combining the panoramic target position, panoramic target movement trend, panoramic target historical trajectory, PTZ camera current turning angle, PTZ camera current zoom state, and historical observation records, it selects to wait for the target to reappear, predict the direction according to the historical trajectory, conduct a small-range search, expand the field of view search after reducing zoom, or return to the default observation state. After confirming the original target, it restores local closed-loop control, thereby reducing the probability of mistracking and loss in cases of occlusion, similar targets, target scale changes, and local image quality degradation.
[0129] 4. This invention has the advantage of enhancing system adaptability; the strategy generation module adaptively generates tracking strategies such as continue tracking, hold and wait, reduce zoom, enhance search, switch candidate targets, enter the re-acquisition process, or return to the default observation state based on the semantic judgment results. It also generates control parameter suggestions such as response intensity level, target holding threshold, target loss tolerance time, and zoom compensation intensity, so that the tracking strategy and control parameters of the local PTZ camera can be dynamically adjusted according to the target status, scene risk, and observation needs, reducing the manual parameter debugging work in different scenarios and improving the system's adaptability in different application scenarios such as airports, ports, and parks.
[0130] 5. This invention can improve safety and controllability; by using a parameter boundary constraint module to perform boundary clipping and legality verification on response intensity levels, target retention thresholds, target loss tolerance time, and zoom compensation intensity, the range of values suggested by the model is limited, abnormal outputs are filtered, and the system can fall back to the preset default strategy and parameters when the model results are unreliable. At the same time, a deterministic controller uses proportional-integral-derivative control, fuzzy control, or model predictive control to perform local closed-loop control. Even if the multimodal semantic judgment results are missing or the confidence level is insufficient, the system can fall back to the preset deterministic control scheme, thereby ensuring the safety boundary and stable operation of the tracking process.
[0131] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0132] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A PTZ camera cooperative target tracking method, characterized in that, The method includes: Target detection is performed on panoramic video using a panoramic camera to obtain panoramic information of the panoramic target; the panoramic information includes panoramic target identifier, panoramic target location, panoramic target region, panoramic target scale, panoramic target motion trend and historical trajectory information; By using a local PTZ camera to detect candidate targets in a local area of the image, the local candidate target identifier, local candidate target position, and local candidate target scale are obtained. The current turning angle and zoom status of the PTZ camera are read and combined with the panoramic information to construct a unified target state. Cross-view consistency judgment is performed on the panoramic target screenshot, local candidate screenshot, target motion summary and historical tracking summary in the unified target state, and the semantic judgment result is output; the semantic judgment result includes the confidence of the same target, the target state category, the occlusion risk level, the image quality level and the candidate target ranking; Based on the semantic determination results, a tracking strategy and control parameter suggestions are generated. After parameter boundary constraints, the local PTZ camera is subjected to closed-loop control for steering and zoom. When the tracking is unstable, the local PTZ camera is searched and controlled in combination with the panoramic target position and historical motion trend. After the target is confirmed, closed-loop tracking is resumed.
2. The PTZ camera cooperative target tracking method as described in claim 1, characterized in that, The step of detecting targets in panoramic video using a panoramic camera to obtain panoramic information of the panoramic targets includes: The panoramic camera reads the target perception result of the current frame in the panoramic video, and analyzes it to obtain the panoramic information of the panoramic target. If the target perception result is abnormal or there is no target to be tracked, the status is updated and the system waits for the next cycle. The identifier is used to refer to the target; the position is used to determine the coordinates of the target in the panoramic image; the region is used to determine the scene area where the target is located; the scale is used to describe the size of the target in the panoramic image; the motion trend is used to describe the direction and state of movement of the target; and the historical trajectory is used to record the position changes of the target in historical cycles.
3. The PTZ camera cooperative target tracking method as described in claim 1, characterized in that, The step of detecting candidate targets in a localized area of the image using a local PTZ camera, obtaining local candidate target identifiers, local candidate target positions, and local candidate target scales, and then reading the current turning angle and zoom status of the PTZ camera to construct a unified target state with the panoramic information includes: The system reads local candidate targets from the current frame of a local image, obtains the local candidate target identifier, local candidate target position, and local candidate target scale, and reads the current PTZ camera's turning angle and current zoom state. The local candidate target identifier is used to distinguish different candidate targets; the local candidate target position is used to determine the coordinates of the local candidate target in the local image; the local candidate target scale is used to describe the size of the local candidate target in the local image; the current PTZ camera turning angle is used to determine the current pointing direction of the local PTZ camera; and the current zoom state of the PTZ camera is used to determine the current magnification of the local PTZ camera. The local candidate target identifier, local candidate target location, local candidate target scale, PTZ camera current turning angle, and PTZ camera current zoom state are integrated with the panoramic information, scene area information, and business identifier information to form a unified target state. If there is no target to be tracked at present, the state is only updated and the system waits for the next cycle.
4. The PTZ camera cooperative target tracking method as described in claim 1, characterized in that, The process involves performing cross-view consistency judgment on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary in the unified target state, and outputting semantic judgment results, including: A panoramic target screenshot is obtained based on the panoramic target location; a local candidate screenshot is obtained based on the local candidate target location; a target motion summary is generated based on the panoramic target motion trend and the panoramic target historical trajectory; and a historical tracking summary is generated based on the panoramic target historical trajectory and the local candidate target identifier. Cross-view consistency judgment is performed on the panoramic target screenshot, local candidate screenshot, target motion summary, and historical tracking summary to determine whether the panoramic target in the panoramic video and each local candidate target in the local PTZ camera image are the same target. The current state of the target, occlusion risk, and image quality are evaluated to output semantic judgment results. The same target confidence is used to indicate the probability that the panoramic target and the local candidate targets belong to the same target. The target state category is used to indicate the target state. The occlusion risk level is used to indicate the degree of risk of the target being occluded. The image quality level is used to indicate the clarity of the local image. The candidate target ranking is used to indicate the ranking of the consistency between each local candidate target and the panoramic target.
5. The PTZ camera cooperative target tracking method as described in claim 1, characterized in that, The step of generating tracking strategy and control parameter suggestions based on the semantic determination result includes: Based on the semantic determination results, a tracking strategy and control parameter suggestions for the current period are generated. The tracking strategy includes continuing tracking, holding and waiting, reducing zoom, enhancing search, switching candidate targets, entering the recapture process, or returning to the default observation state. The control parameter suggestions include response intensity level, target retention threshold, target loss tolerance time, and zoom compensation intensity. The response intensity level is used to indicate the speed of response of local PTZ camera turning and zoom control. The target retention threshold is used to indicate the determination threshold required for the target to remain within a preset area of the local image. The target loss tolerance time is used to indicate the length of time allowed to wait and continue searching after the target disappears briefly. The zoom compensation intensity is used to indicate the magnitude of compensation for zoom control when the target scale changes.
6. The PTZ camera cooperative target tracking method as described in claim 5, characterized in that, The closed-loop control of steering and zoom of the local PTZ camera after parameter boundary constraints includes: The response intensity level, target retention threshold, target loss tolerance time, and zoom compensation intensity are subject to boundary clipping and legality verification, value range restriction, and abnormal output filtering. If the result is unreliable, the system will revert to the preset default strategy and parameters. The steering control quantity is calculated based on the positional deviation between the local candidate target position and the preset area using proportional-integral-derivative control, fuzzy control, or model predictive control. The zoom control quantity is calculated based on the scale deviation between the local candidate target scale and the desired scale. When the local candidate target deviates from the preset area when the local image is magnified, the steering control and zoom control are coupled and compensated.
7. The PTZ camera cooperative target tracking method as described in claim 1, characterized in that, When tracking is unstable, the local PTZ camera is searched and controlled by combining the panoramic target position and historical motion trends. After the target is confirmed, closed-loop tracking is resumed, including: If the confidence level of the same target meets the threshold for continuing tracking and the occlusion risk level is within an acceptable range, then the current tracking is determined to be stable, the status is updated, and the next cycle begins. Otherwise, if the current tracking is deemed unstable, the system will determine whether to continue searching or return to the default observation state based on the panoramic target position, panoramic target movement trend, panoramic target historical trajectory, PTZ camera current turning angle, PTZ camera current zoom state, and historical observation records. The continued search includes waiting for the target to reappear, predicting the direction based on the historical trajectory, searching within a small area, and expanding the field of view by reducing zoom. For newly discovered local candidate targets, a cross-view consistency judgment is made. If the target is confirmed as the original target, local closed-loop control is restored. If the target loss tolerance time is exceeded and the target is still not confirmed, the current tracking is stopped or the target is switched to a new candidate target.
8. A PTZ camera-assisted target tracking device, characterized in that, The device includes: The panoramic target detection module is used to detect targets in panoramic video using a panoramic camera and obtain panoramic information of the panoramic targets. The panoramic information includes panoramic target identifier, panoramic target location, panoramic target region, panoramic target scale, panoramic target motion trend and historical trajectory information. The target state construction module is used to detect candidate targets in a local area of the image using a local PTZ camera, obtain the local candidate target identifier, local candidate target position and local candidate target scale, and read the current turning angle and zoom status of the PTZ camera to construct a unified target state with the panoramic information; The cross-view semantic determination module is used to perform cross-view consistency judgment on the panoramic target screenshot, local candidate screenshot, target motion summary and historical tracking summary in the unified target state, and output the semantic determination result; the semantic determination result includes the confidence of the same target, the target state category, the occlusion risk level, the image quality level and the candidate target ranking. The control and reacquisition module is used to generate tracking strategies and control parameter suggestions based on the semantic determination results, and perform steering and zoom closed-loop control on the local PTZ camera after parameter boundary constraints. When the tracking is unstable, the local PTZ camera is searched and controlled in combination with the panoramic target position and historical motion trend. After the target is confirmed, closed-loop tracking is resumed.
9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein, The memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the PTZ camera cooperative target tracking method as described in any one of claims 1 to 7.
10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the PTZ camera cooperative target tracking method as described in any one of claims 1 to 7.
