Multi-holder camera control method and device based on digital twinning and storage medium

Abstract

The invention discloses a multi-pan-tilt camera control method and device based on digital twinning and a storage medium, the method is applied to a server, and the server is in communication connection with a plurality of pan-tilt cameras in a current scene. Comprising the following steps: constructing a digital twin according to scene information of a current scene and equipment information of each pan-tilt camera in the current scene; in response to the received target detection task, determining an adaptive value of each PTZ camera to the target detection task according to the task priority of the target detection task, the task execution cost of executing the target detection task by each PTZ camera and the comprehensive load of each PTZ camera; determining a target PTZ camera from the PTZ cameras according to the digital twin and the adaptation value; and controlling the target pan-tilt camera to execute the target detection task. According to the scheme, the control efficiency of the multi-pan-tilt camera can be improved.

Description

Technical Field

[0001] This application relates to the field of digital twin technology, and in particular to a method, device and storage medium for controlling a multi-panel camera based on digital twins. Background Technology

[0002] A pan-tilt camera is an image acquisition device equipped with a mechanism that allows the camera to rotate horizontally and / or vertically.

[0003] It is common to see multiple pan-tilt cameras in a target detection scenario. By rotating the pan-tilt unit, the field of view of these cameras can cover the target detection scenario as much as possible, thereby reducing blind spots and improving the reliability of target detection.

[0004] However, in existing PTZ camera control methods, each PTZ camera operates independently, requiring manual control of each camera individually. This makes it extremely easy to lose the target when multiple PTZ cameras are detecting the same target. Even with some preset multi-PTZ linkage strategies, their fixed and rigid control logic is difficult to adapt to dynamically changing targets and scenarios. Summary of the Invention

[0005] This application provides at least one method, apparatus, device, and computer-readable storage medium for controlling a multi-panel camera based on digital twins.

[0006] The first aspect of this application provides a multi-panel camera control method based on digital twins. This method is applied to a server, which has a communication connection with multiple pan-tilt cameras in a current scene. The method includes: constructing a digital twin based on scene information of the current scene and device information of each pan-tilt camera in the current scene; responding to a received target detection task, determining an adaptation value for each pan-tilt camera to the target detection task based on the task priority of the target detection task, the task execution cost of each pan-tilt camera performing the target detection task, and the overall load of each pan-tilt camera; determining a target pan-tilt camera from among the pan-tilt cameras based on the digital twin and the adaptation value; and controlling the target pan-tilt camera to perform the target detection task.

[0007] In one embodiment, before determining the adaptation value of each PTZ camera to the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera, the method further includes: obtaining the basic priority of the target detection task; analyzing the influence factors of the target object on the target detection task; and adjusting the basic priority according to the influence factors to obtain the task priority.

[0008] In one embodiment, the influencing factors include at least one of a behavioral anomaly factor, a spatiotemporal urgency factor, and an observation quality factor. Analyzing the influencing factors of the target object on the target detection task includes: performing behavioral analysis on the target object to obtain the behavioral anomaly factor; performing motion analysis on the target object to obtain the spatiotemporal urgency factor between the target object and the target detection area in the current scene; and determining the observation quality factor based on the observation value of the target object by the PTZ camera.

[0009] In one embodiment, before determining the adaptation value of each PTZ camera to the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera, the method further includes: obtaining the health status information of the PTZ camera, the health status information including at least one of continuous running time, historical running load, historical fault records, and component validity period; and determining the overall load of the PTZ camera based on the current running load of the PTZ camera and the health status information.

[0010] In one embodiment, determining the adaptation value of each PTZ camera to the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera includes: performing a weighted summation of the task execution cost and the overall load to obtain a device screening benchmark value; and determining the adaptation value based on the ratio between the task priority and the device screening benchmark value.

[0011] In one embodiment, determining the target PTZ camera from among the PTZ cameras based on the digital twin and the adaptation value includes: selecting a first PTZ camera with the highest adaptation value to simulate the target detection task in the digital twin, obtaining a first simulation result; in response to the first simulation result not meeting the preset task requirements, selecting a second PTZ camera with an adaptation value lower than that of the first PTZ camera to simulate the target detection task in the digital twin, obtaining a second simulation result; and in response to the second simulation result meeting the preset task requirements, determining the second PTZ camera as the target PTZ camera.

[0012] In one embodiment, before controlling the target PTZ camera to perform the target detection task, the method further includes: constructing a graph model based on each PTZ camera and the target object; analyzing the graph model to obtain target detection conflicts between the target PTZ camera and other PTZ cameras; and adjusting the other PTZ cameras based on the target detection conflicts.

[0013] In one embodiment, after controlling the target PTZ camera to perform the target detection task, the method further includes: performing trajectory prediction processing based on the motion information of the target object to obtain a predicted trajectory of the target object; determining a takeover PTZ camera from among the PTZ cameras based on the predicted trajectory; determining the departure time of the target object leaving the field of view of the target PTZ camera based on the predicted trajectory and the motion information; and assigning the target detection task to the takeover PTZ camera based on the departure time.

[0014] A second aspect of this application provides a multi-panel camera control device based on digital twins. This device is applied to a server, which has a communication connection with multiple pan-tilt cameras in a current scene. The device includes: a twin construction module for constructing a digital twin based on scene information of the current scene and device information of each pan-tilt camera in the current scene; an adaptation value determination module for determining an adaptation value for each pan-tilt camera to the target detection task in response to a received target detection task, based on the task priority of the target detection task, the task execution cost of each pan-tilt camera performing the target detection task, and the overall load of each pan-tilt camera; a pan-tilt determination module for determining a target pan-tilt camera from among the pan-tilt cameras based on the digital twin and the adaptation value; and a pan-tilt control module for controlling the target pan-tilt camera to perform the target detection task.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described multi-panel camera control method based on digital twins.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described multi-panel camera control method based on digital twins.

[0017] The above scheme constructs an equivalent virtual information model by acquiring scene information and device information of each PTZ camera in the current scene, thus obtaining a digital twin corresponding to the current scene. Upon receiving a target detection task for the current scene, it can acquire the task priority, execution cost of each PTZ camera performing the task, and overall load of each PTZ camera, thereby determining the suitability value of each PTZ camera for the target detection task. By analyzing the simulation execution results of each PTZ camera for the target detection task using the digital twin, and combining this with the suitability value of each PTZ camera for the target detection task, a suitable target PTZ camera for performing the target detection task is determined. Therefore, target detection tasks can be assigned to target PTZ cameras in real space, and the target PTZ cameras can be controlled to perform the target detection tasks. By constructing a global digital twin and an intelligent decision engine, it can solve the problem of seamless system-level collaborative control and resource optimization in multi-PTZ camera scenarios, improving the control efficiency of multi-PTZ cameras.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a flowchart illustrating an exemplary embodiment of the multi-panel camera control method based on digital twins according to this application; Figure 2 This is a block diagram illustrating a multi-panel camera control device based on digital twin, as shown in an exemplary embodiment of this application. Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0021] The solution of this embodiment will now be described in detail with reference to the accompanying drawings.

[0022] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0023] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] To facilitate understanding, one of the applicable scenarios of this application will be illustrated by example.

[0025] A pan-tilt camera is an image acquisition device equipped with a mechanism that allows the camera to rotate horizontally and / or vertically.

[0026] It is common to see multiple pan-tilt cameras in a target detection scenario. By rotating the pan-tilt unit, the field of view of these cameras can cover the target detection scenario as much as possible, thereby reducing blind spots and improving the reliability of target detection.

[0027] However, in existing PTZ camera control methods, each PTZ camera operates independently, requiring manual control of each camera individually. This makes it extremely easy to lose the target when multiple PTZ cameras are detecting the same target. Even with some preset multi-PTZ linkage strategies, the fixed and rigid control logic is difficult to adapt to dynamically changing targets and scenes, resulting in low overall PTZ control efficiency and poor target detection quality.

[0028] For ease of explanation, the pan-tilt camera will be referred to as a pan-tilt camera in the following embodiments, and will not be described again hereafter.

[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the multi-pan-tilt camera control method based on digital twins according to this application. The method can be applied to a server, which can have communication connections with multiple pan-tilt cameras in the current scene.

[0030] The server can be considered as the digital twin-based multi-pan-tilt camera control system of this application, or the server and the pan-tilt camera (and / or more types of sensors) can be considered together as the digital twin-based multi-pan-tilt camera control system of this application, which will not be elaborated here.

[0031] Specifically, the multi-panel camera control method of this application may include the following steps: Step S110: Construct a digital twin based on the scene information of the current scene and the device information of each PTZ camera in the current scene.

[0032] A digital twin is a virtual information model, a digital representation created in a computer virtual space that is completely equivalent to a physical entity. It can achieve dynamic mapping of the entire lifecycle of a physical entity by integrating multi-source data such as physical models, sensor data, and operational history.

[0033] The specific methods for constructing digital twins can be found in the explanations of related technologies in this field, and will not be elaborated here.

[0034] The digital twin may include an environment model of the current scene (which may include scene information of the current scene) and a PTZ proxy model (which may include device information of the PTZ camera); in addition, if there is a target object in the current scene, the digital twin may also include a target object model (which may include object information of the target object).

[0035] The current scene refers to the scene where object detection needs to be performed. The current scene can include the field of view of the PTZ camera. For example, the area of ​​the current scene can be greater than, less than or equal to the area corresponding to the field of view of the PTZ camera; there is no limitation here.

[0036] The environment model of the current scene may include, but is not limited to, a two-dimensional and / or three-dimensional scene constructed from environmental information such as maps, CAD drawings, and / or SLAM technology corresponding to the current scene. Furthermore, the environmental information may also include relevant information about some objects and / or some areas in the current scene (such as semantic information about obstacles, key areas, and detection blind spots), which is not limited here.

[0037] The PTZ proxy model is a virtual entity of the PTZ camera in the digital twin. It can include the equivalent position, posture, field of view (which can be represented by a cone), rotation capability, working status (such as idle, target detection, scanning, etc.), real-time load (such as CPU utilization, memory utilization, number of currently detected targets, etc.) and health status of the physical entity of the PTZ camera in real space (current scene).

[0038] The target object model is a virtual representation of the target object detected in the current scene. It can include object information such as the target object's position, velocity, motion trajectory (and may also include the predicted trajectory), posture, and behavior.

[0039] It should be noted that the method for obtaining the above example information (such as scene information, device information, and object information) can be that the PTZ camera collects and transmits the information to the server. Specifically, the PTZ camera can process the collected information before transmitting it to the server, and / or the raw collected information can be transmitted to the server for further processing; this is not limited here.

[0040] For example, the PTZ camera control system of this application can be based on an edge-cloud collaborative architecture. It consists of a central collaborative server (with an embedded collaborative decision engine) and multiple PTZ control nodes. This allows for the construction and maintenance of a globally dynamic digital twin. Optionally, a regional collaborative gateway can also be provided between the server and the PTZ control nodes to coordinate the server and the PTZ.

[0041] Among them, the PTZ control node can be an external node that has a communication connection with the PTZ camera to control the PTZ camera, and / or the PTZ camera has a built-in control module for controlling according to the received PTZ control commands.

[0042] For example, each gimbal can have a built-in lightweight AI processing unit (such as Jetson Nano) with capabilities such as local object detection, trajectory prediction, and emergency obstacle avoidance. Furthermore, it can autonomously maintain its detection task and upload the processed structured data even during network outages or high latency.

[0043] Regional collaboration gateway: A common regional collaboration gateway can be set up for PTZ cameras in a certain area (for example, a regional collaboration gateway can be set up for every 5-10 PTZ cameras that are geographically close).

[0044] For example, a gateway can be used to provide communication conditions between a server and a PTZ camera. And / or the gateway can also have a built-in neural network model to preprocess the data the server needs to process, or to perform some functions on behalf of the server in this application. For example, it can be responsible for the coordinated scheduling of PTZ cameras within its jurisdiction, task takeover pre-calculation, load balancing, and conflict resolution.

[0045] This means that the regional collaborative gateway can also execute some of the methods in this application, thereby sharing the processing pressure of the central collaborative server, realizing regional autonomy, and significantly reducing response latency, which will not be elaborated further.

[0046] Central Collaboration Server: Maintains the digital twin, runs the collaborative decision-making engine, and is responsible for formulating optimization strategies, predicting trajectories, learning from historical data, and managing system-level tasks.

[0047] Specifically, in the example scenario described above, each PTZ control node and regional collaborative gateway can upload local target detection results (which may include target ID, location, speed, orientation, etc.) to the central server in real time. The central server can then merge this data and update the status of the corresponding target object and all PTZs in the digital twin, ensuring synchronization between the virtual space and the real space.

[0048] Step S120: In response to the received target detection task, determine the adaptation value of each PTZ camera to the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera.

[0049] Here, the target detection task refers to the task of detecting target objects. Target detection tasks may include, but are not limited to: tasks generated autonomously by the server through analysis of received data, and / or tasks generated by analysis of data collected by PTZ cameras (and / or other types of sensors) and then uploaded to the server, and / or tasks generated by analysis of received data by regional collaborative gateways and then uploaded to the server, and / or tasks generated upon receiving target detection commands issued by users, etc., etc., without limitation here.

[0050] For example, a target detection task can be triggered by analyzing image data collected by a PTZ camera and determining the presence of a target object in the current scene; and / or by analyzing radar data collected by a radar sensor and determining the presence of a target object in the current scene; and / or by analyzing infrared data collected by an infrared device and determining the presence of a target object in the current scene, etc., which will not be elaborated here. Other types of sensors, such as radar and infrared devices, may have partially overlapping, completely overlapping, or completely non-overlapping fields of view with the PTZ camera's visual sensor; this is not limited here.

[0051] This application may include one or more types of object detection tasks, and different types of object detection tasks may exist simultaneously or in different times and spaces. Different object detection tasks may have the same or different task priorities. For example, object detection tasks may include pedestrian detection tasks, vehicle detection tasks, pet detection tasks, etc., without limitation here.

[0052] It should be noted that the task priority of the object detection task can be preset and fixed, or it can be dynamically adjusted; there is no limitation here.

[0053] For ease of understanding and explanation, the following text will mainly use a newly triggered object detection task as an example application scenario.

[0054] Understandably, in the application scenarios of multi-panel cameras, although the field of view of a multi-panel camera can be non-overlapping, partially overlapping, or completely overlapping, the field of view (or angle of view) of a multi-panel camera is generally different.

[0055] Therefore, if some or all of the PTZ cameras are to perform target detection tasks, the PTZ needs to be rotated to ensure that the target object is present in the camera's field of view after rotation. Common application scenarios also include keeping the target object in the center of the PTZ camera's field of view or other positions, which will not be elaborated here. Furthermore, PTZ control can be adjusted based on the target object's motion information to achieve continuous detection of the target object.

[0056] The task execution cost of a PTZ camera performing a target detection task refers to the time cost incurred when the PTZ camera performs the same target detection task. For example, the time required for the PTZ camera to rotate to the optimal observation angle of the target object (e.g., to center the target object in the PTZ camera's field of view).

[0057] The overall load of the PTZ camera can be calculated and represented based on the current operating load (such as CPU utilization, memory utilization, number of currently detected targets, etc.) when target detection tasks need to be assigned. On this basis, the health status of the PTZ camera can also be selected to participate in the calculation, which is not limited here.

[0058] Specifically, the adaptation value for each PTZ camera can be determined based on its task priority, task execution cost, and overall load when performing target detection tasks. For example, the mathematical expression of the adaptation value can be: Bid_k,i=P_dynamic_i / (T_k,i+λ*L'_k) Where, Bid_k,i refers to the adaptation value when the k-th PTZ camera performs the target detection task for the i-th target object. P_dynamic_i refers to the task priority of the target detection task corresponding to the i-th target object. T_k,i refers to the task execution cost when the k-th PTZ camera performs the target detection task for the i-th target object. L'_k refers to the overall load of the k-th PTZ camera. λ is the load impact coefficient (mainly used to control the impact of operating load on the adaptation value).

[0059] Step S130: Determine the target PTZ camera from each PTZ camera based on the digital twin and the adaptation value.

[0060] Based on the steps described above, after obtaining the adaptation value of each PTZ camera for the target detection task, the PTZ cameras can be selected in descending order of adaptation value to simulate and execute the target detection task in the digital twin, and the corresponding simulation execution results can be obtained.

[0061] For example, a PTZ camera can be selected based on the adaptation value (such as the PTZ camera with the highest adaptation value, or the PTZ camera with the second highest adaptation value, or the PTZ camera with other adaptation value rankings), or multiple PTZ cameras with high adaptation values ​​can be selected based on the adaptation value to simulate the target detection task and obtain their respective simulation execution results.

[0062] Taking the PTZ camera with the highest adaptation value as an example, if its simulation execution result meets the preset task requirements, then the PTZ camera with the highest adaptation value is determined as the target PTZ camera.

[0063] If the simulation execution result does not meet the preset task requirements, the pan-tilt camera with the second highest adaptation value can be selected to simulate the target detection task according to the adaptation value order. The corresponding simulation execution result is obtained and compared with the preset task requirements to determine whether the pan-tilt camera with the second highest adaptation value can be identified as the target pan-tilt camera.

[0064] If the simulation results of the PTZ camera with the second highest adaptation value still do not meet the preset task requirements, the subsequent process can be iterated by referring to the aforementioned example method, which will not be repeated here.

[0065] The process of multiple PTZ cameras simulating the target detection task can be executed in parallel or in order of adaptation value size; this is not limited here.

[0066] Step S140: Control the target PTZ camera to perform target detection task.

[0067] Based on the steps described above, after identifying the target PTZ camera, the target detection task for the target object can be assigned to the target PTZ camera, and the camera can be controlled to execute the target detection task.

[0068] Among them, the control commands for the target PTZ camera can be generated during the simulation execution of the target detection task by the digital twin, and can be obtained from the task simulation results (for example, including but not limited to controlling the angle of PTZ rotation, controlling the speed of PTZ rotation, controlling the aperture and focal length of PTZ when performing the target detection task).

[0069] Alternatively, control commands for the target PTZ camera can be determined and generated again based on the object information of the target object acquired in real time after the target PTZ camera has been identified. Or, control commands for the target PTZ camera can be based on corresponding control commands received from the user; this is not limited here.

[0070] As can be seen, this application constructs an equivalent virtual information model by acquiring scene information of the current scene and device information of each PTZ camera in the current scene, thus obtaining a digital twin corresponding to the current scene. Upon receiving a target detection task for the current scene, the task priority of the target detection task, the execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera can be obtained, thereby determining the adaptation value of each PTZ camera to the target detection task. By analyzing the simulation execution results of each PTZ camera on the target detection task through the digital twin, and combining the adaptation value of each PTZ camera on the target detection task, a suitable target PTZ camera for performing the target detection task can be determined from among the various PTZ cameras. Therefore, the target detection task can be assigned to the target PTZ camera in the real space, and the target PTZ camera can be controlled to perform the target detection task. By constructing a global digital twin and an intelligent decision engine, the problem of seamless collaborative control and resource optimization at the system level in multi-PTZ camera scenarios can be solved, improving the control efficiency of multi-PTZ cameras.

[0071] Based on the above embodiments, this embodiment should also explain that, for target detection tasks that require special attention, this application can also perform simulation execution through multiple digital twins, and determine the final strategy for executing the target detection task based on the simulation execution results output by the multiple digital twins.

[0072] For example, after receiving a target detection task, the task priority of the target detection task can be compared with a preset priority threshold. If the task priority is greater than the priority threshold, the target detection task is a task that needs to be given priority attention.

[0073] Therefore, the decision engine can create multiple parallel predictive twins in the digital twin, and different predictive twins can be responsible for simulating different task allocation strategies or gimbal rotation path planning strategies.

[0074] The decision engine runs these simulation processes in parallel and evaluates the simulation performance of each predictive twin (i.e., the predictive performance of each strategy) according to a predefined cost function. Finally, it selects the optimal strategy and sends it to the target PTZ camera for execution. The cost function can be determined by factors such as the execution time cost for a specific task, the overlap rate of the target PTZ camera's field of view with other PTZ cameras, and the coverage of the target PTZ camera's field of view of the predicted trajectory of the target object. These factors are not limited here. Further details can be found in other methods available in this field for setting cost functions, which will not be elaborated upon here.

[0075] Optionally, the regional collaboration gateway can undertake the policy simulation process for its corresponding region to reduce the load on the central server.

[0076] Finally, based on the simulation execution results of multiple predicted twins, specific and optimized sub-tasks can be assigned to each PTZ camera (such as "PTZ A, responsible for performing task T1 to detect the target object at the optimal angle", "PTZ B, responsible for performing the area scanning task along path P1").

[0077] Based on the above embodiments, this embodiment describes the method prior to step S120.

[0078] The foregoing embodiments illustrate that the task priority of the target detection task can be preset or can be dynamically adjusted. This embodiment mainly illustrates the method of dynamically adjusting the task priority.

[0079] Specifically, in this embodiment, the method before determining the adaptation value of each PTZ camera for the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera includes at least the following steps S210 to S230: Step S210: Obtain the basic priority of the target detection task.

[0080] The basic priority can be preset for various detection tasks. Similarly, referring to the description of the foregoing embodiments, the corresponding basic priority can be determined based on the task type of the target detection task.

[0081] If you do not choose to dynamically adjust the task priority of the target detection task, you can choose to directly determine the basic priority of the target detection task as its task priority, which will not be elaborated here.

[0082] Step S220: Analyze the factors that influence the target object on the target detection task.

[0083] Among them, the impact factor of the target object on the target detection task is mainly used to reflect the degree of influence of the target object on the priority of the target detection task in actual application scenarios.

[0084] The influencing factor can be determined by analyzing the behavior and / or motion information of the target object, and / or by analyzing the observation quality of the target object by the pan-tilt camera, which is not limited here.

[0085] For example, whether a target object exhibits dangerous behavior in certain application scenarios can affect the judgment of whether the target task needs to be given special attention.

[0086] Step S230: Adjust the basic priority based on the impact factor to obtain the task priority.

[0087] Based on the steps outlined above, after determining the impact factors of the target object on the target detection task, the basic priority of the target detection task can be dynamically adjusted according to the impact factors to obtain the task priority of the target detection task.

[0088] For example, the dynamic adjustment method may include, but is not limited to, adjusting the basic priority based on the impact factor to obtain the task priority. Alternatively, it may be to perform a weighted summation of the impact factor and the basic priority to obtain the task priority; this is not limited here.

[0089] Based on the above embodiments, this embodiment describes the method for analyzing the influence factors of the target object on the target detection task in step S220.

[0090] Specifically, the method for analyzing the influence factors of target objects on target detection tasks in this embodiment includes at least the following steps S310 to S330: Step S310: Perform behavioral analysis on the target object to obtain the behavioral anomaly factor.

[0091] Among them, behavior analysis processing is mainly used to analyze the abnormality of the target object's current behavior and obtain the corresponding behavior abnormality factor.

[0092] The specific methods for behavior analysis processing can be found in the relevant methods for behavior analysis of target objects in this field, and will not be elaborated here.

[0093] For example, one can simply choose methods such as numerical comparison or preset calculation functions for analysis and judgment (e.g., if the movement speed is higher than the speed threshold, it is judged as abnormal), or one can choose a pre-trained neural network for analysis and judgment in a more complex way; there is no limitation here.

[0094] The behavior anomaly factor can be divided into two different values ​​depending on whether the target object's current behavior is abnormal or not. It can also be calculated using a specific function or neural network to obtain multiple values ​​(e.g., different movement speeds correspond to different behavior anomaly factors), which is not limited here. The behavior anomaly factor corresponding to abnormal behavior is greater than the behavior anomaly factor corresponding to normal behavior (no anomaly).

[0095] For example, a pre-trained neural network can be used to analyze the motion trajectory of a target object to determine whether the target object exhibits wandering behavior, or the motion trajectory and speed of the target object can be analyzed to determine whether the target object exhibits high-speed change-of-direction behavior, and the corresponding behavior anomaly factor can be output.

[0096] For example, based on the current behavior analyzed in the above examples, the current behavior of the target object can be compared with its historical behavior to obtain the corresponding behavior anomaly factor. The historical behavior is earlier in time than the current behavior, and its acquisition method is the same as that used for the current behavior in the previous examples, so it will not be elaborated here.

[0097] Step S320: Perform motion analysis on the target object to obtain the spatiotemporal urgency factor between the target object and the target detection area in the current scene.

[0098] Motion analysis processing is mainly used to analyze whether there are target objects approaching the target detection area in the current scene. Common applications include perimeter detection and tripwire detection.

[0099] The target detection area can be pre-defined or temporarily generated (for example, the target detection area is defined for the target event when a traffic accident or other target event that needs to be paid attention to is detected in the current scene; for specific methods, please refer to the relevant technologies in this field), and is not limited here.

[0100] It should be noted that the target detection area can refer to the area where the PTZ camera can detect targets, or it can refer to the blind spot where the PTZ camera cannot detect targets; there is no limitation here.

[0101] For example, by combining the current motion information of the target object with the target detection area in the current scene, the estimated remaining distance between the target object and the target detection area can be obtained, and / or the estimated remaining time when the target object is about to reach the target detection area (or is about to enter the target detection area). Based on the estimated remaining distance and / or estimated remaining time, the spatiotemporal urgency factor between the target object and the target detection area can be determined.

[0102] The estimated remaining distance can refer to the straight-line distance between the target object and the target detection area, or the Manhattan distance, etc., without limitation here.

[0103] The estimated remaining distance and / or estimated remaining time are negatively correlated with the spatiotemporal urgency factor. That is, the shorter the estimated remaining distance and / or the less the estimated remaining time, the greater the spatiotemporal urgency factor.

[0104] The method for determining the spatiotemporal urgency factor can refer to the explanation of the behavioral anomaly factor in the aforementioned example. For example, it can be analyzed and judged by simply selecting numerical comparison, preset calculation functions, etc., or it can be analyzed and judged by more complex methods such as selecting pre-trained neural networks. No limitation is made here. Specific methods can also refer to the relevant methods in this field, which will not be elaborated here.

[0105] Step S330: Determine the observation quality factor based on the observation value of the target object by the PTZ camera.

[0106] The observation quality factor is mainly used to quantify the data quality of video image data that a pan-tilt camera can obtain when detecting a target object.

[0107] The calculation model for determining the observation quality factor based on observation value can comprehensively consider the compatibility between spatial geometry, imaging quality and target detection tasks, ensuring that the system prioritizes pan-tilt cameras that can provide clear, rich and relevant perspectives for target detection tasks.

[0108] For example, the mathematical expression for determining the observation quality factor can be: Q_observation_i,k=F_geometry(i,k)×F_image_quality(k)×F_task(i,k)×(1-O(i,k)) Here, Q_observation_i,k refers to the observation quality factor of the k-th PTZ camera (hereinafter referred to as PTZ k) for target detection of the i-th target object (hereinafter referred to as target i).

[0109] F_geometry(i,k) is the geometric relationship factor for gimbal k to perform target detection on target i. Based on the estimated size of target i, the distance and relative angle between target i and gimbal k, and the pre-calibrated optical parameters of gimbal k, it calculates the expected pixel area ratio and key feature visibility (such as the pixel ratio of the frontal area of ​​the target object in the entire image) of target i in the image captured by gimbal k. This geometric relationship factor guides the system to prioritize gimbals that are at a suitable distance and have a directly facing viewpoint.

[0110] F_image_quality(k) is the image quality factor of the image acquired by gimbal k from target i. It can be comprehensively evaluated based on the real-time lighting conditions, weather conditions, and the gimbal's own imaging performance parameters (such as resolution, ISO sensitivity, etc.) at the location of gimbal k. Based on the image quality factor, the system can be guided to prioritize gimbals with better imaging quality, ensuring observation under optimal imaging conditions.

[0111] F_task(i,k) is the task adaptation factor for gimbal k to perform target detection on target i. The matching degree score can be calculated based on the specific requirements of the target detection task (such as feature recognition, license plate shooting, behavior analysis) and the preset functional characteristics of gimbal k (such as telephoto gimbals being better at close-up shooting of target objects, and wide-angle gimbals being better at comprehensive detection of target objects and their surrounding environment).

[0112] O(i,k) (occlusion rate) refers to the percentage of the line of sight from gimbal k to target i that is occluded by static or dynamic objects in the 3D scene of a digital twin, calculated using ray collision detection technology. (1-O(i,k)) is the effective observation coefficient. Specific ray collision detection techniques can be found in relevant fields and will not be elaborated upon here.

[0113] By using the above example method to quantitatively evaluate the observation value from multiple dimensions, the observation quality factor Q_observation_i,k is obtained. This allows the system to accurately guide the allocation of tasks to pursue the highest information acquisition benefit, rather than simply considering a single strategy such as the gimbal being closest to the target object or selecting an idle gimbal.

[0114] Based on the above embodiments, this embodiment describes the method prior to step S120.

[0115] The foregoing embodiments illustrate that the overall load of the PTZ camera of this application can be calculated from at least the operating load of the PTZ camera.

[0116] This embodiment mainly illustrates that, in the process of calculating the comprehensive load based on the operating load of the PTZ camera, the health status information of the PTZ camera can also be incorporated into the calculation.

[0117] Specifically, before determining the adaptation value of each PTZ camera for the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera, this embodiment includes at least the following steps S410 to S420: Step S410: Obtain the health status information of the PTZ camera. The health status information includes at least one of the following: continuous running time, historical operating load, historical fault records, and component validity period.

[0118] Among them, health status information is relevant information used to reflect the health status of the PTZ camera. It may include one or more of the following: continuous operating time, historical operating load, historical fault records, and component expiration date.

[0119] Step S420: Determine the overall load of the PTZ camera based on its current operating load and health status information.

[0120] For ease of calculation, in this embodiment, health status information, including but not limited to the examples above, can be quantified and then combined with the current operating load to determine the overall load.

[0121] For example, the device health index is negatively correlated with continuous operating time. The longer the continuous operating time of the PTZ camera, the lower the device health index. For instance, a baseline health level can be pre-set for each PTZ camera, and then the baseline health level can be adjusted based on the health status information (i.e., the longer the continuous operating time, the lower the baseline health level).

[0122] Historical operating load can be calculated by obtaining the operating load of the PTZ camera during its historical operation (such as calculating the average CPU load and / or average memory load of the PTZ camera during historical operation). Similarly, the device health index is negatively correlated with historical operating load. The higher the historical operating load of the PTZ camera, the lower the device health index.

[0123] Historical fault records can be obtained by analyzing fault counters for recent issues such as communication interruptions and PTZ control malfunctions, or by reading device logs. Similarly, the device health index is negatively correlated with historical fault records. The more historical fault records a PTZ camera has, the lower its device health index.

[0124] The lifespan of a component refers to the service life of the components used in a PTZ camera (such as the PTZ motor and gears). For example, the remaining lifespan of these components can be estimated based on the designed lifespan of the component and the actual rotation mileage (or cumulative operating time).

[0125] By inputting one or more health status information, including but not limited to the examples above, into a pre-defined calculation function or a pre-trained neural network model, a quantified health status index H_k can be obtained.

[0126] H_k can be normalized to between 0 and 1 and updated periodically. During task allocation, the health status index can be used to reduce the adaptation value of PTZs with poor health, thereby naturally reducing the number of tasks assigned to them, achieving self-protection and load balancing of system resources, and extending the overall equipment lifespan.

[0127] After determining the health status index H_k, the overall load L'_k of the PTZ camera can be determined by combining it with the current operating load L_k of the PTZ camera. Its mathematical expression can be: L'_k=L_k / H_k If the current operating load of the PTZ camera is stable, the higher the health status index (the healthier the PTZ camera), the lower the overall load of the PTZ camera, which in turn allows the PTZ camera to have a higher compatibility value.

[0128] Based on the above embodiments, this embodiment describes the method in step S120 for determining the adaptation value of each PTZ camera to the target detection task according to the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera.

[0129] Specifically, the method for determining the adaptation value of each PTZ camera for the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera in this embodiment includes at least the following steps S510 to S520: Step S510: The task execution cost and the overall load are weighted and summed to obtain the equipment screening benchmark value.

[0130] The equipment selection benchmark is used to guide the system to prioritize the selection of gimbals that are efficient, healthy, and have a light load for target detection tasks. Therefore, the equipment selection benchmark can be determined based on both task execution cost and overall load.

[0131] Referring to the foregoing embodiments, T_k,i is the task execution cost, and L'_k is the overall load. Therefore, by performing a weighted summation of T_k,i and L'_k (e.g., T_k,i + λ * L'_k), the device selection benchmark value can be obtained.

[0132] Step S520: Determine the fit value based on the ratio between the task priority and the device screening benchmark value.

[0133] Similarly, referring to the description of the foregoing embodiments, the adaptation value Bid_k,i is determined based on the ratio between the task priority P_dynamic_i and the device screening benchmark value (T_k,i+λ*L'_k), which will not be elaborated here.

[0134] Based on the above embodiments, this embodiment describes the method for determining the target PTZ camera from each PTZ camera in step S130 according to the digital twin and the adaptation value.

[0135] Specifically, the method for determining the target PTZ camera from each PTZ camera based on the digital twin and the adaptation value in this embodiment includes at least the following steps S610 to S630: Step S610: Select the first PTZ camera with the highest adaptation value and simulate the target detection task in the digital twin to obtain the first simulation result; Step S620: In response to the first simulation result not meeting the preset task requirements, a second PTZ camera with an adaptation value smaller than that of the first PTZ camera is selected to simulate the target detection task in the digital twin, and a second simulation result is obtained. Step S630: In response to the second simulation result meeting the preset task requirements, the second PTZ camera is determined as the target PTZ camera.

[0136] The method in this embodiment can be understood in conjunction with the description of the aforementioned embodiment (such as step S130). For example, the pan-tilt cameras are selected sequentially in descending order of their adaptation values ​​to simulate the target detection task in the digital twin until a pan-tilt camera whose simulation execution result meets the preset task requirements is determined. (Or, if there are a large number of pan-tilt cameras, when iterating a certain number of times or when a pan-tilt camera with an adaptation value lower than the preset adaptation threshold is found, if there is still no pan-tilt camera whose simulation execution result meets the preset task requirements, the iteration search is stopped and the corresponding prompt information is output to inform the user.)

[0137] The preset task requirements may include, but are not limited to, whether the simulation process will cause more than a preset number of chain task switching, and / or whether the target loss probability is higher than the loss probability threshold during target detection, and / or whether the field of view overlap rate between the selected gimbal for the simulation task and other gimbals is less than or equal to the field of view overlap rate threshold, and / or whether the field of view blind zone of the selected gimbal for the simulation task is less than or equal to the field of view blind zone threshold, etc. The specific requirements can be set as needed according to the actual application scenario, and will not be elaborated here.

[0138] Based on the above embodiments, this embodiment should also explain that before receiving the target detection task, each PTZ camera may actually be in a task execution state (e.g., performing a periodic area scanning task and / or performing a target detection task for other target objects that was recently triggered).

[0139] Therefore, after receiving a newly triggered target detection task, allocating the newly triggered target detection task may occupy the pan-tilt camera that is currently executing a task (for example, the task priority of the newly triggered target detection task is higher than the task priority of the periodic task, or higher than the task priority of the recently triggered old target detection task).

[0140] To prevent high-priority tasks from frequently consuming PTZ camera resources and causing system instability, this embodiment can provide the following strategy: (1) Dynamic weighting of priority: In the calculation of dynamic priority P_dynamic_i, user permission factor Auth_i and task time-decay factor TimeDecay_i can also be introduced.

[0141] Auth_i can be determined by the user's role, and TimeDecay_i can slowly decay as the task execution time increases, preventing a single task from occupying the PTZ camera's resources for an extended period. The task priority of the currently executing task and the task priority of newly triggered tasks can both refer to and incorporate the above example parameters during calculation.

[0142] Therefore, based on the dynamically weighted adjusted task priority, it is possible to analyze whether the task being executed in the PTZ camera supports being replaced by a newly triggered target detection task.

[0143] (2) Chain switching detection: Before the PTZ camera performing the task is occupied, the decision engine will perform an occupation simulation in the digital twin and determine whether to perform task replacement based on the occupation simulation results.

[0144] For example, if gimbal A is taken over from its currently executing task T1 to execute a newly triggered task T2 (e.g., gimbal A has the highest compatibility value for task T2), can task T1 find another compatible gimbal B to take over the task within a specified time? If the simulation results indicate that this task replacement will lead to the loss of task T1 or trigger more than N (N is a positive integer, such as 2) task reallocations (e.g., if task T1 is assigned to gimbal B currently executing task T3, it may lead to task T3 being assigned to gimbal C, thus triggering a chain reaction of task reallocations), then this task replacement is prohibited, and a second-best (e.g., second-highest compatibility value) gimbal is selected for task T2. This enhances the stability of system scheduling.

[0145] Based on the above embodiments, this embodiment describes the method prior to step S140.

[0146] It should be noted that after the target detection task is assigned to the target PTZ camera, the target PTZ camera may have some overlap in its field of view with other PTZ cameras when performing the target detection task, or there may be blind spots in the field of view where neither the target PTZ camera nor other PTZ cameras can detect the target object.

[0147] Therefore, in order to maximize the sum of the field of view of each PTZ camera, other PTZ cameras besides the target PTZ camera can be coordinated and adjusted before the target PTZ camera performs the target detection task (or when the server periodically performs the conflict resolution task).

[0148] Specifically, the method in this embodiment before controlling the target PTZ camera to perform the target detection task includes at least the following steps S710 to S730: Step S710: Construct a graph model based on each PTZ camera and the target object.

[0149] For example, each PTZ camera and the target object in the target detection task can be modeled as a node in a graph neural network, and the spatiotemporal relationship between the PTZ and the target object, and between the PTZs, can be modeled as an edge in the graph neural network, thus obtaining a graph model.

[0150] Step S720: Analyze the graph model to obtain the target detection conflict between the target PTZ camera and other PTZ cameras in each PTZ camera.

[0151] Then, the graph model can be analyzed using graph neural network techniques such as GNN (Graph Neural Network) to predict in real time the target detection conflict between the target PTZ camera and other PTZ cameras (which can at least include field-of-view overlap conflicts for the same target object).

[0152] Step S730: Adjust other PTZ cameras based on target detection conflicts.

[0153] Therefore, based on the prediction results of target detection conflicts, the poses (such as the pan-tilt-zoom angle) of other pan-tilt-zoom cameras and / or the pan-tilt-zoom assigned to the target detection task can be actively adjusted so that the acquisition fields of other pan-tilt-zoom cameras do not overlap with those of the target pan-tilt-zoom camera as much as possible.

[0154] Optionally, in the specific implementation of this embodiment, in addition to adjusting the pose of other PTZ cameras, the pose of the target PTZ camera can also be adjusted as needed, which is not limited here.

[0155] As another example, each PTZ camera in this application may have an active PTZ camera and / or a dormant PTZ camera. For example, in low-power scenarios, some PTZ cameras may periodically go into sleep mode, which will not be elaborated here.

[0156] Target detection conflicts may also include blind spot conflicts (such as when a target object is about to enter an area that cannot be covered by the field of view of the currently active PTZ cameras). In such cases, other dormant PTZ cameras can be activated in advance based on the prediction results of target detection conflicts to try to fill the blind spot and achieve forward-looking collaboration.

[0157] Based on the above embodiments, this embodiment should also explain that when each PTZ camera performs a periodic area scanning task (when the target detection task is not triggered), the field of view of each PTZ camera may overlap significantly. Similarly, the PTZ pose can be adjusted by constructing a graph model, which will not be elaborated here.

[0158] Furthermore, for region scanning tasks, the decision engine can divide the overall region of the region scanning task into multiple sub-regions and perform simulation optimization in the digital twin for each sub-region.

[0159] For example, the core optimization objectives are maximizing the total field of view coverage area and minimizing the total field of view overlap area. The mathematical expression of the objective function for simulation optimization can be: Max(Σ(A_i*W_r)-ΣO(A_i,A_j)).

[0160] Where A_i is the field of view (field of view range) of gimbal i. O(A_i,A_j) is the overlapping field of view area of ​​gimbal i and gimbal j. W_r is the importance weight of sub-region r, which can be preset and fixed, or dynamically adjusted according to real-time environmental data (for example, the W_r value of densely populated areas and area entrances and exits can be adjusted to be higher).

[0161] Constraints: Limited by the obtained physical rotation range of the gimbal (θ_min≤θ_i≤θ_max).

[0162] Output: A preferred cooperative scanning path sequence {θ1,θ2,...,θn} can be generated for each pan-tilt camera participating in the area scanning task.

[0163] Based on the above embodiments, this embodiment describes the method after step S140.

[0164] After the target PTZ camera performs the target detection task, if the target object is in motion, it may leave the field of view supported by the current target PTZ camera.

[0165] Therefore, in this embodiment, the trajectory prediction method can be used to determine which pan-tilt camera the target object might go to when it leaves the target pan-tilt camera, thus obtaining the takeover pan-tilt camera, and then assigning the target detection task of the target object to the takeover pan-tilt camera for execution.

[0166] Specifically, the method in this embodiment after controlling the target PTZ camera to perform the target detection task includes at least the following steps S810 to S840: Step S810: Perform trajectory prediction processing based on the motion information of the target object to obtain the predicted trajectory of the target object.

[0167] Step S820: Determine the PTZ camera to be taken over from each PTZ camera based on the predicted trajectory.

[0168] Step S830: Based on the predicted trajectory and motion information, determine the timing of the target object leaving the field of view of the target PTZ camera.

[0169] Step S840: Assign the target detection task to the takeover pan-tilt camera based on the departure time.

[0170] For example, when target object i is about to leave the field of view of the pan-tilt unit A currently responsible for detection, the system can determine which pan-tilt unit (such as pan-tilt unit B) it is about to enter based on the predicted trajectory of target object i in the digital twin, thereby determining the pan-tilt unit camera to take over from each pan-tilt camera.

[0171] There are various methods for trajectory prediction processing, and relevant technologies for trajectory prediction in this field can be referenced, which will not be elaborated here.

[0172] Furthermore, the decision engine simulates the task takeover process for target detection within the digital twin. Based on the predicted trajectory and motion information, it determines when the target object leaves the field of view of the target PTZ camera, and can also determine the preset pointing angle (acquisition angle) required to take over the PTZ camera. This ensures that the target object is not lost during the switching of the acquisition field of view, achieving a smooth transition.

[0173] Then, at the departure time (or a preset preparation time before the opening time), the decision engine can issue a pre-rotation command to the takeover PTZ camera to rotate.

[0174] When target object i appears in the field of view of the takeover PTZ camera, the target detection task is assigned to the takeover PTZ camera. At this point, the takeover PTZ camera is ready and can immediately take over the target detection task.

[0175] At the same time, the target PTZ camera can be notified to cancel the target detection task, and can switch to idle state, hibernation state or perform other tasks, which will not be elaborated here.

[0176] Based on the above embodiments, this embodiment should also explain that, during the target detection task performed by the PTZ camera, a predictive anomaly recovery mechanism can be set up, for example: (1) Loss risk warning: The system can assess the "loss risk" of each detected target object in real time. When the target object is about to enter the blind spot of the current PTZ or encounters severe obstruction, it can be marked as "high risk state" in the digital twin.

[0177] (2) Multiple hypothesis trajectory prediction: Based on the historical trajectory, motion model and environmental road information of the target object, several (e.g. 3-5) more likely subsequent hypothetical trajectories of the target object are generated in the digital twin.

[0178] (3) Pre-deployment of recovery network: Along each subsequent hypothetical trajectory, the gimbals near the subsequent hypothetical trajectory are notified in advance to enter "standby state" and pre-adjusted to the preferred observation angle to form a "recovery encirclement" for the target object. For details, please refer to the relevant description of the task takeover process in the previous example, or refer to other related technologies in this field, which will not be elaborated here.

[0179] (4) Intelligent re-detection: If the target object reappears in a target hypothetical trajectory in the subsequent hypothetical trajectory, the standby gimbal that is on the target hypothetical trajectory and has adjusted the observation angle can immediately take over the target detection task in a very short time and realize the rapid re-detection of the target object.

[0180] Based on the above embodiments, this embodiment can also provide a visualization interface based on target detection, which can be rendered based on the digital twin in the aforementioned example.

[0181] In this visualization interface, the visualization elements may include, but are not limited to: 1. Map and / or 3D model of the current scene, etc.

[0182] 2. The icon of the PTZ camera and its real-time dynamically changing field-of-view cone frame, etc.

[0183] 3. The target object's icon, its historical movement trajectory, and its predicted trajectory, etc.

[0184] 4. Collaborative scanning paths planned by the system, etc.

[0185] 5. Highlight the overlapping conflict areas and blind zone conflict areas predicted by GNN.

[0186] 6. Status indicators of the PTZ (such as color to distinguish idle, busy, and alarm), load and health status, etc.

[0187] 7. Dynamic priority indicators for target detection tasks (e.g., color intensity representing urgency).

[0188] 8. Key information prompts, etc.

[0189] As a result, system administrators can get a clear overview of the overall target detection status, system collaboration logic, resource health status, risk warnings, and other information on this interface, achieving a "one-map overview" and greatly improving user efficiency.

[0190] Based on the above embodiments, this embodiment also provides some specific application scenarios for illustrative purposes.

[0191] For example, taking a specific airport runway perimeter detection scenario as an example, the main operation process of this application will be explained in detail.

[0192] Among them, gimbals A, B, C, D, E, F, and G are deployed around the airport runway. The system can receive tasks for "global scan of the runway perimeter" and "focused detection of vehicles T1 and individuals loitering abnormally T2".

[0193] The system adopts an edge-cloud collaborative architecture, where PTZs A and B are managed by regional gateway 1, and PTZs C, D, E, F, and G are managed by regional gateway 2. It is known that PTZ A has a low health index (0.6) due to long-term high-load operation, while the health status of PTZs B, C, and D is good (all greater than 0.9).

[0194] Step 1: Initialization.

[0195] All PTZ cameras and regional collaborative gateways in the current scenario can upload their own status, health index, and detected information in real time, and the central server can update the corresponding digital twins accordingly.

[0196] It can initialize the priority of various tasks that need to be detected, such as: vehicle T1 (high basic priority) and personnel T2 (medium basic priority).

[0197] Step 2: Task decision-making and allocation.

[0198] 1. Dynamic Priority Calculation: The decision engine analysis revealed that in the current scenario, vehicle T1 (driving normally) and person T2 triggered the target detection task. Person T2 exhibited abnormal loitering behavior, leading to an increased abnormality factor. Furthermore, their movement towards a detection blind spot significantly increased their spatiotemporal urgency factor. Consequently, person T2's task priority surpassed that of vehicle T1.

[0199] Task allocation: The decision engine initiates gimbal adaptation selection.

[0200] 2. For T2 (high dynamic priority): Gimbal C is slightly farther away, but T2 is facing C. Calculations show that gimbal C has the highest geometric relationship factor with target T2 (T2 is moving directly towards C), with no obstructions and excellent imaging conditions. Its observation quality factor Q_observation is significantly better than other candidate gimbals, making it the optimal perspective for acquiring identification information. Q_observation (frontal) is 1.0, with low load and a high health index. Calculations show that gimbal C has the highest fit value.

[0201] Therefore, the decision engine can assign the target detection task for T2 to the PTZ C through the regional collaborative gateway 2 corresponding to PTZ C.

[0202] For T1: Gimbal B is the closest, has good Q_observation, low load, and high health index, resulting in the highest adaptation value. Although gimbal A is also close, its health index is low, causing its adaptation value to be lower than that of gimbal B.

[0203] Therefore, the decision engine assigns the target detection task for T1 to PTZ B through the regional collaborative gateway 1 corresponding to PTZ B.

[0204] 3. Scanning path planning: When there are no other target detection tasks, the decision engine can simultaneously plan collaborative scanning paths for gimbal A and gimbal D.

[0205] By optimizing the objective function Max(Σ(A_i*W_r)-ΣO(A_i,A_j)), higher weights W_r are assigned to key areas, and the optimal scanning path sequence (pan-tilt rotation sequence) is calculated to ensure that the area surrounding the key area is fully covered as much as possible, and that the key area has high clarity.

[0206] 4. Conflict prediction: GNN predicts that when gimbal C is detecting T2, its field of view may briefly overlap with that of gimbal A, which is performing the scanning task.

[0207] The decision engine can pre-control gimbal A to fine-tune the scanning path at specific times to avoid this conflict.

[0208] Step 3: Task takeover and abnormal recovery execution.

[0209] 1. Task Takeover: Vehicle T1 is about to leave the field of view of PTZ B. Based on its predicted trajectory, the decision engine determines that PTZ D is the takeover PTZ camera for vehicle T1. After simulation execution in the digital twin, the decision engine can issue a pre-rotation command to PTZ D in advance through the regional collaborative gateway 2 corresponding to PTZ D. T1 smoothly enters the field of view of PTZ D and is taken over.

[0210] 2. Predictive anomaly recovery and avoidance of cascading switches: (1) Personnel T2 suddenly accelerates and runs towards a blind spot of a gimbal D. The system immediately marks T2 as "high risk" in the digital twin and generates three possible hypothetical trajectories (e.g., leading to the east, south, and west exits respectively) based on its last direction of movement and environmental road information.

[0211] (2) The system notifies the backup PTZs (such as PTZs E, F, and G) located near the three exits in advance to enter "standby mode" and preset them to the best observation angle through the regional collaborative gateway.

[0212] (3) At this time, if a higher priority target detection task (such as detecting a drone) appears, it will need to occupy gimbal E. Then the decision engine will start the chain switching avoidance strategy: simulation in the digital twin shows that if gimbal E responsible for the west exit is occupied, when T2 appears from the west exit, there will be no suitable gimbal to take over immediately, which will cause T2 to be lost.

[0213] Therefore, the decision engine can prohibit the current use of gimbal E and select another available gimbal for the task of detecting the drone.

[0214] (4) Two seconds later, T2 appears from the west exit. The gimbal E, which is in standby mode, can immediately detect the target T2 and successfully achieve re-detection. Therefore, the decision engine can officially assign the target detection task for T2 to the gimbal E.

[0215] Step 4: Global visualization.

[0216] On the visualization interface, the administrator sees the following: the dynamic scanning cones and planned paths of PTZs A and D are clearly displayed on the map; the field-of-view cones of PTZs B, C, and E are locked onto the icons of vehicle T1 and personnel T2, respectively. Personnel T2's icon is dark red, indicating its high dynamic priority. PTZ A's icon is yellow, indicating its health status requires monitoring. Before T2 enters the blind spot, its predicted trajectory line and risk warning box are highlighted. The icons of standby PTZs E, F, and G are displayed as standby.

[0217] It should be further noted that the execution entity of the digital twin-based multi-panel camera control method can be a digital twin-based multi-panel camera control device. For example, the digital twin-based multi-panel camera control method can be executed by a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, this digital twin-based multi-panel camera control method can be implemented by a processor calling computer-readable instructions stored in memory.

[0218] Figure 2 This is a block diagram illustrating a multi-panel camera control device based on digital twins, as shown in an exemplary embodiment of this application. The device can be applied to a server, which has a communication connection with multiple pan-tilt cameras in the current scene. Figure 2As shown, the exemplary multi-panel camera control device 200 based on digital twins includes: a twin construction module 210, an adaptation value determination module 220, a pan-tilt determination module 230, and a pan-tilt control module 240. Specifically: The twin construction module 210 is used to construct a digital twin based on the scene information of the current scene and the device information of each PTZ camera in the current scene.

[0219] The adaptation value determination module 220 is used to respond to the received target detection task and determine the adaptation value of each PTZ camera to the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera.

[0220] The PTZ determination module 230 is used to determine the target PTZ camera from among the various PTZ cameras based on the digital twin and the adaptation value.

[0221] The PTZ control module 240 is used to control the target PTZ camera to perform target detection tasks.

[0222] In this exemplary multi-panel camera control device based on digital twins, an equivalent virtual information model is constructed by acquiring scene information of the current scene and device information of each pan-tilt camera in the current scene, resulting in a digital twin corresponding to the current scene. Upon receiving a target detection task for the current scene, the task priority, execution cost of each pan-tilt camera performing the target detection task, and overall load of each pan-tilt camera can be obtained, thereby determining the adaptation value of each pan-tilt camera to the target detection task. By analyzing the simulation execution results of each pan-tilt camera performing the target detection task through the digital twin, and combining the adaptation values ​​of each pan-tilt camera to the target detection task, a suitable target pan-tilt camera for performing the target detection task is determined. Therefore, target detection tasks can be assigned to target pan-tilt cameras in real space, and the target pan-tilt cameras can be controlled to perform the target detection tasks. By constructing a global digital twin and an intelligent decision engine, the problem of seamless collaborative control and resource optimization at the system level in multi-pan-tilt camera scenarios can be solved, improving the control efficiency of multi-pan-tilt cameras.

[0223] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0224] The functions of each module can be found in the embodiment of the multi-panel camera control method based on digital twin, and will not be repeated here.

[0225] Please see Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above embodiments of the multi-panel camera control method based on digital twins. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0226] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above embodiments of the multi-panel camera control method based on digital twins. Processor 102 can also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 102 can be implemented using integrated circuit chips.

[0227] In this exemplary electronic device, by acquiring scene information of the current scene and device information of each PTZ camera in the current scene, an equivalent virtual information model is constructed to obtain a digital twin corresponding to the current scene. Upon receiving a target detection task for the current scene, the task priority of the target detection task, the execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera can be obtained, thereby determining the adaptation value of each PTZ camera to the target detection task. By analyzing the simulation execution results of each PTZ camera on the target detection task through the digital twin, and combining the adaptation values ​​of each PTZ camera on the target detection task, a suitable target PTZ camera for performing the target detection task is determined from among the PTZ cameras. Therefore, the target detection task can be assigned to the target PTZ camera in real space, and the target PTZ camera can be controlled to perform the target detection task. By constructing a global digital twin and an intelligent decision engine, the problem of seamless collaborative control and resource optimization at the system level in multi-PTZ camera scenarios can be solved, improving the control efficiency of multi-PTZ cameras.

[0228] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above embodiments of the multi-panel camera control method based on digital twin.

[0229] In this exemplary storage medium, by running program instructions within the storage medium, scene information of the current scene and device information of each PTZ camera in the current scene are obtained, constructing an equivalent virtual information model to obtain a digital twin corresponding to the current scene. Upon receiving a target detection task for the current scene, the task priority of the target detection task, the task execution cost of each PTZ camera executing the target detection task, and the comprehensive load of each PTZ camera can be obtained, thereby determining the adaptation value of each PTZ camera to the target detection task. By analyzing the simulation execution results of each PTZ camera for the target detection task through the digital twin, and combining the adaptation value of each PTZ camera for the target detection task, a suitable target PTZ camera for executing the target detection task is determined from among the PTZ cameras. Thus, the target detection task can be assigned to the target PTZ camera in the real space, and the target PTZ camera can be controlled to execute the target detection task. By constructing a global digital twin and an intelligent decision engine, the problem of seamless collaborative control and resource optimization at the system level in multi-PTZ camera scenarios can be solved, improving the control efficiency of multi-PTZ cameras.

[0230] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0231] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0233] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A multi-panel camera control method based on digital twin, characterized in that, The method is applied to a server, which has a communication connection with multiple PTZ cameras in the current scene, and the method includes: A digital twin is constructed based on the scene information of the current scene and the device information of each PTZ camera in the current scene; In response to the received target detection task, the adaptation value of each PTZ camera to the target detection task is determined based on the task priority of the target detection task, the task execution cost of each PTZ camera executing the target detection task, and the overall load of each PTZ camera. The target PTZ camera is determined from each PTZ camera based on the digital twin and the adaptation value; Control the target PTZ camera to perform the target detection task.

2. The method according to claim 1, characterized in that, Before determining the adaptation value of each PTZ camera for the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera, the method further includes: Obtain the basic priority of the target detection task; Analyze the factors influencing the target object on the target detection task. The basic priority is adjusted based on the influencing factor to obtain the task priority.

3. The method according to claim 2, characterized in that, The influencing factors include at least one of the following: behavioral anomaly factor, spatiotemporal urgency factor, and observation quality factor. The analysis of the influencing factors of the target object on the target detection task includes: The target object is subjected to behavioral analysis processing to obtain the behavioral anomaly factor; Motion analysis is performed on the target object to obtain the spatiotemporal urgency factor between the target object and the target detection area in the current scene; The observation quality factor is determined based on the observation value of the target object by the pan-tilt camera.

4. The method according to claim 1, characterized in that, Before determining the adaptation value of each PTZ camera for the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera performing the target detection task, and the overall load of each PTZ camera, the method further includes: Obtain the health status information of the PTZ camera, the health status information including at least one of continuous running time, historical operating load, historical fault records, and component validity period; The overall load of the PTZ camera is determined based on its current operating load and health status information.

5. The method according to claim 1, characterized in that, The step of determining the adaptation value of each PTZ camera for the target detection task based on the task priority of the target detection task, the task execution cost of each PTZ camera executing the target detection task, and the overall load of each PTZ camera includes: The task execution cost and the overall load are weighted and summed to obtain the equipment screening benchmark value; The adaptation value is determined based on the ratio between the task priority and the device screening benchmark value.

6. The method according to claim 1, characterized in that, The step of determining the target PTZ camera from each PTZ camera based on the digital twin and the adaptation value includes: The first PTZ camera with the highest adaptation value is selected to simulate the target detection task in the digital twin, and a first simulation result is obtained; In response to the first simulation result not meeting the preset task requirements, a second PTZ camera with an adaptation value smaller than that of the first PTZ camera is selected to simulate the target detection task in the digital twin, and a second simulation result is obtained. In response to the second simulation result satisfying the preset task requirements, the second PTZ camera is identified as the target PTZ camera.

7. The method according to claim 1, characterized in that, Before controlling the target PTZ camera to perform the target detection task, the method further includes: A graph model is constructed based on each pan-tilt camera and the target object; Analyzing the graph model, the target detection conflict between the target PTZ camera and other PTZ cameras is obtained. Adjust the other PTZ cameras based on the target detection conflict.

8. The method according to claim 1, characterized in that, After controlling the target PTZ camera to perform the target detection task, the method further includes: Based on the motion information of the target object, trajectory prediction processing is performed to obtain the predicted trajectory of the target object; The pan-tilt camera to be taken over is determined from among the pan-tilt cameras based on the predicted trajectory. Based on the predicted trajectory and the motion information, determine the timing when the target object leaves the field of view of the target PTZ camera; The target detection task is assigned to the takeover PTZ camera based on the departure timing.

9. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images