Pan-tilt failure inspection and position covering method based on neighbor machine cooperation and related equipment

Abstract

The invention provides a cradle head failure inspection and position covering method based on neighbor machine cooperation and related equipment, and relates to the technical field of general control or adjustment systems. The method comprises the following steps: firstly, determining a digital twinborn model of each holder in a target area so as to judge whether a target holder fails or not; the method comprises the following steps: determining a gap region corresponding to a failure holder preset position by combining a digital twin model and a failure type, dividing the gap region into a plurality of gap sub-regions with gap risk value identifiers, matching candidate neighbor units for each gap sub-region, and determining an optimal takeover holder; and generating a routing inspection execution task sequence containing position complementing newly-added tasks in combination with the digital twin model, controlling the optimal takeover holder to complement positions according to the sequence, triggering a takeover exit process after the failed holder returns to normal, and recovering the original routing inspection plan of the optimal takeover holder. By implementing the method, the optimal resource can be dynamically and intelligently scheduled to make up the monitoring blind area, and the robustness and reliability of the whole monitoring system are improved while the interference to the normal inspection task is minimized.

Description

Technical Field

[0001] This application relates to the field of general control or regulation system technology, and in particular to a gimbal failure inspection and replacement method and related equipment based on neighboring machine cooperation. Background Technology

[0002] In mountainous forest fire monitoring systems, pan-tilt-zoom (PTZ) monitoring equipment serves as a critical infrastructure for fire early warning, requiring continuous 24 / 7 monitoring of designated areas. However, due to harsh mountain environments, aging equipment, and extreme weather, PTZ equipment inevitably experiences malfunctions. When a PTZ device completely fails or partially loses its viewing angle, its original monitoring area becomes a blind spot, increasing the risk of missed fire detection.

[0003] Traditional solutions mainly rely on manual adjustments to the monitoring range of adjacent PTZs to temporarily cover blind spots after a fault is detected, or dispatching maintenance personnel to handle the situation on-site. Some servers have basic automatic takeover capabilities, meaning that when a PTZ goes offline, a pre-set neighboring server automatically switches to the core area of ​​the faulty PTZ.

[0004] However, this approach suffers from problems such as long response times, low scheduling efficiency, and significant resource waste. Especially during peak fire seasons, the existence of any monitoring blind spot can delay fire detection, leading to irreparable losses. Furthermore, the simplistic and crude method of having adjacent units take over often disrupts the original, equally important inspection tasks of those adjacent units, potentially resulting in a "robbing Peter to pay Paul" situation and reducing overall monitoring effectiveness. Therefore, current technology struggles to quickly and accurately schedule adjacent PTZ units to fill monitoring gaps while ensuring there are no blind spots, thus minimizing interference with existing inspection tasks when PTZ monitoring equipment malfunctions. Summary of the Invention

[0005] This application provides a method and related equipment for gimbal failure inspection and replacement based on neighboring machine cooperation, which can be used to achieve rapid, accurate, and low-interference automated filling of monitoring blind spots.

[0006] Firstly, this application provides a method for gimbal failure inspection and replacement based on neighboring machine collaboration, applied to a server. The method includes: determining a digital twin model of each gimbal device within a target area, and determining whether the corresponding target gimbal device is faulty based on the digital twin model; if the target gimbal device is determined to be faulty, determining the type of failure; determining the gap area corresponding to the preset position of the faulty gimbal according to the digital twin model and the type of failure; dividing the gap area into multiple gap sub-areas with gap risk value identifiers; determining candidate neighboring machines matching each gap sub-area, and determining the optimal takeover gimbal corresponding to each gap sub-area from the candidate neighboring machines; determining an inspection execution task sequence containing newly added tasks corresponding to the gap sub-area to be replaced, based on the digital twin model of the optimal takeover gimbal; controlling the optimal takeover gimbal to perform replacement operations according to the inspection execution task sequence; and automatically triggering a takeover exit process and restoring the original inspection plan of the optimal takeover gimbal after determining that the faulty gimbal has recovered.

[0007] By adopting the above technical solution and constructing a digital twin model, accurate mapping and real-time monitoring of the physical state and operating environment of the PTZ device are achieved, providing a high-fidelity information foundation for subsequent decision-making. Based on this, when a device failure is detected, the server does not simply perform a backup switch. Instead, it implements a complete closed-loop automated replacement process through a series of refined steps, including failure type judgment, gap area quantification, risk level classification, dynamic selection of the optimal neighboring device, and intelligent reorganization of inspection tasks. This method can dynamically and intelligently allocate optimal resources to fill monitoring blind spots based on the specific circumstances of the failure and the actual risk of the monitored area, and automatically and smoothly exit after the device recovers. This minimizes interference with normal inspection tasks while improving the robustness, reliability, and intelligent emergency response capabilities of the entire monitoring system.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the step of determining the gap area corresponding to the preset position of the failed PTZ based on the digital twin model and the failure type specifically includes: if the device health of the dynamic data presentation device in the digital twin model is zero or continuous heartbeats are lost, then the failure type is determined to be a full failure, and the area corresponding to all preset positions in the static data of the failed PTZ is the gap area; if the target preset position response times out, then the failure type is determined to be a partial viewpoint failure, and the corresponding area is determined to be the gap area based on the target preset position; if the returned image has local anomalies, then the gap area is obtained by combining the effective screen boundary of the abnormal image with the preset coverage range of the target preset position.

[0009] By employing the aforementioned technical solution, this method's refined differentiation of failure types makes the identification of gap areas more targeted and accurate. By distinguishing between total failure, partial viewpoint failure, and localized image anomalies, the server can precisely define the actual extent of the monitoring blind spot based on the specific manifestation of the failure, rather than broadly treating the entire area of ​​the PTZ as a gap. This refined definition method avoids excessive consumption of supplementary resources, making subsequent supplementary tasks more focused and efficient. It significantly reduces interference to the original tasks of adjacent machines caused by unnecessary large-scale supplementary work, improving the resource utilization and response accuracy of the entire supplementary strategy.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of dividing the gap area into multiple gap sub-regions with gap risk value identifiers specifically includes: obtaining the failure timestamp of the failed PTZ; obtaining a complete panoramic inspection video segment collected by the failed PTZ before the failure timestamp; performing frame-by-frame object detection on the complete panoramic inspection video based on a preset flammable material feature layout library, extracting the flammability of each object, and determining the flammability score of all objects based on the flammability; generating a risk heat map of the gap area by combining the spatial coordinates of each object in the panoramic video and the flammability score using a spatial interpolation algorithm, wherein the risk heat map includes gap risk value identifiers derived from the flammability score; and dividing the gap area into multiple gap sub-regions according to the distribution of gap risk value identifiers in the risk heat map.

[0011] By employing the aforementioned technical solution, and utilizing pre-failure inspection videos combined with a pre-set flammable material characteristic database for object detection and risk quantification, the server can generate an intuitive risk heatmap. This process not only identifies the gap area but, more importantly, assesses the potential hazard level of different locations within that area. The gap area is then divided into sub-regions with different risk indicators, ensuring that subsequent remediation strategies are no longer indiscriminate coverage but rather risk-driven differentiated responses. Priority is given to maintaining monitoring continuity in high-risk sub-regions, allowing limited takeover resources to be deployed where most needed, thus enhancing the security value and effectiveness of emergency remediation.

[0012] In some embodiments of the first aspect, the step of determining the inspection execution task sequence containing the new task corresponding to the gap sub-region to be filled, based on the digital twin model of the optimal takeover PTZ, specifically includes: determining the filling angle covering the gap sub-region based on the digital twin model of the optimal takeover PTZ, and generating a new task including the pre-set position of the takeover task; obtaining the original inspection plan of the optimal takeover PTZ, and determining the original risk value, priority, and time percentage corresponding to each sub-plan in the original inspection plan; associating the time percentage with the original risk value to obtain a risk-time mapping relationship; comparing the gap risk value identifier corresponding to the gap sub-region with each of the original risk values ​​to determine the new priority of the new task to be inserted into the original inspection plan; determining the new time percentage of the new task in the total time corresponding to the original inspection plan based on the risk-time mapping relationship and the gap risk value identifier; and determining the inspection execution task sequence based on the new priority and the new time percentage.

[0013] By adopting the above technical solution, instead of abruptly interrupting or shelving the original inspection plan, a "risk-time" mapping relationship is established to quantitatively compare the risk level of newly added replacement tasks with the original tasks, thereby finding a reasonable priority and time allocation for them in the new sequence. This approach ensures that the insertion of new tasks is justified, meeting the replacement needs while maximizing the preservation of the inspection frequency of the original high-risk areas. The final generated inspection execution task sequence is an optimized solution that comprehensively balances the original responsibilities and temporary emergency tasks, ensuring that the optimal takeover PTZ minimizes its own security value loss while performing replacement operations, thus guaranteeing the overall stability and efficiency of the entire monitoring system.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, the step of determining candidate neighboring machines that match each of the gap sub-regions, and determining the optimal takeover gimbal for each gap sub-region from the candidate neighboring machines, specifically includes: excluding neighboring machines within the target area that cannot cover the gap sub-region or whose overlap rate with the gap sub-region is lower than a preset overlap rate threshold from the candidate neighboring machines, to obtain the remaining candidate neighboring machines; obtaining the coverage capability matching degree based on the ratio of the theoretical maximum visible coverage area of ​​the remaining candidate neighboring machines to the total area of ​​the gap sub-region, and in conjunction with a line-of-sight occlusion model; and, after the virtual insertion of a new task, determining the optimal takeover gimbal for each gap sub-region based on the original high-resolution gimbal of the remaining candidate neighboring machines. The task interference degree is determined by the reciprocal of the average inspection interval extension rate of the priority task; the energy consumption cost is determined by the motor rotation and equipment power consumption generated by the virtual movement of the remaining candidate neighbor from its current position to the pre-set position and the execution of one cycle of the new task, and the pre-set position is determined by the gap sub-region; the response speed is determined by the estimated time required for the remaining candidate neighbor to complete the current task and rotate to the first pre-set position; the comprehensive score of each remaining candidate neighbor is obtained by using a preset weighted summation model based on one or more of the coverage matching degree, task interference degree, energy consumption cost and response speed; the neighbor with the highest comprehensive score is selected as the optimal takeover PTZ.

[0015] By adopting the above technical solution, four core dimensions were comprehensively considered: coverage capability, task interference, energy consumption cost, and response speed. Through a comprehensive evaluation of each candidate neighboring device and the use of a weighted summation model to derive a comprehensive score, the device with the highest "cost-effectiveness" for performing the replacement task under the current circumstances can be objectively selected. This comprehensive decision-making mechanism ensures that the selected optimal takeover PTZ not only "sees" the gap but also completes the task with the lowest overall system cost, the fastest speed, and the least interference, thereby optimizing the efficiency and sustainability of the entire replacement operation.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, before the step of determining the digital twin model of each PTZ device within the target area, the method further includes: periodically, after the inspection task is completed, controlling all normal PTZ devices within the target area to perform panoramic scanning to acquire a multi-angle environmental image sequence; processing the environmental image sequence to generate a three-dimensional mesh model of the target area; comparing the three-dimensional mesh model with the existing static data in the corresponding digital twin model to identify and quantify environmental changes; and automatically updating the static data in the digital twin model according to the environmental changes, the update including correcting the line-of-sight occlusion relationship between each PTZ device and each monitoring area.

[0017] By adopting the above technical solution, and periodically scanning and reconstructing the 3D environment model, then comparing and updating it with data in the digital twin model, it is ensured that the decision-making information upon which the system relies remains synchronized with changes in the physical world. In particular, the automatic correction of line-of-sight occlusion relationships directly affects the accuracy of the assessment of neighboring equipment's coverage capabilities, and is crucial for the subsequent selection of the optimal takeover pan / tilt unit.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, when it is determined that the failed gimbal has recovered to normal, the step of automatically triggering the takeover exit process includes: when the signal of the failed gimbal recovering to normal is detected for the first time, the corresponding state is marked as an observation period, and the optimal takeover gimbal is controlled to continue to perform the replacement operation within a preset duration; during the observation period, the dynamic data of the recovered gimbal is continuously monitored, and if the corresponding continuous and stable operation time reaches the preset duration, the failed gimbal is officially confirmed to have recovered to normal, and the takeover exit process is triggered.

[0019] By adopting the above technical solution, the problem of "flashover" monitoring caused by unstable equipment status is effectively avoided. By setting an "observation period," the server does not immediately terminate the replacement operation after initially detecting a recovery signal from a failed PTZ unit, but instead continuously monitors its stability. This "confirmation-based release" strategy ensures that the takeover exit process is only triggered when the failed PTZ unit is confirmed to be operating stably and normally. This improves the system's fault tolerance, guarantees the continuity and stability of monitoring coverage, and prevents repeated loss of monitoring areas due to frequent start-ups and shutdowns of equipment in the early stages of fault recovery.

[0020] In a second aspect, this application provides a server comprising: one or more processors and a memory; the memory being coupled to the one or more processors, the memory being used to store computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the server to perform the methods described in the first aspect and any possible implementation thereof.

[0021] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on a server, cause the server to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer program product that, when run on a server, causes the server to perform the method described in the first aspect and any possible implementation thereof.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By adopting a closed-loop automated replacement decision-making technology that integrates the construction of a digital twin model for state perception and the integration of failure diagnosis, risk assessment, optimal neighbor selection, and intelligent task reorganization, the technology effectively solves the technical problems of slow response, low efficiency, and poor replacement effect caused by the reliance on manual intervention or the use of fixed replacement strategies after PTZ failure. This achieves unmanned, intelligent, optimized emergency response and automatic recovery of monitoring system failures, thereby improving the overall robustness and operational efficiency of the system.

[0024] 2. By employing a technique that identifies target risks based on historical video and generates risk heatmaps using spatial interpolation algorithms to quantify and delineate gap areas, this technology effectively solves the problem that existing technologies, after identifying monitoring gaps, typically treat them as homogeneous areas for indiscriminate filling, resulting in low utilization efficiency of filling resources and low security value. This enables a refined risk level assessment of monitoring gaps, transforming the filling strategy from "indiscriminate coverage" to "risk-driven differentiated response," prioritizing limited takeover resources for the most critical areas, thereby significantly improving the security accuracy and effectiveness of emergency filling.

[0025] 3. By adopting an intelligent task sequence reorganization technology that establishes a "risk-time" mapping relationship and quantitatively compares the risks of new tasks with those of existing tasks to dynamically determine the priority and time proportion of new tasks, this technology effectively solves the technical problem of existing technologies that, when scheduling neighboring machines for backup, use a crude task insertion method, lack quantitative balancing of new and old tasks, and are prone to "paying attention to one thing but losing attention to another." This achieves dynamic, quantitative fusion and intelligent scheduling of new backup tasks and existing inspection tasks, filling monitoring gaps while minimizing the impact on the normal operation of neighboring machines, and ensuring the overall stability of the entire monitoring system and the comprehensive execution efficiency of inspection tasks. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a gimbal failure inspection and replacement method based on neighboring machine cooperation in an embodiment of this application. Figure 2 This is another flowchart illustrating the gimbal failure inspection and replacement method based on neighboring machine cooperation in this application embodiment; Figure 3 This is a schematic diagram of the physical device structure of a server in an embodiment of this application. Detailed Implementation

[0027] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used herein, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the listed items.

[0028] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0029] For ease of understanding, the method provided in this implementation is described in process below. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a gimbal failure inspection and replacement method based on neighboring machine cooperation in an embodiment of this application.

[0030] S101. Determine the digital twin model of each gimbal device within the target area, and determine whether the corresponding target gimbal device is faulty based on the digital twin model. The target area refers to a specific geographical range or spatial area that needs to be inspected and monitored, such as an industrial park, residential area, or large factory area. It is the physical spatial boundary for the deployment of PTZ equipment and the execution of inspection tasks. PTZ equipment refers to monitoring equipment equipped with rotatable and tiltable lenses, which can achieve monitoring coverage in different directions and ranges by adjusting the angle. It is widely used in security, production monitoring, and other scenarios. The digital twin model refers to a virtual model constructed through digital technology that completely corresponds to and is mapped in real time to the physical PTZ equipment. It includes the equipment's static attributes (such as location, model, hardware parameters, preset position information, etc.) and dynamic data (such as operating status, health, real-time acquired image data, heartbeat signals, etc.) to simulate, monitor, and analyze the behavior and status of the physical equipment. The target PTZ equipment refers to one or more PTZ equipment that are the focus of the current detection and analysis process. Failure refers to the PTZ equipment's inability to perform its preset inspection or monitoring functions normally, which may manifest as the equipment being completely unable to work, some functions being abnormal, or the collected data not meeting requirements.

[0031] In this application, PTZ devices in a region can transmit data to a shared server, which acts as the brain of the entire PTZ system, responsible for computation, decision-making, and issuing commands. After the server starts, it can periodically execute commands according to a preset cycle (e.g., once per second) to monitor the real-time operating status of all PTZ devices within the target area, providing a basis for subsequent failure handling and replacement operations. First, a digital twin model of each PTZ device within the target area is determined. This requires the server to pre-build a digital twin for each PTZ device deployed in the target area. During the construction process, static information of the PTZ devices is collected, including the device's installation location coordinates (e.g., latitude and longitude, relative altitude), hardware model, lens parameters (focal length, field of view), the number of supported preset positions, and the angle parameters corresponding to each preset position. This information is then stored as static data of the digital twin model. Meanwhile, the server will acquire dynamic operating data of the PTZ device in real time through sensors, network communication, etc., such as the device's CPU utilization, memory usage, motor operating status, lens operating temperature, real-time returned monitoring image frames, heartbeat signals between the device and the server (such as periodically sent status confirmation packets), and device health score (calculated based on factors such as hardware status and operational stability). This dynamic data will be updated in the digital twin model in real time to ensure that the virtual model and the physical device are consistent.

[0032] After the digital twin model is built and updated in real time, the server uses the model to determine whether the target gimbal device has failed. The judgment process is mainly achieved by analyzing dynamic data in the digital twin model: First, monitoring the device health. If the device health value recorded in the digital twin model drops to zero, it indicates that the device hardware may have a serious failure and cannot continue to operate, thus being judged as a failure. Second, checking the heartbeat signal. If the server does not receive the heartbeat signal sent by the target PTZ device for a preset time window (e.g., 10 seconds) and multiple communication attempts fail to elicit a response, it is considered that the connection between the device and the server is interrupted, possibly due to power outage, network failure, or device crash, thus being judged as a failure. Third, analyzing image data. If the target PTZ device continuously returns blank images, blurry images, or abnormal images that differ significantly from the preset scene, and cannot recover to normal after ruling out temporary factors such as lens obstruction, it is judged as a failure. Fourth, detecting the functional response. When the server sends control commands to the target PTZ device (e.g., adjusting to a preset position, zooming, etc.), if no execution feedback is received from the device within a specified time (e.g., 5 seconds), or if the feedback execution result does not match the command requirements, and the problem persists after multiple attempts, it is judged as a device functional failure. Through comprehensive monitoring and analysis of the above indicators, the server can accurately determine whether the target PTZ device is in a malfunctioning state, providing accurate triggering conditions for subsequent processing.

[0033] In some embodiments, prior to this step, after all normal PTZ devices in the target area have completed each periodic routine inspection task, the static data of the digital twin model can be updated periodically to ensure a high degree of consistency between the virtual model and the physical environment.

[0034] The server sends scanning commands to all working PTZ cameras in the target area at a preset cycle (e.g., 11 PM daily). These commands contain preset shooting parameters. The PTZ cameras execute these parameters, pausing shooting at each set angle for horizontal rotation and adjusting and shooting vertically at intervals at each horizontal angle, generating a multi-angle image sequence of "horizontal + vertical". All PTZ images are then aggregated into a comprehensive image dataset for the area. The server imports the image sequence into a 3D reconstruction server: first, preprocessing; then, image registration, determining the relative pose of the images through feature point matching; next, generating a dense point cloud and estimating the spatial point depth; finally, using the Poisson reconstruction algorithm to transform the point cloud into a textured 3D mesh model, clearly presenting the spatial state of objects within the area. The server extracts historical static data from the digital twin model, aligns the new 3D mesh model with the old model using the ICP algorithm, calculates the deviation of object surface points, and determines a change if the deviation exceeds 0.5 meters. Simultaneously, a ray tracing algorithm is used to simulate the line of sight from the PTZ camera to the monitoring area, calculating the occlusion rate (e.g., from 20% previously to 80% now), quantifying the change in occlusion relationships. The server writes the quantified changes to the static database: when a new object is added, its coordinates, size, and type are added; when an object is moved / changed in size, the corresponding fields are updated; when an object is removed, the record is deleted. The focus is on correcting the line-of-sight occlusion table, updating the occlusion status, obstructing objects, and occlusion rate of the "PTZ-Monitoring Area" according to the new occlusion rate, and recalculating and updating the effective monitoring range of the PTZ.

[0035] By periodically scanning the panorama and reconstructing the 3D, we can capture environmental changes such as equipment relocation, tree growth, and new facilities in the target area in real time. After quantifying the changes, we update the static data of the digital twin model, avoiding the disconnect between the virtual model and the physical world. This provides reliable data support for "accurately judging the coverage capability of neighboring machines" and "correcting the impact of line-of-sight obstruction" when the PTZ fails and needs to be replaced. It solves the problem of replacement decision bias caused by environmental changes in traditional models.

[0036] S102. If it is determined that the target gimbal device is faulty, then determine the type of fault. The purpose of this step is to identify the specific type of equipment failure, as different failure types lead to different gaps in monitoring coverage, requiring different compensation strategies. In practical applications, PTZ device failures are not singular but involve multiple scenarios; therefore, detailed analysis is necessary to determine the specific type of failure.

[0037] First, regarding the determination of total failure, when the dynamic data in the digital twin model shows a device health level of zero, it indicates that the core hardware of the device (such as the motherboard, motor, lens, etc.) may have been completely damaged, causing the device to be completely inoperable; or when the device continuously loses heartbeat signals for more than a preset threshold (such as 30 seconds), and the connection cannot be restored after multiple communication retries and troubleshooting (such as checking network lines, power supply, etc.), it is also determined to be a total failure. Total failure means that the monitoring areas corresponding to all preset positions of the PTZ device will lose coverage, requiring comprehensive backup support.

[0038] Secondly, the determination of partial viewpoint failure mainly targets functional abnormalities of specific preset positions. When the server sends a command to the target PTZ device to switch to a specific preset position, if no feedback is received within the specified response time (e.g., 8 seconds) indicating that the device has successfully reached the preset position, or if the image returned by the device shows that it has not reached the specified viewpoint, then the viewpoint corresponding to that preset position is determined to be faulty. This type of failure is usually caused by reasons such as motor failure, incorrect preset position parameters, or mechanical jamming, and only affects the monitoring coverage of the specific preset position; other preset positions may still function normally.

[0039] Finally, for cases where the returned image contains local anomalies, image analysis techniques are needed for assessment. Local anomalies may include blurred areas, occlusion, color distortion, or the presence of fixed noise. The server will compare the abnormal image with the preset normal coverage area and image features for that location. Using algorithms such as edge detection and feature matching, it will determine the boundaries of the valid image. Areas outside the valid image boundaries, or areas within the boundaries but whose image quality does not meet monitoring requirements, will be corrected as gaps. This type of failure is usually caused by lens damage, lens contamination, or sudden changes in lighting conditions, affecting only a portion of the monitoring image at a specific preset location.

[0040] Through the above analysis, the server can accurately determine the type of failure of the target PTZ device, laying the foundation for subsequent targeted identification of the gap area and ensuring the accuracy and effectiveness of the replacement operation.

[0041] S103. Determine the notch area corresponding to the preset position of the failed gimbal based on the digital twin model and the failure type; Among them, a failed PTZ refers to a PTZ device that has been determined to be ineffective; a preset position refers to a fixed monitoring angle and range that the PTZ device is set in advance. Each preset position corresponds to a specific monitoring area. The device can quickly switch preset positions to inspect different areas; a gap area refers to an area where the monitoring coverage is missing due to the failure of the PTZ device. It is a target area that needs to be filled by other neighboring devices.

[0042] Different types of failures correspond to different methods of determining the gap region, which requires specific analysis in conjunction with relevant data in the digital twin model.

[0043] When the failure type is "total failure," it means that the PTZ device is completely inoperable, and the monitoring areas corresponding to all its preset positions will be lost. In this case, the server will access the static data of the failed PTZ from the digital twin model. This data stores detailed information about all preset positions, including the monitoring angle corresponding to each preset position, the geographical coordinate boundaries of the coverage area, and key monitoring areas. The server integrates the areas that all preset positions are required to cover, removes overlapping parts, and the resulting overall area is the gap area in the case of total failure. For example, if a failed PTZ has three preset positions, covering the east corner, west corner, and central area of ​​workshop A respectively, then the sum of these three areas (excluding overlapping parts) is the gap area of ​​the failed PTZ.

[0044] When the failure type is partial view failure, meaning one or more specific preset positions cannot respond normally, resulting in the corresponding monitored area being out of coverage, the server will directly determine the preset coverage area corresponding to that preset position as the gap area based on the information of the target preset position (i.e., the failed preset position) recorded in the digital twin model. For example, if preset position 2 of a PTZ device cannot respond due to a motor malfunction, and this preset position was originally responsible for covering the entrance area of ​​a warehouse, then the warehouse entrance area becomes the gap area. At the same time, the server will check whether other preset positions are working normally to ensure that only the area corresponding to the failed preset position is marked as a gap, avoiding misjudgment.

[0045] When the failure type is a local anomaly in the returned image, the handling method is relatively complex. The server first acquires the abnormal image returned by the preset position and uses image analysis algorithms (such as edge detection and feature extraction) to determine the boundary of the valid image, i.e., the area range of the monitored object that can be clearly identified. Then, the boundary of the valid image is compared with the preset coverage area of ​​the target preset position in the digital twin model. The preset coverage area is the area that the preset position should cover when working normally, usually stored in static data in the form of geographic coordinates or pixel coordinates. For areas within the preset coverage area but not within the valid image boundary, and for areas within the valid image boundary but whose image quality does not meet monitoring requirements (such as blurriness or severe occlusion), the server will correct them as gap areas. For example, if the preset coverage area of ​​a preset position is the entire parking lot, but only half of the returned image is clear, and the other half is blurry due to lens damage, then the blurry half will be identified as a gap area.

[0046] By employing the aforementioned methods for handling different types of failures, the server can accurately and comprehensively identify the missing areas corresponding to the preset positions of the failed PTZ cameras, ensuring that subsequent replacement operations are targeted and cover all areas lacking monitoring.

[0047] S104. Divide the gap area into multiple gap sub-regions with gap risk value labels; Among them, the gap sub-region refers to a smaller area unit obtained by dividing the gap region according to certain rules. Each sub-region has relatively independent monitoring needs and risk characteristics. The gap risk value identifier is a numerical or level identifier used to indicate the monitoring importance or potential risk level of each gap sub-region. The higher the risk value, the more likely the sub-region needs to be prioritized for monitoring.

[0048] In forest fire prevention scenarios, it is crucial to accurately assess the fire risk distribution in gap areas, because differences in the types and quantities of flammable materials and environmental conditions in different areas can lead to significant differences in the likelihood of fire occurrence and spread. Therefore, it is necessary to prioritize monitoring coverage in high-risk areas.

[0049] Specifically, the server receives real-time operational status data from each PTZ device. When a PTZ device is detected to be malfunctioning through the digital twin model, the server immediately records the exact time when the PTZ device transitioned from a normal state to a malfunctioning state—the failure timestamp. Only an accurate timestamp can guarantee that the video data obtained is the latest and most accurate representation of the gap area before the failure.

[0050] The server stores video data collected during the daily inspections of each PTZ device, and these videos are categorized and archived chronologically. After obtaining the failure timestamp, the server retrieves the video footage captured by the failed PTZ device during its most recent complete panoramic inspection before failure. A "complete panoramic inspection" here refers to the PTZ device comprehensively capturing the entire monitoring area it is responsible for according to a preset inspection route and angle, ensuring no monitoring perspective is missed. The video duration is usually consistent with the PTZ device's inspection cycle (e.g., 1 hour, 2 hours, etc.). For example, if PTZ A's inspection cycle is 1 hour, and it performs panoramic inspections every hour on the hour, with a failure timestamp of 14:30:25, then the server will retrieve and obtain the panoramic inspection video captured by PTZ A between 13:00 and 14:00, covering its entire monitoring area. This video is the complete panoramic inspection video before failure. If the expiration timestamp is close to the end time of the last complete panoramic inspection (e.g., the expiration timestamp is 14:10), in order to ensure that the acquired video can fully reflect the situation of the gap area, the server will also acquire the video of the last complete panoramic inspection (13:00-14:00) instead of the incomplete video from 14:00-14:10.

[0051] Next, the server invokes a pre-defined flammable material feature layout library. This library was constructed before server deployment, based on common flammable materials in mountainous areas (such as dry trees, shrubs, weeds, leaf litter, flammable crop straw, and temporarily stored timber), through image acquisition, feature extraction, and annotation. The library contains not only visual features of various flammable materials (such as color, shape, and texture) but also flammability information (such as ignition point, burning speed, and the ability to spread after combustion). During frame-by-frame object detection, the server employs advanced computer vision algorithms, such as the deep learning-based YOLO (YouOnlyLookOnce) algorithm, to process each frame of the complete panoramic inspection video. For each frame, the algorithm first preprocesses the image (such as denoising and image enhancement), then compares the processed image with the flammable material features in the pre-defined flammable material feature layout library to identify various objects in the image, determine which objects are flammable, and identify the specific location and category of each flammable material.

[0052] After identifying flammable materials, the server extracts the flammability level of each material based on the flammability characteristics stored in a pre-defined flammable material feature layout database. For example, the database records that dry weeds have a low ignition point (approximately 200-300℃), burn quickly, and spread easily after burning, thus having a high flammability level. Fresh evergreen leaves, containing more moisture, have a higher ignition point (approximately 400-500℃), burn slowly, and do not spread easily, thus having a low flammability level. Subsequently, the server converts the extracted flammability level into a specific flammability score according to a pre-defined flammability scoring rule. This scoring rule typically divides flammability into multiple levels, each corresponding to a scoring range. For example, flammability might be divided into five levels: extremely high, high, medium, low, and extremely low, corresponding to scores of 90-100, 70-89, 50-69, 30-49, and 0-29, respectively. When determining the score, the server makes fine adjustments based on the specific circumstances of the flammable material. For example, even with dry weeds, if the density of dry weeds in a certain area is higher and the pile is thicker, its flammability will be higher than that of dry weeds with lower density and a thinner pile, and the corresponding flammability score will also be higher, perhaps adjusting from 80 points to 85 points. If a dry tree has a large trunk diameter, although its ignition point is not high, it releases more heat when burning, which may cause the fire to spread faster, and its flammability score will also be appropriately increased. In this way, the server assigns an accurate flammability score to all flammable materials identified in each frame of the video image.

[0053] Determining the spatial coordinates of each object in the panoramic video requires combining the gimbal's own positioning information (such as the GPS coordinates of its installation location), the gimbal's shooting angle parameters (such as horizontal rotation angle and vertical pitch angle), and lens parameters (such as focal length and field of view). A coordinate transformation algorithm is then used to convert the two-dimensional coordinates (pixel coordinates) of the objects in the image into three-dimensional coordinates (such as latitude, longitude, and altitude) in real physical space. For example, given the GPS coordinates of the gimbal's installation location (120.5°E, 30.2°N, altitude 500m), and considering that the horizontal rotation angle is 30°, the vertical pitch angle is -10°, and the lens focal length is 10mm when shooting a region of dry weeds, the coordinates of this region in real space can be calculated using a coordinate transformation algorithm as (120.501°E, 30.202°N, altitude 495m).

[0054] After obtaining the spatial coordinates and corresponding flammability ratings of each object, the server uses a spatial interpolation algorithm to estimate the flammability risk values ​​of other undetected objects within the gap area. Since only obvious objects in the image can be identified during frame-by-frame object detection, the flammability risk values ​​of blank areas between objects or areas not covered by objects cannot be directly obtained. Therefore, spatial interpolation algorithms are needed for estimation. Commonly used spatial interpolation algorithms include inverse distance weighted interpolation and Kriging interpolation. Taking inverse distance weighted interpolation as an example, the basic principle of this algorithm is that the attribute value of an unknown point (in this case, the flammability risk value) is related to the attribute values ​​of surrounding known points, and the correlation weakens with increasing distance. Specifically, for an unknown point P within the gap area, the server searches for all known object points with determined flammability ratings within a certain range (e.g., 100 meters) around the point, calculates the distances from these known points to the unknown point P, and then uses the reciprocal of the distance as a weight to perform a weighted sum of the flammability ratings of each known point to obtain the flammability risk value of the unknown point P. For example, there are three known points A, B, and C around an unknown point P, with flammability scores of 80, 70, and 60 respectively, and distances from point P of 10 meters, 20 meters, and 30 meters respectively. Then the weights of each known point are 1 / 10, 1 / 20, and 1 / 30 respectively. The flammability risk value of the unknown point P is approximately 72.3 points (80×1 / 10+70×1 / 20+60×1 / 30)÷(1 / 10+1 / 20+1 / 30).

[0055] After calculating the flammability risk values ​​for all locations within the gap area using a spatial interpolation algorithm, the server visualizes these risk values ​​as a risk heatmap. The risk heatmap typically uses color gradients to represent the risk levels, for example, using five colors: red, orange, yellow, green, and blue, corresponding to extremely high risk (80-100 points), high risk (60-79 points), medium risk (40-59 points), low risk (20-39 points), and extremely low risk (0-19 points), respectively. Simultaneously, the specific risk value for each location is marked on the risk heatmap, i.e., the gap risk value identifier, such as "Risk Value: 92" in the red area and "Risk Value: 75" in the orange area, to provide a more intuitive and accurate understanding of the risk distribution within the gap area.

[0056] Finally, based on the distribution pattern of the gap risk values ​​in the risk heatmap, the server divides the entire gap area into multiple independent gap sub-regions according to a preset risk level classification standard. The classification standard is usually formulated based on the actual needs and experience of forest fire prevention in mountainous areas. For example, areas with risk values ​​of 80-100 are classified as high-risk gap sub-regions, which are high-risk fire areas and require priority monitoring and remediation; areas with risk values ​​of 50-79 are classified as medium-risk gap sub-regions, requiring secondary priority remediation; and areas with risk values ​​of 0-49 are classified as low-risk gap sub-regions, where remediation priority can be appropriately reduced.

[0057] Optionally, frame-by-frame object detection and flammability scoring can be performed on the complete panoramic video. The server calls an object detection model based on the YOLOv8 algorithm. This model has been trained in advance with a large number of mountain flammable material images and flammability level labeled samples. It can accurately identify more than 10 common flammable materials such as dry trees and weeds. During detection, the video is extracted frame by frame, and the resolution of each frame image is adjusted to 640×640 pixels. Batch processing is used, processing 32 frames of images each time. After detection, the flammable materials in each frame image are classified and statistically analyzed. An initial score is assigned to each flammable material according to a preset flammable material scoring table (e.g., dry weeds 85 points, dry shrubs 90 points, fresh trees 30 points, etc.). Then, the score is fine-tuned by combining the area size of the flammable material in the image (the larger the area, the higher the score, such as dry weeds with an area of ​​more than 100 pixels × 100 pixels, the score increases by 5 points) and the density (the higher the density, the higher the score, such as more than 10 dry shrubs per square meter, the score increases by 8 points) to obtain the final flammability score value. Optionally, if the pan-tilt device has an infrared imaging function, it can combine visible light images and infrared images to detect flammable materials. Infrared images can more clearly show the temperature distribution of objects, which helps to identify flammable materials hidden under vegetation (such as the underground leaf litter layer), further improving the accuracy of detection.

[0058] S105. Determine the candidate neighboring machine that matches each of the gap sub-regions, and determine the optimal takeover gimbal corresponding to each of the gap sub-regions from the candidate neighboring machine; Among them, candidate neighboring devices refer to other normally operating PTZ devices within the target area that may have the capability to cover the missing sub-area, and are potential fillers; the optimal takeover PTZ refers to the PTZ selected from the candidate neighboring devices whose comprehensive performance is most suitable for the corresponding missing sub-area filler needs, and must meet the optimal balance in terms of coverage capability, task interference, energy consumption cost and response speed.

[0059] After identifying candidate neighboring devices, multi-dimensional screening and evaluation are needed to find the best-performing replacement device for the missing sub-region, ensuring that the replacement task is completed efficiently, with low interference and low cost.

[0060] The initial screening of candidate neighboring devices is performed. Based on the digital twin model of each candidate neighboring device, the server analyzes its monitoring parameters (such as rotation angle range, lens focal length, installation height, etc.), calculates the coverage area of ​​each neighboring device for the gap sub-region, and compares this coverage with the total area of ​​the sub-region to obtain the overlap rate. If a neighboring device cannot cover the gap sub-region at all, or its overlap rate is lower than a preset overlap rate threshold (e.g., 20%), it is excluded, and the remaining neighboring devices become the remaining candidate neighboring devices. This step quickly eliminates obviously unsuitable devices, reducing the computational workload of subsequent evaluations.

[0061] Next, the coverage capability matching degree is calculated. First, the theoretical maximum visible coverage area (the area that can cover the gap sub-region assuming no obstructions) is calculated based on the parameters of the remaining candidate neighboring units, and the ratio of this area to the total area of ​​the sub-region is used to obtain the initial ratio. Then, the line-of-sight occlusion model is invoked, and the 3D environmental data of the target area (such as building height, tree positions, etc.) and the installation position and angle parameters of the neighboring units are input to simulate the obstruction of the line of sight by obstructions in the actual scene, and the initial ratio is corrected. For example, if a neighboring unit has a theoretical coverage ratio of 80%, but a building obstructs 30% of the line of sight, the corrected coverage capability matching degree is 56%. This indicator directly reflects the actual monitoring capability of the neighboring unit for the gap sub-region and is one of the core dimensions of the evaluation.

[0062] Then, the task interference level is evaluated. The server virtually inserts a replacement task into the original task plan of the remaining candidate neighboring machines using a digital twin model, calculating the change in the inspection interval of the original high-priority tasks (such as key area inspections). For example, assuming the original inspection interval for task A is 30 minutes, after inserting the replacement task, the interval is extended to 45 minutes, an extension rate of (45-30) / 30 = 50%. The average extension rate of all original high-priority tasks is taken, and its reciprocal is used as the task interference level (e.g., if the average extension rate is 50%, the interference level is 1 / 0.5 = 2). The higher the interference level, the smaller the impact of the replacement task on the original important tasks, and the more suitable the neighboring machine is to take over the replacement.

[0063] Next, the energy consumption cost is calculated. The server calculates the angle and distance the gimbal needs to rotate based on the current positions of the remaining candidate neighboring machines (dynamic data from the digital twin model) and the pre-set replacement position (determined based on the coordinates of the gap sub-region and neighboring machine parameters), and estimates the energy consumption during movement by combining this with motor power parameters. Simultaneously, based on the estimated execution time of the replacement task (e.g., 5 minutes per inspection) and the device's power consumption per unit time, the energy consumption during the task execution phase is calculated. The sum of these two figures represents the energy consumption cost for the neighboring machine to execute one cycle of the replacement task. For example, if the movement energy consumption is 0.2 kWh and the execution energy consumption is 0.3 kWh, the total energy cost is 0.5 kWh. In scenarios with limited energy or where energy conservation is a priority, the weight of this indicator will be increased accordingly.

[0064] Next, the response speed is determined. The server analyzes the current task status of the remaining candidate neighboring machines: if a neighboring machine is currently executing a task, it first calculates the remaining time required to complete the current task; then, based on the rotation angle from the current position to the preset replacement position and the maximum rotation speed of the gimbal, it estimates the time required for rotation. The sum of these two is the estimated response time. For example, if it takes 2 minutes to complete the current task and 1 minute to rotate to the preset replacement position, the response speed is 3 minutes. For high-urgency gap sub-areas (such as high-risk areas), neighboring machines with faster response speeds have an advantage.

[0065] Next, a weighted summation model is used to calculate the overall score. First, the coverage matching degree (0-100 points), task interference degree (0-10 points), energy consumption cost (reverse scoring, 0-100 points, lower energy consumption results in a higher score), and response speed (reverse scoring, 0-100 points, shorter response time results in a higher score) are standardized. Then, weights are assigned according to actual needs. For example, in security-priority scenarios such as industrial parks, the weights are set as follows: coverage matching degree 40%, response speed 30%, task interference degree 20%, and energy consumption cost 10%. Finally, the weighted total score for each remaining candidate neighbor is calculated: Overall Score = Coverage Matching Degree × 40% + Task Interference Degree × 20% + Energy Consumption Cost × 10% + Response Speed ​​× 30%.

[0066] Finally, the remaining candidate neighbor with the highest overall score is selected as the optimal takeover gimbal. If there are cases with the same score, a secondary selection can be performed according to preset rules (such as prioritizing neighbor with faster response speed) to ensure that the final selected gimbal achieves the best balance between coverage, task interference, energy consumption, and response speed.

[0067] Optionally, prioritize the indicators (e.g., coverage capability > response speed > task interference > energy consumption cost). First, sort the remaining candidate neighboring machines from highest to lowest coverage capability matching degree, and select the top 50% of neighboring machines; then sort them from fastest to slowest response speed within the same range, and select the top 50%; next, sort them from highest to lowest task interference degree, and select the top 50%; finally, calculate the energy consumption cost of the remaining neighboring machines and select the one with the lowest energy consumption as the optimal takeover PTZ. For example, first select the top 5 in coverage capability from 10 neighboring machines, then select the top 3 in response speed, then select the top 2 in task interference degree, and finally select the one with the lowest energy consumption from the remaining 2 as the optimal one. It is understandable that other methods can also be used to determine the optimal takeover PTZ, such as introducing a machine learning model to train the model using historical data to predict and score the replacement effect of each neighboring machine, etc., which is not limited here. It should be added that the preset overlap rate threshold and the weight of each indicator can be dynamically adjusted according to the actual situation of the target area (e.g., environmental complexity, urgency of inspection tasks) to improve the flexibility and accuracy of the selection.

[0068] S106. Based on the digital twin model of the optimal takeover PTZ, determine the inspection execution task sequence that includes the newly added task corresponding to the sub-region that needs to fill the gap. After determining the optimal takeover PTZ for each gap sub-region, the newly added replacement tasks can be reasonably integrated into the original inspection plan of the optimal takeover PTZ to form a complete, efficient, and feasible task sequence. This ensures that the gap sub-regions can be inspected in a timely manner, while minimizing interference with the original inspection work of the optimal takeover PTZ. The specific process of this step will be explained in detail in steps S201-S206, and will not be repeated here.

[0069] S107. Control the optimal takeover PTZ to perform a replacement operation according to the inspection execution task sequence; The server sends the generated inspection task sequence to the control module of the optimal takeover PTZ. The control module, the component on the optimal takeover PTZ responsible for receiving instructions and controlling device operation, parses various information from the task sequence, including the execution time, target location (preset position), inspection duration, and shooting parameters for each task. Then, the control module executes each task sequentially according to the task sequence. When executing a new replacement task, the control module controls the PTZ motor to rotate according to the preset position specified in the task, adjusting the PTZ's horizontal and vertical angles to ensure the camera accurately aligns with the gap area. Simultaneously, it sets the camera's focal length, exposure, and other parameters according to the task requirements to ensure clear and effective images or videos are captured.

[0070] During execution, the optimal takeover PTZ will feed back its operational status to the server in real time through its digital twin model. The server uses the digital twin model to monitor the PTZ's dynamic data such as position, angle, operating speed, and energy consumption in real time, ensuring that the PTZ executes operations normally according to the task sequence.

[0071] S108. Once it is determined that the failed PTZ has returned to normal, the takeover exit process is automatically triggered, and the original inspection plan of the optimal takeover PTZ is restored.

[0072] After a failed PTZ unit recovers from its failed state to its normal working state, the goal is to promptly terminate the replacement task of the optimal takeover PTZ unit, restore it to its normal inspection working state, avoid resource waste and task duplication, and ensure the efficient and orderly operation of the entire inspection server.

[0073] First, the server needs to continuously monitor the status of the failed PTZ (panel camera) to determine if it has recovered. Monitoring methods include receiving heartbeat signals, device health data, and self-test reports from the failed PTZ. When the server first detects a recovery signal from the failed PTZ, it does not immediately confirm full recovery but marks the PTZ's status as an observation period. The observation period is set to avoid frequent switching between takeover and retake due to PTZs briefly recovering and then failing again, ensuring server stability. The duration of the observation period can be preset according to actual needs, such as 30 minutes or 1 hour.

[0074] During the observation period, the server will control the optimal takeover PTZ to continue performing the replacement operation to ensure that the missing sub-area still receives effective inspection coverage during this time. Simultaneously, the server will continuously and closely monitor the dynamic data of the recovered PTZ. This dynamic data includes, but is not limited to, device operating parameters (such as motor speed, camera focus adjustment accuracy), data transmission stability, inspection image quality, and the continuity of the heartbeat signal. By analyzing this data, the server determines whether the PTZ can operate continuously and stably. If, during the observation period, the recovered PTZ operates continuously and stably for the preset observation period duration, and no abnormalities occur (such as heartbeat signal loss, image transmission interruption, or decreased device health), the server officially confirms that the failed PTZ has returned to normal. At this point, the server will automatically trigger the takeover exit process.

[0075] The takeover and exit process mainly includes the following steps: First, the server sends a stop-fill task instruction to the optimal takeover platform, informing it that it can stop inspecting the gap sub-region; second, after receiving the instruction, the optimal takeover platform completes the currently executing fill task (if any) to avoid information loss due to interruption halfway through task execution; third, the optimal takeover platform adjusts from the fill preset position back to the preset position corresponding to the next task in its original inspection plan, preparing to resume the original inspection work; fourth, the server switches the inspection task sequence of the optimal takeover platform back to its original inspection plan, deleting the newly added fill task, ensuring that the platform executes the inspection task according to the original arrangement.

[0076] In this embodiment, a digital twin model is constructed to perceive the operating status of the entire monitoring system in real time, and multiple links such as failure diagnosis, risk assessment, optimal neighbor selection and intelligent task reorganization are integrated. Therefore, a closed-loop intelligent decision-making process is formed from accurate fault perception to automatic generation of the optimal replacement scheme. This effectively solves the technical problems of existing technologies that rely on manual intervention or use fixed replacement strategies after PTZ failure, resulting in slow response, low efficiency and poor replacement effect. As a result, the unmanned, intelligent, optimized emergency response and automatic recovery of monitoring system failures are realized, and the overall robustness and operation and maintenance efficiency of the system are improved.

[0077] Based on the above, the following is a more detailed description of the process provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the gimbal failure inspection and replacement method based on neighboring machine cooperation in this application.

[0078] S201. Based on the digital twin model of the optimal takeover PTZ, determine the filling angle covering the gap sub-region and generate a new task including the takeover task preset position; After determining the optimal takeover PTZ, it is necessary to formulate precise replacement task parameters for the optimal takeover PTZ to ensure that it can accurately cover the gap sub-area, laying the foundation for subsequent integration into the original inspection plan.

[0079] The server invokes the digital twin model of the optimal takeover PTZ (pan-tilt unit), extracting key parameters including static data such as the PTZ's installation position, horizontal rotation range (e.g., 0°-360°), vertical rotation range (e.g., -30°-90°), lens focal length range, and field of view, as well as dynamic data such as the current horizontal angle, vertical angle, and lens focal length. This data provides the basis for calculating the compensation angle. Next, based on the 3D spatial coordinates of the notch sub-region (obtained from the 3D model of the target region) and the installation coordinates of the optimal takeover PTZ, the server uses spatial geometric calculations to initially determine the horizontal and vertical angles the PTZ needs to be adjusted to ensure the lens is aligned with the center of the notch sub-region. For example, if the notch sub-region is located 30° northeast of the optimal takeover PTZ with a 10° elevation angle, the initially calculated compensation angle is 30° horizontally and 10° vertically.

[0080] Then, by combining the lens parameters (such as the field of view) in the digital twin model, verify whether the initially determined replacement angle can completely cover the notch sub-region. If the field of view is too small, resulting in insufficient coverage, adjust the focal length (shorten the focal length to expand the field of view) or fine-tune the angle to ensure that the entire area of ​​the notch sub-region is within the lens's field of view. At the same time, it is also necessary to refer to the line-of-sight occlusion data in the digital twin model (from previously updated static data) to avoid obstructions such as buildings and trees, and correct the replacement angle to ensure unobstructed coverage.

[0081] After determining the final positioning angle, it is combined with corresponding parameters such as focal length and shooting duration to generate a takeover task preset. The takeover task preset is a set of instructions containing specific operational parameters, such as "horizontal angle 35°, vertical angle 12°, focal length 50mm, shooting duration 3 minutes." Finally, based on the takeover task preset, information such as task number, execution frequency (e.g., once every 15 minutes), and data upload requirements are added to form a complete new task. New tasks must be clear and specific to ensure that the optimal takeover gimbal can accurately understand and execute them.

[0082] S202. Obtain the original inspection plan for the optimal takeover PTZ and determine the original risk value, priority and time percentage of each sub-plan in the original inspection plan. The original inspection plan refers to the routine inspection task arrangement before the optimal takeover PTZ assumes the supplementary task, which includes multiple sub-original plans; the sub-original plan refers to the specific inspection task unit divided in the original inspection plan, corresponding to a specific monitoring area or equipment; the original risk value refers to the risk level quantification value of the area covered by the sub-original plan, used to indicate the inspection importance of the area; the time proportion refers to the proportion of the sub-original plan in the total time of the original inspection plan, reflecting the time resources required to execute the task.

[0083] After generating new tasks, the goal is to obtain the original task schedule and key attributes of the optimal takeover gimbal, so as to provide data support for the subsequent reasonable integration of new tasks into the original plan and ensure that the replacement tasks are executed in coordination with the original tasks.

[0084] First, the server communicates with the control server of the optimal takeover PTZ to obtain its stored original inspection plan. The original inspection plan typically exists in the form of an electronic document or database record, containing information such as task name, execution time, coverage area, and execution frequency. For example, the original inspection plan might include sub-plans such as "Inspect area A once per hour" and "Inspect device B once every two hours." Next, the original inspection plan is parsed and broken down into multiple independent sub-plans. The breakdown can be based on different coverage areas, different inspection objects, or different execution times. For example, three independent inspection tasks covering areas A, B, and C can be broken down into three sub-plans.

[0085] Next, determine the original risk value corresponding to each sub-plan. The original risk value is usually preset based on factors such as the environmental characteristics of the area covered by the sub-plan (e.g., whether there are flammable materials, the density of equipment), historical failure data, and safety level. Alternatively, it can be determined using the method for determining the gap risk value in step S104 above. The value range can be 0-100 (the higher the value, the higher the risk). For example, the original risk value of a sub-plan covering a flammable warehouse might be set to 90, while the original risk value of a sub-plan covering ordinary areas might be set to 30.

[0086] Next, determine the priority of each sub-plan. Priority is usually positively correlated with the original risk value; the higher the original risk value, the higher the priority. However, this may be adjusted based on actual needs. Priorities can be represented numerically (e.g., levels 1-5, with level 1 being the highest) or described in words (e.g., "urgent," "routine," "low priority"). For example, a sub-plan with an original risk value of 90 is assigned a priority of level 1, and one with an original risk value of 30 is assigned a priority of level 3. Calculate the time percentage for each sub-plan. Calculate the total duration of the original inspection plan (e.g., 24 hours a day), then calculate the product of the single execution time and execution frequency for each sub-plan (e.g., 3 minutes per execution, once per hour, totaling 3 × 24 = 72 minutes per day). Divide this value by the total duration of the original inspection plan (24 × 60 = 1440 minutes) to obtain the time percentage (72 / 1440 = 5%). The time percentage reflects the weight of the sub-plan in the allocation of time resources.

[0087] S203. Associate the time percentage with the original risk value to obtain the risk-time mapping relationship; It is necessary to analyze the correlation between the two to provide a basis for the allocation of time resources for new tasks, ensuring that the time ratio of the supplementary tasks matches their own risk level, while also conforming to the resource allocation logic of the original plan.

[0088] First, collect the original risk values ​​and corresponding time percentages for all sub-plans to form a dataset. For example, sub-plan A (risk value 90, time percentage 8%), sub-plan B (risk value 70, time percentage 5%), sub-plan C (risk value 50, time percentage 3%), etc. Next, organize and classify the dataset, grouping sub-plans with similar original risk values ​​together. For example, group risk values ​​of 80-100 into the high-risk group, 60-79 into the medium-high-risk group, 40-59 into the medium-risk group, 20-39 into the medium-low-risk group, and 0-19 into the low-risk group. Then, calculate the average time percentage for each group of sub-plans. For example, if the high-risk group has 3 sub-plans with time percentages of 8%, 7%, and 9% respectively, the average time percentage is (8%+7%+9%) / 3=8%; the average time percentage for the medium-high-risk group is 5%, and so on.

[0089] Plot a scatter plot or line graph with the original risk value range on the horizontal axis and the average time percentage on the vertical axis to visually demonstrate the correlation between the two. Generally, the higher the original risk value, the larger the average time percentage, showing a positive correlation. For example, for every 20 units increase in the risk value, the average time percentage increases by 2%-3%. Finally, based on the data distribution and correlation trend, establish a risk-time mapping model. This model can be a linear equation (e.g., time percentage = 0.1 × risk value - 2%), a piecewise function (e.g., time percentage = 0.05 × risk value when risk value is 0-50; time percentage = 0.08 × risk value - 1.5% when risk value is 51-100), or a reference table (e.g., risk value 90 corresponds to 8%, 70 corresponds to 5%, etc.). The mapping relationship needs to be verified to ensure it accurately reflects the risk and time allocation pattern in the original plan.

[0090] S204. Compare the gap risk value identifier corresponding to the gap sub-area with each of the original risk values ​​to determine the new priority of the new task to be inserted into the original inspection plan; This step requires determining the execution priority of new tasks based on the risk level of the gap sub-region, ensuring that high-risk fill-in tasks can be executed first, while avoiding disrupting the priority logic of the original plan.

[0091] First, clearly define the gap risk value identifier corresponding to the gap sub-area. This value is determined based on factors such as flammability characteristics and environmental risks when dividing the gap sub-areas. For example, the gap risk value identifier for a certain gap sub-area is 85. Collect the original risk values ​​of all sub-plans in the original inspection plan to form a risk value list. For example, sub-plan A (90), sub-plan B (75), sub-plan C (60), and sub-plan D (40). Compare the gap risk value identifier with the original risk values ​​in the risk value list one by one to determine its position in the risk level sequence. For example, after comparing the gap risk value 85 with the original risk value, it is located between 90 (A) and 75 (B), that is, higher than 75 in B and lower than 90 in A.

[0092] Based on the comparison results and the priority rules of the original plan (generally, the higher the risk value, the higher the priority), determine the new priority of the new task. If the original priorities are ranked from highest to lowest risk value as A (Level 1), B (Level 2), C (Level 3), and D (Level 4), then the new priority corresponding to the gap risk value of 85 should be between Level 1 and Level 2, which can be set to Level 1.5, or adjusted to Level 1 (same level as A) or Level 2 (but ranked before B) according to actual needs. If the gap risk value is higher than all original risk values, for example, a gap risk value of 95, higher than A's 90, then the new priority is set to the highest level (e.g., Level 0), ensuring that the new task is executed before all original sub-plans; if the gap risk value is lower than all original risk values, for example, 30, lower than D's 40, then the new priority is set to the lowest level (e.g., Level 5), executed after all original sub-plans.

[0093] In addition, special cases need to be considered. For example, if there are multiple gap sub-regions corresponding to multiple new tasks, the priority of the new tasks should also be determined according to the gap risk value. The new tasks with higher risk should be executed first.

[0094] S205. Determine the proportion of new time for the new task in the total time corresponding to the original inspection plan based on the risk-time mapping relationship and the gap risk value identifier. After determining the new priority of the new tasks, it is necessary to allocate a reasonable proportion of time to the new tasks based on the risk level of the gap sub-area and the original time allocation pattern, to ensure that the filling tasks have enough time resources to complete, while avoiding excessive time consumption that would affect the execution of the original inspection tasks.

[0095] First, clarify the specific form of the established risk-time mapping relationship, whether it's a linear equation, a piecewise function, or a lookup table. For example, the mapping relationship might be a linear rule where "for every 20 increase in risk value, the time percentage increases by 2%", or a lookup table where "risk values ​​of 80-100 correspond to 8%, 60-79 to 5%, and 40-59 to 3%". Next, extract the gap risk value identifier corresponding to the gap sub-region, assuming this value is 85. Substitute this risk value into the risk-time mapping relationship to find or calculate the corresponding time percentage. Then, fine-tune the initially obtained time percentage based on the actual situation. Factors to consider include the size of the gap sub-region (larger areas may require longer inspection times), the complexity of the inspection (e.g., high equipment density within the area requires more time), and the time margin of the original inspection plan (if the original plan has limited time, the time percentage of the new task can be appropriately reduced, but basic coverage requirements must be guaranteed). For example, if the initially obtained time percentage is 8%, but the gap sub-region is small, it can be fine-tuned to 6%; if the original plan has ample time and the area is complex, it can be increased to 9%.

[0096] Simultaneously, it is necessary to ensure that the time allocation for newly added tasks matches the new priorities. High-priority tasks should generally be allocated a relatively higher time allocation to guarantee inspection quality. For example, the time allocation for a new task with a new priority of Level 1 (the highest) should not be lower than the average time allocation of the original Level 1 sub-plan. Finally, determine the final new time allocation and record the calculation basis and adjustment factors corresponding to this allocation for subsequent evaluation and optimization of the inspection plan.

[0097] S206. Determine the sequence of inspection tasks by combining the new priority and the new time percentage.

[0098] Collect the priorities and time percentages of all existing sub-plans, as well as the new priorities and time percentages of newly added tasks, to form a task attribute list. For example, the existing sub-plans are A (priority level 1, time percentage 8%), B (priority level 2, time percentage 5%), and C (priority level 3, time percentage 3%), and the newly added task is D (new priority level 1.5, new time percentage 6%).

[0099] Next, all tasks (including existing sub-plans and new tasks) are initially sorted according to priority from highest to lowest. Tasks with the same priority can be sorted according to their original execution order or task generation time. For example, the initial sorting of the above tasks is A (Level 1), D (Level 1.5), B (Level 2), and C (Level 3). Then, based on the time proportion of each task and the total time of the original inspection plan (e.g., 24 hours a day), the actual execution time of each task is calculated. Based on the sorting results and execution time, the specific execution time and order of each task are planned. For tasks that are executed continuously, the next task starts immediately after the previous task is completed; for tasks with time intervals (e.g., periodic inspections), other tasks are inserted during the interval.

[0100] At the same time, the execution frequency and periodicity of the tasks need to be considered. If the original sub-plan is a periodic task (such as executing once every 2 hours), then its periodic regularity must be ensured when planning the sequence, and new tasks should be inserted into the periodic intervals to avoid breaking the original cycle.

[0101] In this embodiment, since the replacement angle is accurately determined based on the digital twin model to generate new tasks, and a mapping relationship is established by combining the risks, priorities and time proportions of the original inspection plan, the new priority and new time proportion of the new tasks are determined and integrated to form a task sequence. Therefore, the replacement tasks and the original inspection plan can be efficiently coordinated, effectively solving the problems of plan chaos, unreasonable time allocation and priority conflict caused by the insertion of replacement tasks. Thus, the orderliness of the optimal takeover of the PTZ inspection tasks, the rationality of resource allocation and the accuracy of replacement inspection are achieved.

[0102] The server in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the physical device structure of a server in an embodiment of this application.

[0103] It should be noted that, Figure 3 The server structure shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0104] like Figure 3As shown, the server includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 302 or a program loaded from storage portion 308 into Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for server operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0105] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0106] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0107] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution server, device, or apparatus.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of servers, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0109] Specifically, the server in this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the gimbal failure inspection and replacement method based on neighboring machine cooperation provided in the above embodiment.

[0110] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the server described in the above embodiments; or it may exist independently and not assembled into the server. The storage medium carries one or more computer programs that, when executed by a processor of the server, cause the server to implement the gimbal failure inspection and replacement method based on neighbor-machine cooperation provided in the above embodiments.

[0111] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0112] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A gimbal failure inspection and replacement method based on neighboring machine cooperation, applied to a server, characterized in that, The method includes: Determine the digital twin model of each gimbal device within the target area, and determine whether the corresponding target gimbal device is faulty based on the digital twin model; If the target gimbal device is determined to be faulty, then the type of fault is determined; The notch area corresponding to the preset position of the failed gimbal is determined based on the digital twin model and the failure type. The gap region is divided into multiple gap sub-regions with gap risk value identifiers; Determine candidate neighboring machines that match each of the aforementioned gap sub-regions, and determine the optimal takeover gimbal corresponding to each of the aforementioned gap sub-regions from the candidate neighboring machines; Based on the digital twin model of the optimal takeover PTZ, determine the inspection execution task sequence that includes the newly added tasks corresponding to the gap sub-regions that need to be filled; The optimal takeover PTZ is controlled to perform a replacement operation according to the inspection execution task sequence described above. Once the failed PTZ unit is confirmed to be back to normal, the takeover exit process is automatically triggered, restoring the original inspection plan of the optimal takeover PTZ unit.

2. The method according to claim 1, characterized in that, The step of determining the notch region corresponding to the preset position of the failed gimbal based on the digital twin model and the failure type specifically includes: If the dynamic data in the digital twin model shows that the device health is zero or the heartbeat is lost continuously, the failure type is determined to be a complete failure, and the area corresponding to all preset positions in the static data of the failed PTZ is the gap area. If the target preset position response times out, the failure type is determined to be partial view failure, and the corresponding area is determined to be the gap area based on the target preset position; If the returned image has local anomalies, the missing area is corrected by combining the effective image boundary of the anomaly image with the preset coverage of the target preset position.

3. The method according to claim 1, characterized in that, The step of dividing the gap region into multiple gap sub-regions with gap risk value identifiers specifically includes: Obtain the failure timestamp of the failed gimbal; Obtain a complete panoramic inspection video segment collected by the failed PTZ camera before the failure timestamp; Based on a pre-set flammable material feature layout library, the complete panoramic inspection video is subjected to frame-by-frame object detection, the flammability of each object is extracted, and the flammability score of all objects is determined based on the flammability. By combining the spatial coordinates of each object in the panoramic video with the flammability score, a risk heat map of the gap area is generated through a spatial interpolation algorithm. The risk heat map includes a gap risk value identifier based on the flammability score. The gap region is divided into multiple gap sub-regions based on the gap risk value identifier distribution in the risk heat map.

4. The method according to claim 1, characterized in that, The steps for determining the inspection execution task sequence containing the newly added tasks corresponding to the gap sub-region that need to be filled, based on the digital twin model of the optimal takeover PTZ, specifically include: Based on the digital twin model of the optimal takeover PTZ, the filling angle covering the gap sub-region is determined, and a new task including the takeover task preset position is generated; Obtain the original inspection plan for the optimal takeover PTZ and determine the original risk value, priority, and time percentage of each sub-plan in the original inspection plan. By associating the time percentage with the original risk value, a risk-time mapping relationship is obtained; The gap risk value identifier corresponding to the gap sub-region is compared with each of the original risk values ​​to determine the new priority of the new task to be inserted into the original inspection plan; Based on the risk-time mapping relationship and the gap risk value identifier, determine the new time percentage of the new task in the total time corresponding to the original inspection plan; The sequence of inspection tasks is determined by combining the new priority and the new time percentage.

5. The method according to claim 1, characterized in that, The steps of determining candidate neighboring machines that match each of the aforementioned gap sub-regions, and determining the optimal takeover gimbal corresponding to each of the aforementioned gap sub-regions from the candidate neighboring machines, specifically include: From the candidate neighboring machines, exclude those neighboring machines in the target area that cannot cover the gap sub-region or whose overlap rate with the gap sub-region is lower than a preset overlap rate threshold, and obtain the remaining candidate neighboring machines; The coverage capability matching degree is obtained by comparing the theoretical maximum visible coverage area of ​​the remaining candidate neighboring machine to the total area of ​​the sub-region with the total area of ​​the sub-region, and by combining the line-of-sight occlusion model. After a new task is virtually inserted, the task interference degree is determined based on the reciprocal of the average inspection interval extension rate of the original high-priority tasks of the remaining candidate neighboring machines. The remaining candidate neighboring machines are virtually moved from their current positions to the pre-filled position, and the energy consumption cost is determined by the motor rotation and equipment power consumption generated during one cycle of the new task. The pre-filled position is determined by the gap sub-region. The response speed is determined based on the estimated time required for the remaining candidate neighboring machines to complete their current tasks and rotate to the first pre-set replacement position. The comprehensive score of each remaining candidate neighbor is obtained by using a pre-set weighted summation model based on one or more of the following factors: coverage capability matching degree, task interference degree, energy consumption cost, and response speed. The neighboring unit with the highest overall score is selected as the optimal takeover gimbal.

6. The method according to claim 1, characterized in that, Before the step of determining the digital twin model of each gimbal device within the target area, the method further includes: After the periodic inspection task is completed, control all normal PTZ devices in the target area to perform panoramic scanning and acquire multi-angle environmental image sequences; The environmental image sequence is processed to generate a three-dimensional mesh model of the target area; The three-dimensional mesh model is compared with the existing static data in the corresponding digital twin model to identify and quantify environmental changes; Based on the environmental changes, the static data in the digital twin model is automatically updated, including correcting the line-of-sight obstruction relationship between each PTZ device and each monitoring area.

7. The method according to claim 1, characterized in that, Once it is confirmed that the failed PTZ has returned to normal, the takeover exit process will be automatically triggered, including the following steps: Once the failed gimbal is detected to have returned to normal for the first time, the corresponding state will be marked as an observation period, and the optimal takeover gimbal will be controlled to continue to perform the replacement operation within a preset time. During the observation period, the dynamic data of the recovered PTZ is continuously monitored. If the corresponding continuous and stable operation time reaches the preset time, the failed PTZ is officially confirmed to have returned to normal and the takeover exit process is triggered.

8. A server, characterized in that, The server includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the server to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the server, the server causes the server to perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the server, the server performs the method as described in any one of claims 1-7.

Images