A gimbal failure inspection and replacement method based on neighboring machine cooperation and related equipment

Abstract

The application provides a gimbal failure inspection and replacement method based on neighboring machine cooperation and related equipment, and relates to the technical field of general control or regulation systems. The method comprises the following steps: first, determining the digital twin model of each gimbal in the target area to determine whether the target gimbal is failed; combining the digital twin model and the failure type to determine the gap area corresponding to the preset position of the failed gimbal, and dividing the gap area into multiple gap sub-areas with gap risk value labels; then, matching candidate neighboring machines for each gap sub-area and determining the optimal takeover gimbal; combining the digital twin model to generate an inspection execution task sequence containing a replacement task; controlling the optimal takeover gimbal to replace according to the sequence; after the failed gimbal returns to normal, triggering the takeover exit process and restoring the original inspection plan of the optimal takeover gimbal. The method can dynamically and intelligently schedule optimal resources to compensate for the monitoring blind area, minimize the interference with normal inspection tasks, and improve the robustness and reliability of the entire monitoring system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of general control or regulation systems, and particularly relates to a gimbal failure inspection and replacement method based on neighboring machine cooperation and related equipment. BACKGROUND

[0002] In a mountain forest fire prevention monitoring system, a gimbal monitoring device is a key infrastructure for fire warning and needs to perform inspection and monitoring on a designated area all day long. However, due to the harsh environment in the mountains, device aging, extreme weather and other factors, the gimbal device will inevitably fail. When a gimbal device fails completely or partially, the monitoring area originally responsible for it will form a monitoring blind area, increasing the risk of missing a fire.

[0003] The traditional response mainly relies on manual adjustment of the monitoring range of the neighboring gimbal to temporarily cover the monitoring blind area after the failure is found by artificial discovery, or dispatching maintenance personnel to handle it on site. Some servers have a basic automatic takeover function, that is, when a gimbal device is offline, a preset neighboring machine automatically turns to the core area of the failed gimbal device.

[0004] However, this method has the problems of long response time, low scheduling efficiency, and serious waste of resources. In particular, during the high-risk period, the existence of any monitoring blind area may cause a delay in the discovery of a fire, resulting in irreparable losses. At the same time, the simple and rough neighboring machine takeover method often disrupts the original and equally important inspection tasks of the neighboring machine, which may lead to the consequence of "repairing the west wall with the east wall", and reduces the overall monitoring efficiency. Therefore, the existing technology is difficult to quickly and accurately schedule neighboring gimbals for monitoring and replacement when the gimbal monitoring device fails, while ensuring no monitoring blind area, to minimize the interference with the original inspection tasks. SUMMARY

[0005] The present application provides a gimbal failure inspection and replacement method based on neighboring machine cooperation and related equipment, which is used to realize rapid, accurate and low-interference automatic filling of monitoring blind areas.

[0006] In a first aspect, the application provides a gimbal failure inspection and replacement method based on neighbor cooperation, applied to a server, the method comprising: determining digital twin models of each gimbal device in a target area, and determining whether the corresponding target gimbal device is failed based on the digital twin models; if it is determined that the target gimbal device is failed, determining a failure type; determining a gap area corresponding to a preset position of the failed gimbal according to the digital twin models and the failure type; dividing the gap area into a plurality of gap sub-areas with gap risk value labels; determining candidate neighbors matched with each gap sub-area, and determining optimal takeover gimbals corresponding to each gap sub-area from the candidate neighbors; determining an inspection execution task sequence containing new tasks for replacing the gap sub-areas according to the digital twin models of the optimal takeover gimbals; controlling the optimal takeover gimbals to perform replacement operations according to the inspection execution task sequence; and when it is determined that the failed gimbal is restored to normal, triggering an automatic takeover exit process to restore the original inspection plan of the optimal takeover gimbal.

[0007] By adopting the above technical solution, the precise mapping and real-time monitoring of the physical state and operating environment of the gimbal device are realized by constructing the digital twin models, thereby providing a high-fidelity information basis for subsequent decision-making. On this basis, when the device failure is detected, the server does not simply perform backup switching, but realizes a complete closed-loop automatic replacement process through a series of fine steps such as failure type judgment, gap area quantification, risk level division, optimal neighbor dynamic selection, and intelligent inspection task reorganization. The method can dynamically and intelligently dispatch optimal resources to compensate for the monitoring blind area according to the specific circumstances of the failure and the actual risks of the monitoring area, and automatically and smoothly exits after the device is restored, thereby minimizing the interference with normal inspection tasks while improving the robustness, reliability, and intelligent emergency handling capability of the entire monitoring system.

[0008] In some embodiments in combination with the first aspect, the step of determining the gap area corresponding to the preset position of the failed gimbal according to the digital twin models and the failure type specifically comprises: if the dynamic data in the digital twin model presents a device health degree of zero or continuous heartbeat loss, then determining that the failure type is total failure, and determining the area corresponding to all preset preset position requirements in the static data of the failed gimbal as the gap area; if the target preset position response is timed out, determining that the failure type is partial view failure, and determining the area corresponding to the target preset position as the gap area; and if the returned image has local abnormalities, then combining the effective picture boundary of the abnormal image with the preset coverage range of the target preset position to correct the gap area.

[0009] By adopting the technical solution, the method can finely distinguish the failure types, so that the determination of the gap area is more targeted and accurate. By distinguishing full failure, partial view failure and image local anomaly, the server can accurately define the actual range of the monitoring blind area according to the specific performance of the failure, instead of regarding the entire pan-tilt responsible area as a gap. This fine definition avoids excessive occupation of replacement resources, makes the subsequent replacement task more focused and efficient, significantly reduces the interference on the original tasks of neighboring machines caused by unnecessary large-scale replacement, and improves the resource utilization and response accuracy of the entire replacement strategy.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the step of dividing the gap area into a plurality of gap sub-areas with gap risk value labels specifically includes: obtaining a failure timestamp of the failed pan-tilt; obtaining an entire complete panoramic inspection video collected by the failed pan-tilt before the failure timestamp; performing frame-by-frame object detection on the complete panoramic inspection video based on a preset flammable material feature library, extracting the flammability of each object, and determining the flammable score value of all objects based on the flammability; generating a risk heat map of the gap area by a spatial interpolation algorithm in combination with the spatial coordinates of each object in the panoramic video and the flammable score value, wherein the risk heat map includes gap risk value labels based on the flammable score value; and dividing the gap area into a plurality of gap sub-areas according to the distribution of the gap risk value labels in the risk heat map.

[0011] By adopting the technical solution, by utilizing the pre-failure inspection video, the server can generate an intuitive risk heat map by performing object detection and quantifying the risk in combination with the preset flammable material feature library. This process not only identifies the gap area, but more importantly, it evaluates the potential danger level at different positions in the area. Dividing the gap area into sub-areas with different risk labels makes the subsequent replacement strategy no longer indiscriminate coverage, but risk-driven differentiated response, which prioritizes the monitoring continuity of high-risk sub-areas, so that limited takeover resources can be invested in the most needed places, improving the security value and effectiveness of emergency replacement.

[0012] In combination with some embodiments of the first aspect, in some embodiments, in combination with the digital twin model of the optimal takeover PTZ, the step of determining the inspection execution task sequence of the new task corresponding to the gap sub-region to be filled in comprises: determining a filling angle covering the gap sub-region based on the digital twin model of the optimal takeover PTZ, generating a new task including a takeover task preset position; obtaining an original inspection plan of the optimal takeover PTZ, and determining original risk values, priorities and time proportions of each sub-original plan in the original inspection plan; associating the time proportion with the original risk value to obtain a risk-time mapping relationship; comparing the gap risk value identifier corresponding to the gap sub-region with each original risk value to determine a new priority of the new task inserted into the original inspection plan; determining a new time proportion of the new task in the total time of the original inspection plan according to the risk-time mapping relationship and the gap risk value identifier; and determining the inspection execution task sequence in combination with the new priority and the new time proportion.

[0013] By adopting the above technical solution, instead of brutally interrupting or shelving the original inspection plan, the risk level of the new filling task is quantitatively compared with the original task by establishing a "risk-time" mapping relationship, so as to find a reasonable priority and time allocation for it in the new sequence. This way can ensure that the insertion of the new task is reasonable, which can meet the filling demand and maximize the retention of the inspection frequency of the original high-risk area. The finally generated inspection execution task sequence is an optimized solution that balances the original responsibility and the temporary emergency task, which ensures that the optimal takeover PTZ minimizes the loss of its own security value while performing the filling operation, and guarantees the overall stability and efficiency of the entire monitoring system.

[0014] In some embodiments of the first aspect, in some embodiments, the step of determining the optimal takeover gimbal corresponding to each of the gap sub-regions from the candidate neighboring machines matching each of the gap sub-regions specifically comprises: excluding from the candidate neighboring machines the neighboring machines in the target region that cannot cover the gap sub-regions or have an overlap rate with the gap sub-regions lower than a preset overlap rate threshold, to obtain remaining candidate neighboring machines; obtaining a coverage capacity matching degree according to the ratio of the theoretical maximum visible coverage area of the gap sub-regions to the total area of the sub-regions in the remaining candidate neighboring machines and combining the line-of-sight occlusion model for correction; determining a task interference degree based on the reciprocal of the average patrol interval extension rate of the original high-priority tasks of the remaining candidate neighboring machines after virtually inserting the new task; determining an energy consumption cost generated by the motor rotation and equipment power consumption of the remaining candidate neighboring machines virtually moving from the current position to the replacement preset position and executing the new task for one cycle, the replacement preset position being determined by the gap sub-regions; determining a response speed according to the estimated time required for the remaining candidate neighboring machines to complete the current task and rotate to the first replacement preset position; obtaining a comprehensive score of each of the remaining candidate neighboring machines according to one or more of the coverage capacity matching degree, the task interference degree, the energy consumption cost, and the response speed, and then through a preset weighted summation model; and selecting the neighboring machine with the highest comprehensive score as the optimal takeover gimbal.

[0015] By adopting the above technical solutions, the four core dimensions of coverage capacity, task interference degree, energy consumption cost, and response speed are comprehensively considered. By comprehensively evaluating each candidate neighboring machine and using a weighted summation model to obtain a comprehensive score, the device with the highest "performance-price ratio" for performing the replacement task under the current conditions can be objectively selected. This comprehensive decision-making mechanism ensures that the selected optimal takeover gimbal not only "sees" the gap, but also completes the task with the lowest system comprehensive cost, the fastest speed, and the least interference, thereby optimizing the efficiency and sustainability of the entire replacement action.

[0016] In some embodiments of the first aspect, before the step of determining the digital twin model of each gimbal device in the target region, the method further comprises: periodically controlling all normal gimbal devices in the target region to perform panoramic scanning after the completion of a patrol task, to obtain a multi-angle environment image sequence; processing the environment image sequence to generate a three-dimensional grid model of the target region; comparing the three-dimensional grid model with the existing static data in the respective corresponding digital twin model to identify and quantify environmental changes; and automatically updating the static data in the digital twin model according to the environmental changes, the updating including correcting the line-of-sight occlusion relationship of each gimbal device to each monitoring region.

[0017] By adopting the technical solutions, the three-dimensional environment model is periodically scanned and reconstructed, and then compared and updated with the data in the digital twin model, so that the decision basis relied on by the system can be kept synchronized with the changes in the physical world. In particular, the automatic correction of the line-of-sight occlusion relationship directly affects the accuracy of the neighboring machine coverage capability evaluation and is crucial for the subsequent selection of the optimal takeover gimbal.

[0018] In some embodiments of the first aspect, when it is determined that the failed gimbal has returned to normal, the takeover exit process is automatically triggered, specifically including: when the failed gimbal returns to normal signal is first detected, the corresponding state is marked as an observation period, and the optimal takeover gimbal is controlled to continue performing the repositioning operation within a preset time period; during the observation period, the dynamic data of the recovered gimbal is continuously monitored, and if the time period of continuous stable operation reaches the preset time period, it is formally confirmed that the failed gimbal has returned to normal, and the takeover exit process is triggered.

[0019] By adopting the technical solutions, the monitoring "flash-off" problem caused by unstable device state is effectively avoided. By setting an "observation period", the server does not immediately terminate the repositioning operation after detecting the failed gimbal recovery signal for the first time, but continuously monitors the stability. This "confirmation and release" strategy ensures that the takeover exit process will only be triggered when the failed gimbal is confirmed to be able to operate stably and normally. This improves the fault tolerance of the system and ensures the continuity and stability of the monitoring coverage, preventing the monitoring area from being repeatedly lost due to frequent start-stop of the device in the early stage of failure recovery.

[0020] In a second aspect, the present application provides a server, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code comprising computer instructions, the one or more processors invoking the computer instructions to enable the server to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0021] In a third aspect, the present application provides a computer readable storage medium comprising instructions that, when executed on a server, cause the server to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a fourth aspect, the present application provides a computer program product that, when executed on a server, causes the server to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0024] 1. The application adopts the construction of a digital twin model for state perception, and integrates failure diagnosis, risk assessment, optimal neighbor selection and intelligent task reorganization into a closed-loop automatic replacement decision-making technology, effectively solving the technical problems of slow response, low efficiency and poor replacement effect caused by relying on manual intervention or using fixed replacement strategies after the PTZ fails, and achieving unmanned, intelligent and optimized emergency response and automatic recovery of the monitoring system failure, thereby improving the overall robustness and operation efficiency of the system.

[0025] 2. The application adopts a technology based on historical video for target risk identification, and generates a risk heat map through a spatial interpolation algorithm to quantify and divide the gap area, effectively solving the technical problems of low resource utilization efficiency and low security value caused by treating the monitoring gap as a homogeneous area for indiscriminate replacement after determining the monitoring gap, and achieving fine risk level assessment of the monitoring gap, changing the replacement strategy from "indiscriminate coverage" to "risk-driven differentiated response", and preferentially investing limited takeover resources in the most critical areas, thereby significantly improving the security precision and effectiveness of emergency replacement.

[0026] 3. The application adopts an intelligent task sequence reorganization technology that establishes a "risk-time" mapping relationship and quantitatively compares the new task risk with the original task risk to dynamically determine the priority and time proportion of the new task, effectively solving the technical problems of rough task insertion and lack of quantitative trade-off between new and old tasks when scheduling neighbor replacement, which can easily cause "trade-off", and achieving dynamic and quantitative integration and intelligent scheduling of new replacement tasks and original inspection tasks, minimizing the impact on neighbor normal work while filling the monitoring gap, thereby ensuring the overall stability of the monitoring system and the comprehensive execution efficiency of the inspection tasks. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flowchart of the PTZ failure inspection replacement method based on neighbor cooperation in the embodiments of the application;

[0028] Figure 2 is another flowchart of the PTZ failure inspection replacement method based on neighbor cooperation in the embodiments of the application;

[0029] Figure 3 is a schematic diagram of an entity device structure of a server in the embodiments of the application. DETAILED DESCRIPTION

[0030] The terminology used in the following embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used in this application, refers to and encompasses any or all possible combinations of one or more of the associated listed items.

[0031] Hereinafter, the terms "first", "second" are only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise specified.

[0032] For the convenience of understanding, the method provided by the present embodiment is described in the flow. Please refer to Figure 1 , a flowchart of a pan-tilt failure inspection and replacement method based on neighboring machine cooperation in the embodiments of the present application.

[0033] S101, determine the digital twin model of each pan-tilt device in the target area, and determine whether the corresponding target pan-tilt device is failed based on the digital twin model;

[0034] Wherein, the target area refers to a specific geographical range or spatial area that needs to be inspected and monitored, for example, an industrial park, a residential area or a large factory area, etc., which is the physical space boundary of the deployment of pan-tilt devices and the execution of inspection tasks; the pan-tilt device refers to a monitoring device equipped with a rotatable and tilting lens, which can realize monitoring coverage in different directions and ranges by adjusting the angle, and is widely used in security and production monitoring scenes; the digital twin model refers to a virtual model that is completely corresponding to the physical pan-tilt device and is mapped in real time through digital technology, which contains the static attributes of the device (such as position, model, hardware parameters, preset position information, etc.) and dynamic data (such as running state, health degree, real-time collected image data, heartbeat signal, etc.), used to simulate, monitor and analyze the behavior and state of the physical device; the target pan-tilt device refers to one or more pan-tilt devices that are focused on in the current detection and analysis process; failure refers to the failure of the pan-tilt device to normally execute its preset inspection or monitoring function, which may manifest as the device being completely unable to work, partial function abnormality or the collected data not meeting the requirements, etc.

[0035] In this application, the PTZ device in a region can transmit data to a common server, which is equivalent to the brain of the entire PTZ system, responsible for calculation, decision-making and instruction issuance. After the server starts, it can be executed regularly according to the preset period (such as once every second), the purpose is to master the running state of all PTZ devices in the target area in real time, and provide basis for subsequent failure processing and replacement operation. First, the digital twin model of each PTZ device in the target area is determined, which requires the server to pre-construct a digital twin for each PTZ device deployed in the target area. During the construction process, the static information of the PTZ device will be collected, including the installation position coordinates (such as latitude and longitude, relative height) of the device, hardware model, lens parameters (focal length, field of view), the number of supported preset positions and the angle parameters of each preset position, etc., and these information will be stored as static data of the digital twin model. At the same time, the server will obtain the dynamic running data of the PTZ device in real time through sensors, network communication and other ways, such as CPU usage, memory occupation, motor running state, lens working temperature, real-time returned monitoring image frame, heartbeat signal (such as periodic state confirmation packet) between the device and the server, device health score (calculated based on hardware state, running stability, etc.) and other factors. These dynamic data will be updated to the digital twin model in real time, ensuring that the virtual model and the physical device keep consistent.

[0036] After completing the construction and real-time updating of the digital twin model, the server judges whether the target PTZ device is failed based on the model. The judgment process is mainly realized by analyzing the dynamic data in the digital twin model: first, monitor the device health, if the device health value recorded in the digital twin model drops to zero, it means that the device hardware may have a serious fault and cannot continue to run, which is judged as failure; second, check the heartbeat signal, if the server does not receive the heartbeat signal sent by the target PTZ device within the preset time window (such as 10 seconds), and multiple attempts to communicate have no response, it is considered that the connection between the device and the server is interrupted, and problems such as power failure, network failure or device downtime may occur, which is judged as failure; third, analyze image data, if the target PTZ device continuously returns blank images, fuzzy images or abnormal images with large differences from the preset scene, and after excluding temporary factors such as lens obstruction, it still cannot recover to normal, it is judged as failure; fourth, detect function response, if the server sends a control instruction (such as adjusting to a certain preset position, zooming, etc.) to the target PTZ device, and does not receive the execution feedback of the device within the specified time (such as 5 seconds), or the feedback execution result does not meet the requirements of the instruction, and the problem still exists after multiple attempts, it is judged as device function failure. Through the comprehensive monitoring and analysis of the above multiple indicators, the server can accurately judge whether the target PTZ device is in a failure state, and provide accurate trigger conditions for the subsequent processing flow.

[0037] In some embodiments, before this step, the static data of the digital twin model can be updated periodically after all normal PTZ devices in the target area complete each periodic routine inspection task, to ensure high consistency between the virtual model and the physical environment.

[0038] The server sends a scanning instruction to all normal PTZ devices in the target area at a preset period (e.g., 23:00 every day), which contains preset shooting parameters. The PTZ devices execute the instruction according to the parameters, pause shooting at a preset angle for each horizontal rotation, and shoot at intervals at each horizontal angle, generating a sequence of “horizontal + vertical” multi-angle images. All PTZ images are combined into a full-coverage image dataset of the region. The server imports the image sequence into a three-dimensional reconstruction server: first, pre-processing, then image registration, determining the relative pose of the images through feature point matching; then generating a dense point cloud, estimating the depth of the space points; finally, using the Poisson reconstruction algorithm to convert the point cloud into a three-dimensional mesh model with texture, clearly presenting the spatial state of objects in the region. The server extracts historical static data from the digital twin model, aligns the new three-dimensional mesh model with the old model using the ICP algorithm, calculates the deviation of the object surface points, and determines that there is a change if the deviation is greater than 0.5 meters. At the same time, use the ray tracing algorithm to simulate the line of sight from the PTZ to the monitoring area, calculate the occlusion rate (e.g., from 20% to 80%), and quantify the change in occlusion relationship. The server writes the quantified changes to the static database: adds the coordinates, size, and type of new objects; updates the corresponding fields if the objects are moved or resized; and deletes the records if the objects are removed. Focus on correcting the line-of-sight occlusion table, updating the occlusion status, occlusion object, and occlusion rate of “PTZ-monitoring area” according to the new occlusion rate, and recalculating and updating the effective monitoring range of the PTZ.

[0039] Through periodic panoramic scanning and three-dimensional reconstruction, changes in the environment such as device displacement, tree growth, and new facilities in the target area can be captured in real time, and the changes are quantified and updated to the static data of the digital twin model, avoiding the disconnection between the virtual model and the physical world. This provides reliable data support for “accurately judging the coverage ability of neighboring devices” and “correcting the impact of line-of-sight occlusion” when the PTZ fails to be replaced, solving the problem of bias in replacement decision-making caused by environmental changes in traditional models.

[0040] S102、If it is determined that the target PTZ device is failed, determine the type of failure;

[0041] The purpose of this step is to determine the specific type of device failure, as different types of failure will result in different monitoring coverage gaps and require different replacement strategies. In practical applications, PTZ device failures are not a single mode, but rather multiple scenarios, so a detailed analysis is needed to determine the specific type of failure.

[0042] First, regarding the determination of total failure, when the dynamic data in the digital twin model shows a device health level of zero, it indicates that the core hardware of the device (such as the motherboard, motor, lens, etc.) may have been completely damaged, causing the device to be completely inoperable; or when the device continuously loses heartbeat signals for more than a preset threshold (such as 30 seconds), and the connection cannot be restored after multiple communication retries and troubleshooting (such as checking network lines, power supply, etc.), it is also determined to be a total failure. Total failure means that the monitoring areas corresponding to all preset positions of the PTZ device will lose coverage, requiring comprehensive backup support.

[0043] Secondly, the determination of partial viewpoint failure mainly targets functional abnormalities of specific preset positions. When the server sends a command to the target PTZ device to switch to a specific preset position, if no feedback is received within the specified response time (e.g., 8 seconds) indicating that the device has successfully reached the preset position, or if the image returned by the device shows that it has not reached the specified viewpoint, then the viewpoint corresponding to that preset position is determined to be faulty. This type of failure is usually caused by reasons such as motor failure, incorrect preset position parameters, or mechanical jamming, and only affects the monitoring coverage of the specific preset position; other preset positions may still function normally.

[0044] Finally, for cases where the returned image contains local anomalies, image analysis techniques are needed for assessment. Local anomalies may include blurred areas, occlusion, color distortion, or the presence of fixed noise. The server will compare the abnormal image with the preset normal coverage area and image features for that location. Using algorithms such as edge detection and feature matching, it will determine the boundaries of the valid image. Areas outside the valid image boundaries, or areas within the boundaries but whose image quality does not meet monitoring requirements, will be corrected as gaps. This type of failure is usually caused by lens damage, lens contamination, or sudden changes in lighting conditions, affecting only a portion of the monitoring image at a specific preset location.

[0045] Through the above analysis, the server can accurately determine the type of failure of the target PTZ device, laying the foundation for subsequent targeted identification of the gap area and ensuring the accuracy and effectiveness of the replacement operation.

[0046] S103. Determine the notch area corresponding to the preset position of the failed gimbal based on the digital twin model and the failure type;

[0047] Among them, a failed PTZ refers to a PTZ device that has been determined to be ineffective; a preset position refers to a fixed monitoring angle and range that the PTZ device is set in advance. Each preset position corresponds to a specific monitoring area. The device can quickly switch preset positions to inspect different areas; a gap area refers to an area where the monitoring coverage is missing due to the failure of the PTZ device. It is a target area that needs to be filled by other neighboring devices.

[0048] Different failure categories correspond to different ways of determining the gap area, which requires specific analysis combined with relevant data in the digital twin model.

[0049] When the failure category is full failure, it means that the gimbal device has completely failed to work, and all the preset positions corresponding to the monitoring area will lose coverage. At this time, the server will call the static data of the failed gimbal in the digital twin model, which stores detailed information of all preset positions, including the monitoring angle corresponding to each preset position, the geographical coordinate boundary of the coverage range, the monitoring key area, etc. The server integrates all the areas required to be covered by these preset positions, and after removing the overlapping parts, the overall area obtained is the gap area under the condition of full failure. For example, if a failed gimbal has 3 preset positions, covering the east corner, west corner and central area of workshop A respectively, the sum of these three areas (after removing the overlapping parts) is the gap area of the failed gimbal.

[0050] When the failure category is partial angle failure, that is, one or more specific preset positions cannot respond normally, resulting in the loss of coverage of the corresponding monitoring area. At this time, the server will directly determine the preset coverage area corresponding to the target preset position (i.e. the failed preset position) as the gap area according to the information recorded in the digital twin model. For example, if preset position 2 of a gimbal device cannot respond due to motor failure, and this preset position is originally responsible for covering the entrance area of the warehouse, then the entrance area of the warehouse becomes the gap area. At the same time, the server will check whether other preset positions are working normally to ensure that only the area corresponding to the failed preset position is marked as a gap to avoid misjudgment.

[0051] When the failure category is that the returned image has local abnormalities, the processing method is relatively complex. The server first obtains the abnormal image returned by the preset position, determines the boundary of the effective picture in the image through image analysis algorithms (such as edge detection, feature extraction, etc.), which is the area range that can clearly identify the monitoring object. Then, compare the effective picture boundary with the preset coverage range of the target preset position in the digital twin model. The preset coverage range is the area that should be covered by the preset position when it is working normally, usually stored in the form of geographical coordinates or pixel coordinates in the static data. For the areas within the preset coverage range but not within the effective picture boundary, and the areas within the effective picture boundary but with image quality not meeting the monitoring requirements (such as blur, severe obstruction), the server will correct them as gap areas. For example, the preset coverage range of a preset position is the entire parking lot, but the returned image is only half clear and the other half is blurred due to lens damage, then the blurred half will be determined as a gap area.

[0052] Through the above processing mode for different failure types, the server can accurately and comprehensively determine the gap area corresponding to the preset position of the failed pan-tilt, ensuring that the subsequent gap filling operation can be targeted and cover all missing monitoring areas.

[0053] S104, divide the gap area into a plurality of gap sub-areas with gap risk value identifiers;

[0054] Wherein, the gap sub-area refers to a smaller area unit obtained by dividing the gap area according to certain rules, and each sub-area has relatively independent monitoring requirements and risk characteristics; the gap risk value identifier refers to a numerical value or level identifier used to represent the monitoring importance or potential risk degree of each gap sub-area. The higher the risk value, the higher the priority of monitoring the sub-area.

[0055] In the forest fire prevention scene, it is crucial to accurately assess the fire risk distribution of the gap area, because the differences in the types, quantities and environmental conditions of combustible materials in different areas will lead to significant differences in the possibility of fire occurrence and spread, and it is necessary to prioritize monitoring coverage in high-risk areas.

[0056] Specifically, the server will receive the running state data sent by each pan-tilt device in real time, and when it detects that a pan-tilt device has failure signs through the digital twin model, it will immediately record the specific time point when the pan-tilt device changes from a normal state to a failure state, i.e. the failure timestamp. Only accurate timestamps can ensure that the latest video data before failure is obtained, which can truly reflect the situation of the gap area.

[0057] The server will store the video data collected by each pan-tilt device during daily inspection, and these videos will be classified and archived in chronological order. After obtaining the failure timestamp, the server will retrieve the video taken by the failed pan-tilt during the last complete panoramic inspection before failure according to the timestamp. Here, "complete panoramic inspection" means that the pan-tilt performs a comprehensive shooting of the entire monitoring area according to the preset inspection route and angle, ensuring that no monitoring angle is missed. The video duration is usually consistent with the inspection period of the pan-tilt (e.g. 1 hour, 2 hours, etc.). For example, the inspection period of pan-tilt A is 1 hour, it starts panoramic inspection at the top of each hour every day, and the failure timestamp is 14:30:25. Then the server will retrieve and obtain the panoramic inspection video taken by pan-tilt A during 13:00-14:00, which covers the entire monitoring area of pan-tilt A. This video is a complete panoramic inspection video before failure. If the failure timestamp is close to the end time of the last complete panoramic inspection (e.g. failure timestamp is 14:10), in order to ensure that the obtained video can fully reflect the situation of the gap area, the server will also obtain the last complete panoramic inspection video (13:00-14:00), instead of selecting the incomplete video from 14:00-14:10.

[0058] Then, the server will call the preset flammable material feature layout library, which is built before the server is deployed, according to the common flammable material types in the mountains (such as dry trees, shrubs, weeds, leaf litter, flammable crop straw, and temporary stacked wood, etc.), through image collection, feature extraction, and labeling, etc. The library not only contains the visual features of various flammable materials (such as color, shape, texture, etc.), but also contains the flammable characteristics information of various flammable materials (such as ignition point, burning speed, and spreading ability after burning, etc.). When performing frame-by-frame object detection, the server will use advanced computer vision algorithms, such as the YOLO (You Only Look Once) algorithm based on deep learning, to process each frame of the complete panoramic inspection video. For each frame of image, the algorithm will first preprocess the image (such as denoising, image enhancement, etc.), and then compare the processed image with the flammable material features in the preset flammable material feature layout library, identify the various objects present in the image, and determine which objects belong to flammable materials, as well as the specific location and type of each flammable material.

[0059] After identifying the flammable materials, the server will extract the flammability of each flammable material according to the flammable characteristics information of various flammable materials stored in the preset flammable material feature layout library. For example, the library records that dry grass has a low ignition point (about 200-300°C), a fast burning speed, and is easy to spread after burning, so its flammability is high; while fresh evergreen leaves contain more water, have a high ignition point (about 400-500°C), burn slowly, and are not easy to spread, so their flammability is low. Then, the server will convert the extracted flammability into a specific flammable score value according to the preset flammable scoring rules. The scoring rules usually divide the flammability into several levels, each level corresponds to a score range, for example, the flammability is divided into five levels: very high, high, medium, low, and very low, corresponding to 90-100 points, 70-89 points, 50-69 points, 30-49 points, and 0-29 points. When determining the score, the server will make fine adjustments based on the specific circumstances of the flammable material, for example, if the dry grass in a certain area has a high density and thick accumulation, its flammability will be higher than that of dry grass with a small density and thin accumulation, and the corresponding flammable score value will also be higher, possibly from 80 points to 85 points; if a dry tree has a large diameter, although its ignition point is not high, the heat released during burning is more, which may cause the fire to spread faster, so its flammable score value will also be appropriately increased. In this way, the server will assign an accurate flammable score value to all flammable materials identified in each frame of the video.

[0060] The spatial coordinates of each object in the panoramic video need to be determined, which requires combining the self-positioning information of the gimbal device (such as the GPS coordinates of the gimbal installation position), the shooting angle parameters of the gimbal (such as the horizontal rotation angle and the vertical pitch angle), and the lens parameters (such as the focal length and the field of view angle), and converting the two-dimensional coordinates (pixel coordinates) of the object in the image into three-dimensional coordinates (such as longitude, latitude, and altitude) in the real physical space through a coordinate conversion algorithm. For example, given that the GPS coordinates of the gimbal installation position are (East 120.5°, North 30.2°, and altitude 500m), the horizontal rotation angle is 30°, the vertical pitch angle is -10°, and the lens focal length is 10mm when shooting a dry grass area, the coordinates of the dry grass area in the real space can be calculated as (East 120.501°, North 30.202°, and altitude 495m) through the coordinate conversion algorithm.

[0061] After obtaining the spatial coordinates of each object and the corresponding flammable score value, the server will use a spatial interpolation algorithm to estimate the flammable risk value of other objects not detected in the gap area. Since only obvious objects in the image can be identified during the frame-by-frame object detection process, the flammable risk value of the blank area between objects or the area not covered by objects cannot be directly obtained, so a spatial interpolation algorithm is needed to estimate it. Common spatial interpolation algorithms include inverse distance weighted interpolation and Kriging interpolation. Taking the inverse distance weighted interpolation as an example, the basic principle of this algorithm is that the attribute value (here, the flammable risk value) of a certain unknown point is related to the attribute values of the surrounding known points, and the correlation decreases with increasing distance. Specifically, for a certain unknown point P in the gap area, the server will search for all known points within a certain range (such as 100 meters) around the point, calculate the distance from these known points to the unknown point P, and then calculate the weighted sum of the flammable score values of each known point according to the inverse distance as the weight, to obtain the flammable risk value of the unknown point P. For example, there are three known points A, B, and C around the unknown point P, with flammable score values of 80, 70, and 60 respectively, and distances to point P of 10 meters, 20 meters, and 30 meters respectively. The weights of each known point are 1 / 10, 1 / 20, and 1 / 30 respectively, and the flammable risk value of the unknown point P is (80 x 1 / 10 + 70 x 1 / 20 + 60 x 1 / 30) ÷ (1 / 10 + 1 / 20 + 1 / 30) ≈ 72.3 points.

[0062] After calculating the flammable risk values of all positions in the gap area through the spatial interpolation algorithm, the server will visualize these risk values in the form of a risk heat map. The risk heat map usually uses a color gradient to represent the risk values, for example, using red, orange, yellow, green, and blue colors to represent extremely high risk (80-100 points), high risk (60-79 points), medium risk (40-59 points), low risk (20-39 points), and extremely low risk (0-19 points). At the same time, the specific risk values of each position, i.e., the gap risk value markers, will be marked on the risk heat map, such as "Risk value: 92" in the red area and "Risk value: 75" in the orange area, to more intuitively and accurately understand the risk distribution in the gap area.

[0063] Finally, according to the distribution of the gap risk value markers in the risk heat map, the server will divide the entire gap area into multiple independent gap sub-areas according to the pre-set risk level division standard. The division standard is usually formulated in combination with the actual needs and experience of forest fire prevention in mountainous areas, for example, dividing the area with a risk value of 80-100 points into a high-risk gap sub-area, which is a high-risk area for fire and needs to be monitored and supplemented first; dividing the area with a risk value of 50-79 points into a medium-risk gap sub-area, which needs to be supplemented second; and dividing the area with a risk value of 0-49 points into a low-risk gap sub-area, which can appropriately reduce the priority of supplementation.

[0064] Optionally, the complete panoramic video can be subjected to frame-by-frame object detection and flammable scoring. The server calls an object detection model based on the YOLOv8 algorithm, which has been trained in advance through a large number of mountain flammable material images and flammable degree labeled samples, and can accurately identify more than 10 common flammable materials such as dry trees and grass. During detection, the video is extracted frame by frame, the resolution of each frame image is adjusted to 640x640 pixels, and batch processing is used, with 32 frames of images processed each time. After detection, the flammable materials in each frame image are classified and counted, and an initial score is assigned to each flammable material according to a pre-set flammable material scoring table (e.g., dry grass 85 points, dry shrubs 90 points, fresh trees 30 points, etc.). Then, the area size (the larger the area, the higher the score, e.g., dry grass with an area of more than 100x100 pixels increases the score by 5 points) and density (the higher the density, the higher the score, e.g., more than 10 dry shrubs per square meter increases the score by 8 points) of the flammable materials in the image are adjusted to obtain the final flammable score value. Optionally, if the gimbal device has infrared shooting function, the visible light image and the infrared image can be combined for flammable material detection. The infrared image can more clearly show the temperature distribution of the object, which helps to identify flammable materials hidden under vegetation (such as the underground leaf layer), further improving the detection accuracy.

[0065] S105. Determine the candidate neighboring machine that matches each of the gap sub-regions, and determine the optimal takeover gimbal corresponding to each of the gap sub-regions from the candidate neighboring machine;

[0066] Among them, candidate neighboring devices refer to other normally operating PTZ devices within the target area that may have the capability to cover the missing sub-area, and are potential fillers; the optimal takeover PTZ refers to the PTZ selected from the candidate neighboring devices whose comprehensive performance is most suitable for the corresponding missing sub-area filler needs, and must meet the optimal balance in terms of coverage capability, task interference, energy consumption cost and response speed.

[0067] After identifying candidate neighboring devices, multi-dimensional screening and evaluation are needed to find the best-performing replacement device for the missing sub-region, ensuring that the replacement task is completed efficiently, with low interference and low cost.

[0068] The initial screening of candidate neighboring devices is performed. Based on the digital twin model of each candidate neighboring device, the server analyzes its monitoring parameters (such as rotation angle range, lens focal length, installation height, etc.), calculates the coverage area of ​​each neighboring device for the gap sub-region, and compares this coverage with the total area of ​​the sub-region to obtain the overlap rate. If a neighboring device cannot cover the gap sub-region at all, or its overlap rate is lower than a preset overlap rate threshold (e.g., 20%), it is excluded, and the remaining neighboring devices become the remaining candidate neighboring devices. This step quickly eliminates obviously unsuitable devices, reducing the computational workload of subsequent evaluations.

[0069] Next, the coverage capability matching degree is calculated. First, the theoretical maximum visible coverage area (the area that can cover the gap sub-region assuming no obstructions) is calculated based on the parameters of the remaining candidate neighboring units, and the ratio of this area to the total area of ​​the sub-region is used to obtain the initial ratio. Then, the line-of-sight occlusion model is invoked, and the 3D environmental data of the target area (such as building height, tree positions, etc.) and the installation position and angle parameters of the neighboring units are input to simulate the obstruction of the line of sight by obstructions in the actual scene, and the initial ratio is corrected. For example, if a neighboring unit has a theoretical coverage ratio of 80%, but a building obstructs 30% of the line of sight, the corrected coverage capability matching degree is 56%. This indicator directly reflects the actual monitoring capability of the neighboring unit for the gap sub-region and is one of the core dimensions of the evaluation.

[0070] Then, the task interference level is evaluated. The server virtually inserts a replacement task into the original task plan of the remaining candidate neighboring machines using a digital twin model, calculating the change in the inspection interval of the original high-priority tasks (such as key area inspections). For example, assuming the original inspection interval for task A is 30 minutes, after inserting the replacement task, the interval is extended to 45 minutes, an extension rate of (45-30) / 30 = 50%. The average extension rate of all original high-priority tasks is taken, and its reciprocal is used as the task interference level (e.g., if the average extension rate is 50%, the interference level is 1 / 0.5 = 2). The higher the interference level, the smaller the impact of the replacement task on the original important tasks, and the more suitable the neighboring machine is to take over the replacement.

[0071] Recalculate the energy consumption cost. The server calculates the angle and distance that the PTZ needs to rotate according to the current position of the remaining candidate neighbors (dynamic data from the digital twin model) and the reposition preset position (determined according to the gap sub-area coordinates and neighbor parameters), and estimates the energy consumption in the moving process combined with the motor power parameters; at the same time, according to the estimated execution time of the reposition task (such as 5 minutes for each inspection) and the unit time power consumption of the equipment, the energy consumption in the task execution stage is calculated. The sum of the two is the energy consumption cost of the neighbor executing a periodic reposition task. For example, the moving energy consumption is 0.2 degrees, the execution energy consumption is 0.3 degrees, and the total energy consumption cost is 0.5 degrees. In the scene of limited energy or energy saving, the weight of this indicator will be increased accordingly.

[0072] Then determine the response speed. The server analyzes the current task status of the remaining candidate neighbors: if the neighbor is executing a task, first calculate the remaining time required to complete the current task; then estimate the time required for rotation according to the rotation angle from the current position to the reposition preset position and the maximum rotation speed of the PTZ. The sum of the two is the estimated time of the response speed. For example, it still needs 2 minutes to complete the current task, and it needs 1 minute to rotate to the reposition preset position, so the response speed is 3 minutes. For high-risk areas such as high-risk areas, neighbors with faster response speed have more advantages.

[0073] Next, calculate the comprehensive score through the weighted sum model. First, standardize the coverage ability matching degree (0-100 points), task interference degree (0-10 points), energy consumption cost (reverse scoring, 0-100 points, the lower the energy consumption, the higher the score), and response speed (reverse scoring, 0-100 points, the shorter the time, the higher the score). Then allocate weights according to actual needs, for example, in the security-first scene of an industrial park, the coverage ability matching degree weight is set to 40%, the response speed is 30%, the task interference degree is 20%, and the energy consumption cost is 10%. Finally, calculate the weighted total score of each remaining candidate neighbor: comprehensive score = coverage ability matching degree x 40% + task interference degree x 20% + energy consumption cost x 10% + response speed x 30%.

[0074] Finally, select the remaining candidate neighbor with the highest comprehensive score as the optimal takeover PTZ. If the scores are the same, secondary screening can be performed according to the preset rules (such as preferentially selecting neighbors with faster response speed) to ensure that the final selected PTZ achieves the best balance between coverage effect, task interference, energy consumption, and response speed.

[0075] Optionally, the priority of the indicators is set (such as coverage ability > response speed > task interference > energy consumption cost), the remaining candidate neighbors are first sorted by coverage ability matching degree from high to low, and the top 50% of the neighbors are selected; then, the remaining neighbors are sorted by response speed from fast to slow, and the top 50% are selected; then, the remaining neighbors are sorted by task interference from high to low, and the top 50% are selected; finally, the energy consumption cost of the remaining neighbors is calculated, and the one with the lowest energy consumption is selected as the optimal takeover gimbal. For example, the top 5 in coverage ability are selected from 10 neighbors, then the top 3 in response speed are selected from them, then the top 2 in task interference are selected, and finally the one with the lowest energy consumption is selected from the 2 as the optimal. It can be understood that other ways can also be used to determine the optimal takeover gimbal, such as introducing a machine learning model to predict the filling effect of each neighbor and score, etc., which is not limited here. It should be noted that the preset overlap rate threshold and the weight of each indicator can be dynamically adjusted according to the actual situation of the target area (such as environmental complexity and the urgency of the inspection task) to improve the flexibility and accuracy of the screening.

[0076] S106, determine the inspection execution task sequence containing the newly added task for filling the gap sub-area in combination with the digital twin model of the optimal takeover gimbal;

[0077] After determining the optimal takeover gimbal corresponding to each gap sub-area, the newly added filling task can be reasonably integrated into the original inspection plan of the optimal takeover gimbal to form a complete, efficient and feasible task sequence, so as to ensure that the gap sub-area can be inspected in time and at the same time minimize the interference with the original inspection work of the optimal takeover gimbal. The specific process of this step will be described in detail in steps S201-S206, which will not be repeated here.

[0078] S107, control the optimal takeover gimbal to perform filling operation according to the inspection execution task sequence;

[0079] The server sends the generated inspection execution task sequence to the control module of the optimal takeover gimbal. The control module is a component on the optimal takeover gimbal responsible for receiving instructions and controlling the operation of the device, which can parse the information in the task sequence, including the execution time, target position (preset position), inspection duration, shooting parameters, etc. of each task. Then, the control module of the optimal takeover gimbal executes each task in turn according to the order of the task sequence. When executing the newly added filling task, the control module controls the motor of the gimbal to rotate according to the takeover task preset position specified in the task, adjusts the horizontal and vertical angles of the gimbal, so that the camera can accurately aim at the gap sub-area. At the same time, the focal length, exposure, etc. of the camera are set according to the task requirements to ensure that clear and effective images or videos are captured.

[0080] During execution, the optimal takeover gimbal will feed its running state back to the server in real time through its digital twin model. The server monitors the gimbal's position, angle, running speed, energy consumption, and other dynamic data in real time through the digital twin model to ensure that the gimbal performs normally according to the task sequence.

[0081] S108, when it is determined that the failed gimbal has recovered to normal, the takeover exit process is automatically triggered to restore the original inspection plan of the optimal takeover gimbal.

[0082] After the failed gimbal recovers from the failed state to the normal working state, the purpose is to terminate the position-filling task of the optimal takeover gimbal in time, so that it returns to the normal inspection working state, avoids resource waste and task repetition, and ensures the efficient and orderly operation of the entire inspection server.

[0083] First, the server needs to continuously monitor the status of the failed gimbal to determine whether it has recovered to normal. The monitoring methods include receiving the heartbeat signal, device health data, self-check report, etc. sent by the failed gimbal. When the server first detects the recovery signal sent by the failed gimbal, it will not immediately confirm that it has completely recovered, but will mark the status of the gimbal as an observation period. The observation period is set to avoid frequent switching of takeover exit and re-takeover due to the temporary recovery and re-failure of the gimbal, and to ensure the stability of the server. The length of the observation period can be preset according to actual conditions, such as 30 minutes, 1 hour, etc.

[0084] During the observation period, the server will control the optimal takeover gimbal to continue performing the position-filling operation to ensure that the gap sub-area can still be effectively inspected during this period. At the same time, the server will continuously and closely monitor the dynamic data of the recovered gimbal, including but not limited to the running parameters of the device (such as motor speed, camera focus adjustment accuracy), data transmission stability, inspection image quality, heartbeat signal continuity, etc. Through the analysis of these data, it is determined whether the gimbal can continue to run stably. If the recovered gimbal has been running stably for a length of time that reaches the preset observation period length during the observation period, and no abnormal conditions (such as loss of heartbeat signal, interruption of image transmission, decrease in device health, etc.) have occurred, the server will officially confirm that the failed gimbal has recovered to normal. At this time, the server will automatically trigger the takeover exit process.

[0085] The takeover exit process mainly includes the following steps: first, the server sends a stop filling task instruction to the optimal takeover PTZ, informing it to stop inspecting the gap sub-area; second, the optimal takeover PTZ completes the currently executing filling task (if any) after receiving the instruction, to avoid information loss caused by interruption of the task execution halfway; third, the optimal takeover PTZ adjusts from the filling preset position to the preset position corresponding to the next task in the original inspection plan, to prepare for resuming the original inspection work; and fourth, the server switches the inspection task sequence of the optimal takeover PTZ back to its original inspection plan, and deletes the added filling task, to ensure that the PTZ executes the inspection task according to the original arrangement.

[0086] In the embodiments of the present application, since the digital twin model is constructed to perceive the running state of the entire monitoring system in real time, and failure diagnosis, risk assessment, optimal neighbor selection, and intelligent task reorganization are integrated, a closed-loop intelligent decision-making process from accurate fault perception to automatic generation of an optimal filling scheme is formed, effectively solving the technical problems of slow response, low efficiency, and poor filling effect caused by relying on manual intervention or using a fixed filling strategy after the PTZ fails in the prior art, and achieving unmanned, intelligent, and optimized emergency response and automatic recovery of the monitoring system failure, thereby improving the overall robustness and operation efficiency of the system.

[0087] After combining the above content, the method provided by the present embodiment is further described in more detail. Please refer to Figure 2 , which is another flowchart of the PTZ failure inspection and filling method based on neighbor cooperation in the embodiments of the present application.

[0088] S201, determining a filling angle covering the gap sub-area based on the digital twin model of the optimal takeover PTZ, and generating an added task including a takeover task preset position;

[0089] After determining the optimal takeover PTZ, accurate filling task parameters need to be developed for the optimal takeover PTZ to ensure that it can accurately cover the gap sub-area, laying a foundation for subsequent integration into the original inspection plan.

[0090] The server calls the digital twin model of the optimal takeover PTZ, extracts the key parameters therein, including the installation position of the PTZ, the horizontal rotation range (such as 0°-360°), the vertical rotation range (such as -30°-90°), the lens focal length range, the field of view angle, and other static data, as well as the current horizontal angle, vertical angle, lens focal length, and other dynamic data. These data provide a basis for calculating the compensation angle. Then, the server preliminarily determines the horizontal angle and vertical angle to which the PTZ needs to be adjusted to ensure that the lens can be aimed at the center position of the gap sub-region according to the three-dimensional spatial coordinates of the gap sub-region (obtained from the three-dimensional model of the target region) and the installation coordinates of the optimal takeover PTZ through spatial geometric calculation. For example, if the gap sub-region is located at the northeast direction 30° and the elevation angle 10° of the optimal takeover PTZ, the preliminary calculation of the compensation angle is horizontal 30° and vertical 10°.

[0091] In combination with the lens parameters (such as the field of view angle) in the digital twin model, it is verified whether the preliminarily determined compensation angle can completely cover the gap sub-region. If the coverage range is insufficient due to a small field of view angle, the focal length is adjusted (the focal length is shortened to expand the field of view angle) or the angle is fine-tuned to ensure that the entire range of the gap sub-region is within the viewable range of the lens. At the same time, the line-of-sight blocking data (from the static data updated in the early stage) in the digital twin model needs to be referred to to avoid blocking objects such as buildings and trees, and the compensation angle is corrected to ensure unblocked coverage.

[0092] After determining the final compensation angle, it is combined with corresponding parameters such as focal length and shooting duration to generate a takeover task preset position. The takeover task preset position is a set of instructions containing specific operation parameters, such as "horizontal angle 35°, vertical angle 12°, focal length 50mm, shooting duration 3 minutes". Finally, taking the takeover task preset position as the core, the task number, execution frequency (such as every 15 minutes), data upload requirements, and other information are supplemented to form a complete new task. The new task needs to be clear and specific to ensure that the optimal takeover PTZ can accurately understand and execute.

[0093] S202, obtain the original inspection plan of the optimal takeover PTZ, and determine the original risk value, priority, and time proportion corresponding to each sub-original plan in the original inspection plan;

[0094] The original inspection plan refers to the regular inspection task arrangement of the optimal takeover PTZ before undertaking the compensation task, which includes multiple sub-original plans. The sub-original plan refers to a specific inspection task unit divided in the original inspection plan, which corresponds to a specific monitoring area or device. The original risk value refers to the risk level quantization value of the area covered by the sub-original plan, which is used to represent the importance of the inspection of the area. The time proportion refers to the proportion of the sub-original plan in the total time length of the original inspection plan, which reflects the time resources required to execute the task.

[0095] After generating the new task, the purpose is to obtain the optimal takeover gimbal's original task arrangement and the key attributes of each task, to provide data support for reasonably integrating the new task into the original plan, and to ensure the coordination of the filling task and the original task.

[0096] Firstly, the server obtains the original inspection plan stored by the control server of the optimal takeover gimbal by communicating with it. The original inspection plan usually exists in the form of electronic documents or database records, containing information such as task name, execution time, coverage area, execution frequency, etc. For example, the original inspection plan may include "Area A is inspected once an hour" and "Device B is inspected once every two hours". Then, the original inspection plan is parsed and split into multiple independent sub-plans. The split can be based on different coverage areas, different inspection objects, or different execution times. For example, three independent inspection tasks covering areas A, B, and C are split into three sub-plans.

[0097] Then, the original risk value corresponding to each sub-plan is determined. The original risk value is usually pre-set according to the environmental characteristics of the sub-plan coverage area (such as whether there are flammable materials, the degree of equipment concentration), historical failure data, safety level, etc. It can also be determined in the same way as the gap risk value in step S104, with a value range of 0-100 (the higher the value, the higher the risk). For example, the original risk value of a sub-plan covering a flammable warehouse may be set to 90, while the original risk value of a sub-plan covering a normal area may be set to 30.

[0098] Then, the priority of each sub-plan is determined. The priority is usually positively related to the original risk value, i.e., the higher the original risk value, the higher the priority, but it can also be adjusted according to actual needs. The priority can be represented by numbers (such as 1-5 levels, with level 1 being the highest) or words (such as "urgent", "routine", "low priority"). For example, the priority of a sub-plan with an original risk value of 90 is set to level 1, and the priority of a sub-plan with an original risk value of 30 is set to level 3. Calculate the time proportion of each sub-plan. Calculate the total duration of the original inspection plan (e.g., 24 hours a day), then calculate the product of the single execution duration and execution frequency of each sub-plan (e.g., 3 minutes each time, once an hour, the total duration of a day is 3x24=72 minutes), and divide the value by the total duration of the original inspection plan (24x60=1440 minutes) to get the time proportion (72 / 1440=5%). The time proportion reflects the weight of the sub-plan in time resource allocation.

[0099] S203, associate the time proportion with the original risk value to obtain a risk-time mapping relationship;

[0100] The association between the two needs to be analyzed to provide a basis for the time resource allocation of the new task, ensuring that the time proportion of the gap-filling task matches its risk level and meets the original plan's resource allocation logic.

[0101] First, collect the original risk values and corresponding time proportion data of all sub-plans to form a dataset. For example, sub-plan A (risk value 90, time proportion 8%), sub-plan B (risk value 70, time proportion 5%), sub-plan C (risk value 50, time proportion 3%), etc. Then, sort and classify the dataset, grouping sub-plans with similar original risk values. For example, group risk values 80-100 as high-risk, 60-79 as medium-high-risk, 40-59 as medium-risk, 20-39 as medium-low-risk, and 0-19 as low-risk. Then, calculate the average time proportion of each group of sub-plans. For example, the high-risk group has three sub-plans with time proportions of 8%, 7%, and 9%, so the average time proportion is (8% + 7% + 9%) / 3 = 8%; the medium-high-risk group has an average time proportion of 5%, and so on.

[0102] Draw a scatter plot or line graph with the original risk value interval as the horizontal axis and the average time proportion as the vertical axis to visually display the correlation trend between the two. Generally, the higher the original risk value, the greater the average time proportion, showing a positive correlation. For example, for every 20-unit increase in risk value, the average time proportion increases by 2-3%. Finally, based on the data distribution and correlation trend, establish a risk-time mapping relationship model. This model can be a linear equation (such as time proportion = 0.1 x risk value - 2%), a piecewise function (such as risk value 0-50, time proportion = 0.05 x risk value; risk value 51-100, time proportion = 0.08 x risk value - 1.5%), or a lookup table (such as risk value 90 corresponds to 8%, 70 corresponds to 5%, etc.). The mapping relationship needs to be verified to ensure that it accurately reflects the original plan's risk and time allocation rules.

[0103] S204, compare the gap risk value corresponding to the gap sub-region with each original risk value to determine the new priority of the new task inserted into the original inspection plan;

[0104] This step needs to determine the execution priority of the new task based on the risk level of the gap sub-region, ensuring that high-risk gap-filling tasks are executed first while avoiding disrupting the original plan's priority logic.

[0105] First, the gap sub-region corresponding gap risk value identification. The value is determined based on the flammable characteristics, environmental risk factors, such as a gap sub-region gap risk value identification for 85. Collect all original risk value of the original sub-plan in the original inspection plan, form a risk value list. For example, sub-plan A (90), sub-plan B (75), sub-plan C (60), sub-plan D (40). The gap risk value identification and the original risk value in the risk value list are compared one by one to determine its position in the risk level sequence. For example, the gap risk value 85 is compared with the original risk value, which is between 90 (A) and 75 (B), that is, higher than 75 (B) and lower than 90 (A).

[0106] According to the comparison result and the priority rule of the original plan (usually the higher the risk value, the higher the priority), the new priority of the new task is determined. If the original priority is sorted by risk value from high to low as A (1st), B (2nd), C (3rd), D (4th), then the new priority corresponding to the gap risk value 85 should be between 1st and 2nd, which can be set to 1.5th, or adjusted to 1st (same as A) or 2nd (but ahead of B) according to actual needs. If the gap risk value identification is higher than all original risk values, for example, the gap risk value is 95, which is higher than 90 (A), then the new priority is set to the highest level (such as 0th), ensuring that the new task is executed before all original sub-plans; if the gap risk value identification is lower than all original risk values, for example, it is 30, which is lower than 40 (D), then the new priority is set to the lowest level (such as 5th), which is executed after all original sub-plans.

[0107] In addition, special cases also need to be considered, such as when there are multiple gap sub-regions corresponding to multiple new tasks, the priority between new tasks also needs to be determined according to the gap risk value identification, and the new task with high risk is executed first.

[0108] S205, according to the risk-time mapping relationship and the gap risk value identification, determine the new time proportion of the new task in the total time corresponding to the original inspection plan;

[0109] After determining the new priority of the new task, a reasonable time proportion needs to be allocated to the new task according to the risk level of the gap sub-region and the time allocation rule of the original plan, to ensure that the new task has enough time resources to complete, while avoiding excessive occupation of time and affecting the execution of the original inspection task.

[0110] First, determine the specific form of the established risk-time mapping relationship, whether it is a linear equation, a piecewise function, or a lookup table. For example, the mapping relationship may be a linear rule of "risk value increases by 20, time ratio increases by 2%", or a lookup table of "risk value 80-100 corresponds to 8%, 60-79 corresponds to 5%, 40-59 corresponds to 3%". Then extract the gap risk value identifier corresponding to the gap sub-region, assuming the value is 85. Substitute this risk value into the risk-time mapping relationship to find or calculate the corresponding time ratio. Then, adjust the preliminary time ratio obtained according to the actual situation. Factors to consider include the size of the gap sub-region (the larger the area, the longer the inspection time may be needed), the complexity of the inspection (such as the degree of equipment density in the region, which requires more time), and the time margin of the original inspection plan (if the original plan is tight, the time ratio of the new task can be appropriately reduced, but the basic coverage requirement must be ensured). For example, the preliminary time ratio is 8%, but the gap sub-region is small, so it can be adjusted to 6%; if the original plan has sufficient time and the region is complex, it can be increased to 9%.

[0111] At the same time, it is necessary to ensure that the time ratio of the new task matches the new priority. High-priority tasks should generally be allocated a relatively high time ratio to ensure the quality of the inspection. For example, a new task with a new priority of 1 (highest) should not have a time ratio lower than the average time ratio of the original 1-level sub-plan. Finally, determine the final new time ratio and record the calculation basis and adjustment factors corresponding to the ratio, so as to evaluate and optimize the inspection plan in the future.

[0112] S206, determine the inspection execution task sequence in combination with the new priority and the new time ratio.

[0113] Collect the priority, time ratio of all original sub-plans, and new priority, new time ratio of new tasks to form a task attribute list. For example, original sub-plans A (priority 1, time ratio 8%), B (priority 2, time ratio 5%), C (priority 3, time ratio 3%), new task D (new priority 1.5, new time ratio 6%).

[0114] In order to realize the efficient cooperation of the replacement task and the original inspection plan, the application establishes a mapping relationship among the risk, the priority and the time proportion of the original inspection plan, and then determines the new priority and the new time proportion of the new task and fuses to form a task sequence.

[0115] Meanwhile, the execution frequency and periodicity of the task need to be considered. If the original sub-plan is a periodic task (such as being executed once every 2 hours), the periodicity needs to be ensured when planning the sequence, and the new task needs to be inserted into the interval to avoid breaking the original cycle.

[0116] In the embodiment of the application, the replacement angle is accurately determined based on the digital twin model to generate a new task, a mapping relationship is established in combination with the risk, the priority and the time proportion of the original inspection plan, and then the new priority and the new time proportion of the new task are determined and fused to form a task sequence, so that the efficient cooperation of the replacement task and the original inspection plan can be realized, the problems of plan confusion, unreasonable time allocation and priority conflict caused by the insertion of the replacement task can be effectively solved, and the orderliness of the optimal takeover of the pan-tilt inspection task, the rationality of resource allocation and the accuracy of the replacement inspection can be realized.

[0117] The server in the embodiment of the application will be described from the perspective of hardware processing. Please refer to Figure 3 Fig. 1 is a schematic diagram of an entity device structure of the server in the embodiment of the application.

[0118] It should be noted that Figure 3 The structure of the server shown is only an example, and should not bring any limitation to the function and use range of the embodiment of the application.

[0119] As Figure 3As shown, the server includes a Central Processing Unit (CPU) 301 which can perform various appropriate actions and processes in accordance with a program stored in a Read-Only Memory (ROM) 302 or a program loaded from a storage section 308 into a Random Access Memory (RAM) 303, such as performing the methods described in the above embodiments. In the RAM 303, various programs and data required for the operation of the server are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0120] Connected to the I / O interface 305 are an input section 306 including an audio input device, a push button switch, and the like; an output section 307 including a Liquid Crystal Display (LCD), an audio output device, an indicator, and the like; a storage section 308 including a hard disk and the like; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as necessary. A removable media 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 310 as necessary, so that a computer program read therefrom is installed into the storage section 308 as necessary.

[0121] In particular, in accordance with embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from the removable media 311. When the computer program is executed by the Central Processing Unit (CPU) 301, various functions defined in the present application are performed.

[0122] Note that specific examples of computer-readable storage media can include but are not limited to an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, computer-readable storage media can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0123] The flow charts and block diagrams in the attached drawings are used to illustrate the architecture, functionality, and operation of possible implementations of servers, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow charts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved.

[0124] In particular, the server of the embodiment includes a processor and a memory, and the memory stores a computer program, which, when executed by the processor, implements the gimbal failure inspection and replacement method based on neighbor cooperation provided by the above-mentioned embodiments.

[0125] As another aspect, the present application also provides a computer-readable storage medium, which can be included in the server described in the above-mentioned embodiments, or can exist separately and not be assembled into the server. The storage medium carries one or more computer programs, which, when executed by a processor of the server, enable the server to implement the gimbal failure inspection and replacement method based on neighbor cooperation provided by the above-mentioned embodiments.

[0126] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0127] In the above embodiments, the term "when" can be interpreted to mean "if" or "after" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "on determining" or "if detecting (a stated condition or event)" can be interpreted to mean "if determining" or "in response to determining" or "on detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)" depending on the context.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program instructing the relevant hardware to complete, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The aforementioned storage medium includes ROM or random storage memory RAM, magnetic disk or optical disk and various storage program codes.

Claims

1. A pan-tilt failure inspection and replacement method based on neighbor cooperation, applied to a server, characterized in that, The method comprises: determining the digital twin model of each PTZ device in the target area, and determining whether the corresponding target PTZ device is invalid based on the digital twin model and the target PTZ device continuously returning blank images, blurred images, or abnormal images with large differences from the preset scene; if it is determined that the target PTZ device is invalid, determining the invalid type; determining the gap area corresponding to the preset position of the invalid PTZ according to the digital twin model and the invalid type; dividing the gap area into a plurality of gap sub-areas with gap risk value labels; determining candidate neighboring machines matched with each gap sub-area, and determining the optimal takeover PTZ corresponding to each gap sub-area from the candidate neighboring machines; determining the inspection execution task sequence containing the newly added task of the gap sub-area requiring repositioning in combination with the digital twin model of the optimal takeover PTZ; controlling the optimal takeover PTZ to perform the repositioning operation according to the inspection execution task sequence; when it is determined that the invalid PTZ has returned to normal, automatically triggering the takeover exit process to restore the original inspection plan of the optimal takeover PTZ; the step of determining the gap area corresponding to the preset position of the invalid PTZ according to the digital twin model and the invalid type specifically comprises: if the dynamic data in the digital twin model presents a device health degree of zero or continuous heartbeat loss, it is determined that the invalid type is total invalid, and the area corresponding to all preset preset position requirements in the static data of the invalid PTZ is the gap area; if the target preset position response times out, it is determined that the invalid type is partial view angle invalid, and the area corresponding to the target preset position is the gap area; if there is a local abnormality in the returned image, the gap area is obtained by combining the effective picture boundary of the abnormal image and the preset coverage range of the target preset position; the step of dividing the gap area into a plurality of gap sub-areas with gap risk value labels specifically comprises: obtaining the invalid timestamp of the invalid PTZ; obtaining a complete panoramic inspection video collected by the invalid PTZ before the invalid timestamp; based on a preset flammable material feature layout library, performing frame-by-frame object detection on the complete panoramic inspection video, extracting the flammability of each object, and determining the flammable score value of all objects based on the flammability; combining the spatial coordinates of each object in the panoramic video, combining the flammable score value, and generating a risk heat map of the gap area through a spatial interpolation algorithm, wherein the risk heat map includes gap risk value labels based on the flammable score value; dividing the gap area to obtain a plurality of gap sub-areas according to the distribution of the gap risk value labels in the risk heat map; the step of determining the inspection execution task sequence containing the newly added task of the gap sub-area requiring repositioning in combination with the digital twin model of the optimal takeover PTZ specifically comprises: determining the repositioning angle covering the gap sub-area based on the digital twin model of the optimal takeover PTZ, and generating a newly added task including a takeover task preset position; obtaining the original inspection plan of the optimal takeover PTZ, and determining the original risk value, priority, and time proportion corresponding to each sub-original plan in the original inspection plan; Correlate the time proportion with the original risk value to obtain a risk-time mapping relationship; Compare the gap risk value mark corresponding to the gap sub-region with each original risk value to determine a new priority of inserting the new task into the original inspection plan; Determine a new time proportion of the new task in the total time of the original inspection plan according to the risk-time mapping relationship and the gap risk value mark; Determine an inspection execution task sequence in combination with the new priority and the new time proportion.

2. The method of claim 1, wherein, The step of determining a candidate neighboring machine matched with each gap sub-region from the candidate neighboring machines, specifically includes: Exclude neighboring machines in the target region that cannot cover the gap sub-region or have an overlap rate with the gap sub-region lower than a preset overlap rate threshold from the candidate neighboring machines to obtain remaining candidate neighboring machines; Obtain a coverage capacity matching degree by correcting the ratio of the theoretically maximum visible coverage area of the gap sub-region to the total area of the sub-region according to the remaining candidate neighboring machines and in combination with a line-of-sight occlusion model; Determine a task interference degree based on the reciprocal of the average inspection interval extension rate of the original high-priority task of the remaining candidate neighboring machine after virtually inserting the new task; Determine an energy consumption cost by the motor rotation and equipment power consumption generated by the remaining candidate neighboring machine moving from the current position to the replacement preset position and executing the new task for one cycle, where the replacement preset position is determined by the gap sub-region; Determine a response speed according to the estimated time required for the remaining candidate neighboring machine to complete the current task and rotate to the first replacement preset position; Obtain a comprehensive score of each remaining candidate neighboring machine by a preset weighted summation model according to one or more of the coverage capacity matching degree, the task interference degree, the energy consumption cost, and the response speed; Select the neighboring machine with the highest comprehensive score as the optimal takeover gimbal.

3. The method of claim 1, wherein, The step of determining the digital twin model of each gimbal device in the target region further includes: Periodically control all normal gimbal devices in the target region to perform panoramic scanning after the inspection task is completed to obtain a multi-angle environment image sequence; Process the environment image sequence to generate a three-dimensional grid model of the target region; Compare the three-dimensional grid model with the existing static data in the respective corresponding digital twin model to identify and quantify environmental changes; Automatically update the static data in the digital twin model according to the environmental changes, and the updating includes correcting the line-of-sight occlusion relationship of each gimbal device to each monitoring region.

4. The method of claim 1, wherein, The step of automatically triggering the takeover exit process when the failed gimbal is determined to be restored to normal, specifically includes: When the failed gimbal is first detected to be restored to normal, mark the corresponding state as an observation period, and control the optimal takeover gimbal to continue performing the replacement operation within a preset time period; During the observation period, continuously monitor the dynamic data of the restored gimbal, and if the corresponding continuous stable operation time reaches the preset time period, formally confirm that the failed gimbal is restored to normal, and trigger the takeover exit process.

5. A server, characterized by The server comprises one or more processors and a memory; the memory is coupled with the one or more processors, the memory is configured to store computer program codes, the computer program codes comprise computer instructions, and the one or more processors invoke the computer instructions to enable the server to perform the method in any one of claims 1-4.

6. A computer-readable storage medium comprising instructions, characterized in that, The instructions, when executed on a server, enable the server to perform the method in any one of claims 1-4.

7. A computer program product, characterised in that, The computer program product, when executed on a server, enables the server to perform the method in any one of claims 1-4.

Images