A Method and System for Monitoring Operational Scenes Based on Panoramic Stitching and Pan-Tilt-Zoom (PTZ) Linkage

Abstract

This invention provides a method and system for monitoring work scenarios based on panoramic stitching and PTZ camera linkage, belonging to the field of video surveillance technology. The method includes: acquiring real-time panoramic video data and historical video data of the work scenario, as well as partial discharge signals from high-voltage electrical equipment; simulating electrical malfunction characteristics of the high-voltage electrical equipment based on environmental parameters from the partial discharge signals and real-time panoramic video data; obtaining work prediction information from the real-time panoramic video based on historical video data and electrical malfunction characteristics; calculating deviation information of the real-time panoramic video based on the work prediction information and the real-time panoramic video data; locating risk target areas of the work scenario based on the deviation information; determining risk warning signals for the work scenario based on a preset risk rule base; and sending the risk warning signals to the PTZ camera of the work scenario to monitor close-up images of the risk target areas in real time. This invention can improve the efficiency of risk handling in work scenario monitoring.

Description

Technical Field

[0001] This invention relates to the field of video surveillance technology, specifically to a method and system for monitoring work scenes based on panoramic stitching and PTZ linkage. Background Technology

[0002] Traditional monitoring methods often rely on single or fixed-angle cameras, making it difficult to achieve panoramic coverage and leaving blind spots. Furthermore, they lack a systematic approach to monitoring the condition of critical facilities such as high-voltage electrical equipment. Existing technologies often rely solely on video footage for manual inspections or simple anomaly identification, failing to consider the equipment's electrical signals and environmental parameters. This makes it impossible to accurately simulate potential electrical problems, leading to delayed predictions of equipment failures.

[0003] Traditional PTZ cameras are mostly controlled manually or via simple preset tracks, making it impossible to integrate with panoramic monitoring systems to respond to risk warnings. When a risk occurs, the PTZ camera cannot quickly move to track the target area in close-up, resulting in delayed capture of risk details and impacting subsequent decision-making. These shortcomings make existing monitoring methods inefficient in terms of safety assurance and fault prevention in operational scenarios, and they fail to meet the real-time monitoring and risk management needs of complex working environments. Summary of the Invention

[0004] The purpose of this invention is to provide a method for monitoring work scenarios based on panoramic stitching and PTZ linkage to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, embodiments of the present invention provide a method for monitoring work scenes based on panoramic stitching and PTZ linkage, including: S1: Acquire real-time panoramic video data and historical video data of the work scene, as well as partial discharge signals of high-voltage electrical equipment; S2: Simulate the electrical malfunction characteristics of the high-voltage electrical equipment based on the environmental parameters of the partial discharge signal and the real-time panoramic video data; S3: Based on the historical video data and the electrical fault characteristics, perform time-series extrapolation on the real-time panoramic video data to obtain the operation prediction information of the real-time panoramic video; S4: Calculate the deviation between the job prediction information and the real-time panoramic video data to obtain the deviation information of the real-time panoramic video; S5: Based on the deviation information, locate the risk target area of ​​the work scenario, and perform risk judgment on the deviation information based on the preset risk rule base to obtain the risk warning signal of the work scenario; S6: Send the risk warning signal to the PTZ camera in the work scene, and drive the PTZ camera to monitor the close-up monitoring screen of the risk target area in real time.

[0006] Optionally, acquiring real-time panoramic video data and historical video data of the work scene, as well as the partial discharge signal of the high-voltage electrical equipment, includes: The real-time video streams from multiple different angles in the work scene are collected and stitched together to form real-time panoramic video data of the work scene; Retrieve historical video data of the aforementioned work scenario; The non-intrusive signals of the high-voltage electrical equipment are monitored, and the non-intrusive signals are identified as partial discharge signals of the high-voltage electrical equipment.

[0007] Optionally, the simulation of electrical malfunction characteristics of the high-voltage electrical equipment based on the environmental parameters of the partial discharge signal and the real-time panoramic video data includes: The partial discharge characteristics of the partial discharge signal are analyzed, and the environmental parameters of the real-time panoramic video data are identified. A dynamic mapping relationship for the high-voltage electrical equipment is constructed based on the partial discharge characteristics and the environmental parameters. The electrical malfunction characteristics of the high-voltage electrical equipment are determined based on the dynamic mapping relationship and the physical structure of the high-voltage electrical equipment.

[0008] Optionally, the step of performing time-series extrapolation on the real-time panoramic video data based on the historical video data and the electrical malfunction characteristics to obtain the operation prediction information of the real-time panoramic video includes: Analyze the dynamic change characteristics of the historical video data; The dynamic change features and the electrical malfunction features are combined to form the scene state features of the work scenario; The state characteristics of the scenario are predicted in time series based on a pre-trained long short-term memory network to obtain the state evolution trend of the task scenario. Based on the state evolution trend, a forward-looking extrapolation is performed on the real-time panoramic video to obtain the operation prediction information of the real-time panoramic video.

[0009] Optionally, the calculation formula for the scene state features includes:

[0010] in, These are the scene state features. yes The dynamic change characteristics described at any given time. It refers to the physical structure of the high-voltage electrical equipment. These are the environmental parameters. It is the temperature change rate of the historical video data. It is the discharge change rate of the historical video data. yes The time complexity coefficient, It is a time marker.

[0011] Optionally, the step of locating the risk target area of ​​the work scenario based on the deviation information, and performing risk assessment on the deviation information based on a preset risk rule base to obtain a risk warning signal for the work scenario includes: Based on the deviation information, a deviation heatmap of the work scenario is drawn, and the risk target area of ​​the work scenario in the deviation heatmap is located. The overall risk level of the work scenario is matched based on the deviation information and the preset risk rule base; A risk warning signal for the work scenario is generated based on the comprehensive risk level.

[0012] Optionally, the construction of the preset risk rule base includes: The safety regulations in the aforementioned work scenario will be used as the initial risk rule base for that work scenario. Collect risk events associated with the risk warning signals as feedback samples; The feedback samples are updated to the initial risk rule base to obtain the risk rule base for the work scenario.

[0013] Optionally, the step of sending the risk warning signal to the PTZ camera in the work scene and driving the PTZ camera to monitor the close-up surveillance footage of the risk target area in real time includes: Based on the risk warning signal, extract the risk target information of the work scenario; Control commands for the PTZ camera are generated based on the risk target information and the risk target area. The control command is sent to the PTZ camera to monitor a close-up view of the risk target area.

[0014] Optionally, the step of sending the risk warning signal to the PTZ camera in the work scene and driving the PTZ camera to monitor the close-up surveillance footage of the risk target area in real time further includes: A risk assessment is performed on the close-up surveillance footage to determine its risk status. The risk status is compared with the risk warning signal to obtain the feedback information of the close-up monitoring screen; The control commands for the gimbal camera are regenerated based on the feedback information.

[0015] To address the aforementioned problems, this invention also provides a work scene monitoring system based on panoramic stitching and PTZ linkage, the system comprising: The data acquisition module is used to acquire real-time panoramic video data and historical video data of the work scene, as well as partial discharge signals of high-voltage electrical equipment. An electrical malfunction characteristic simulation module is used to simulate the electrical malfunction characteristics of the high-voltage electrical equipment based on environmental parameters of the partial discharge signal and the real-time panoramic video data. The time-series extrapolation module is used to perform time-series extrapolation on the real-time panoramic video data based on the historical video data and the electrical malfunction characteristics to obtain the operation prediction information of the real-time panoramic video. The deviation information calculation module is used to calculate the deviation between the job prediction information and the real-time panoramic video data to obtain the deviation information of the real-time panoramic video. The risk warning module is used to locate the risk target area of ​​the work scenario based on the deviation information, determine the risk of the deviation information based on a preset risk rule base, and obtain a risk warning signal for the work scenario. The PTZ control module is used to send the risk warning signal to the PTZ camera in the work scene and drive the PTZ camera to monitor the close-up monitoring screen of the risk target area in real time.

[0016] Beneficial effects 1. By capturing video streams from multiple pan-tilt cameras at different angles and stitching them together, real-time panoramic video data is formed, effectively eliminating blind spots in traditional single-view monitoring and achieving comprehensive coverage of the work scene. Simultaneously, by combining partial discharge signals from high-voltage electrical equipment with environmental parameters from real-time video to simulate electrical malfunctions, this method overcomes the limitations of relying solely on video footage or single signal monitoring, enabling the detection of potential equipment faults and providing a reliable basis for risk prediction.

[0017] 2. Based on historical video data and the time-series extrapolation mechanism of electrical malfunction characteristics, operational predictions can be made from real-time video data. Risk target areas can be located through deviation calculation, and warning signals can be generated by combining this with a pre-set risk rule base, significantly improving the timeliness and accuracy of risk identification. Furthermore, the warning signals are sent to PTZ cameras, driving them to track close-ups of risk areas in real time, achieving linkage between panoramic monitoring and local close-ups. This allows for seamless transitions from panoramic coverage to detailed focus, significantly improving the ability to respond quickly to and handle risks, and providing strong support for the safety management of operational scenarios.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a method for monitoring work scenarios based on panoramic stitching and gimbal linkage, provided in an embodiment of the present invention. Figure 2 This is a functional module diagram of a work scene monitoring system based on panoramic stitching and PTZ linkage provided in an embodiment of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0021] This application provides a method for monitoring work scenes based on panoramic stitching and PTZ linkage. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for monitoring work scenes based on panoramic stitching and PTZ linkage can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0022] Reference Figure 1 The diagram shown is a flowchart illustrating a work scene monitoring method based on panoramic stitching and PTZ linkage according to an embodiment of the present invention. In this embodiment, the work scene monitoring method based on panoramic stitching and PTZ linkage includes: S1: Acquire real-time panoramic video data and historical video data of the work scene, as well as partial discharge signals of high-voltage electrical equipment.

[0023] In this embodiment of the invention, the acquisition of real-time panoramic video data and historical video data of the work scene, as well as the partial discharge signal of the high-voltage electrical equipment, includes: The real-time video streams from multiple different angles in the work scene are collected and stitched together to form real-time panoramic video data of the work scene; Retrieve historical video data of the aforementioned work scenario; The non-intrusive signals of the high-voltage electrical equipment are monitored, and the non-intrusive signals are identified as partial discharge signals of the high-voltage electrical equipment.

[0024] Specifically, the process involves using multiple cameras integrated into a device to capture video streams from different angles and then fusing them into a seamless, wide-angle, or 360-degree panoramic video. Camera calibration corrects lens distortion in each video stream, a feature point detection algorithm finds and matches key feature points in overlapping areas of the video streams, the transformation matrix is ​​calculated and optimized, projection transformations are applied to each video frame to align them in a unified coordinate system, and a linear weighted algorithm is used to smooth the stitching seams, eliminating stitching artifacts and generating a coherent and natural real-time panoramic video.

[0025] In detail, retrieving historical video data for a work scenario refers to the technical step of retrieving and extracting recorded video clips from a video repository according to specific needs. This step relies on a robust video management library and an efficient storage architecture. First, metadata indexes such as timestamps and event tags are created for the stored video files. When a retrieval request is initiated, it includes parameters such as the camera ID, start timestamp, and end timestamp. The video management library uses the index to quickly locate the specific file or data block on the storage medium, and then streams the compressed video data.

[0026] Furthermore, monitoring partial discharge signals in high-voltage electrical equipment specifically refers to using non-invasive techniques to capture the physical signals generated when the equipment's insulating medium undergoes partial electrical breakdown under a strong electric field. "Non-invasive" means that the sensor does not need to directly contact the high-voltage live parts, ensuring the safety of the monitoring process. Ultrasonic sensors are used to pick up the sound waves or ultrasonic signals generated by the discharge. These specific physical signals collected are then identified as partial discharge signals.

[0027] For example, a comprehensive digital inspector can be deployed in the main transformer area of ​​a substation. Its six built-in image sensors activate simultaneously, capturing video of the surrounding environment. The device's high-performance processor utilizes its 4Kp60 video encoding capabilities to stitch and fuse multiple video streams in real time, ultimately generating a stable, distortion-free 360-degree panoramic video. This high-quality video stream is then uploaded to a remote inspection center via a 5G network.

[0028] Furthermore, as the full-area monitoring system begins real-time monitoring, the backend automatically retrieves historical video data from the video server for the main transformer area over the past 30 days. After analyzing this data, the algorithm establishes a dynamic model. Under normal circumstances, between 10:00 AM and 11:00 AM daily, the movement speed of large lifting machinery in this area will not exceed 5 km / h, and the safe distance between the machinery and energized equipment will not be less than 15 meters.

[0029] Specifically, a magnetic ultrasonic sensor was installed on the main transformer tank to be replaced. During the hoisting preparation phase, the sensor successfully picked up the ultrasonic signal emitted by a partial breakdown caused by residual internal charge in the insulating oil. After eliminating noise interference from on-site construction, this signal was identified as a valid partial discharge event, and its intensity was quantified as 75 pC. This signal data was timestamped and integrated with the panoramic video data of the full-area device into the risk analysis module, providing key inputs from both visual and electrical dimensions for subsequent comprehensive assessment.

[0030] S2: Simulate the electrical malfunction characteristics of the high-voltage electrical equipment based on the environmental parameters of the partial discharge signal and the real-time panoramic video data.

[0031] In this embodiment of the invention, the simulation of electrical malfunction characteristics of the high-voltage electrical equipment based on environmental parameters derived from the partial discharge signal and the real-time panoramic video data includes: The partial discharge characteristics of the partial discharge signal are analyzed, and the environmental parameters of the real-time panoramic video data are identified. A dynamic mapping relationship for the high-voltage electrical equipment is constructed based on the partial discharge characteristics and the environmental parameters. The electrical malfunction characteristics of the high-voltage electrical equipment are determined based on the dynamic mapping relationship and the physical structure of the high-voltage electrical equipment.

[0032] Specifically, the acquired partial discharge signals undergo in-depth processing. Through phase-resolved partial discharge spectrum analysis, their inherent partial discharge characteristics are analyzed, including the discharge amplitude, phase, frequency, and rise time. These characteristics collectively constitute the fingerprint of the discharge event. Simultaneously, computer vision algorithms are used to analyze real-time panoramic video data to identify key environmental parameters. By reading the readings of physical instruments in the video, real-time information such as temperature and humidity at the work site is obtained.

[0033] Furthermore, a dynamic mapping relationship is constructed for high-voltage electrical equipment. High temperatures accelerate the chemical aging of insulating materials, while high humidity reduces the surface flashover voltage of the insulating medium. This dynamic mapping relationship is typically trained using neural network machine learning methods based on massive amounts of equipment experimental data and historical fault cases. It can dynamically and quantitatively assess the potential risk level and fault evolution trend under the current combination of conditions based on real-time input discharge characteristics and environmental parameters.

[0034] In detail, the results of the dynamic mapping relationship are deeply integrated with the physical structural information of the high-voltage electrical equipment itself to ultimately determine its electrical malfunction characteristics. The physical structural information of high-voltage electrical equipment is a comprehensive data archive, including its model, production year, insulation type (e.g., epoxy resin, oil-impregnated paper), key component materials, historical maintenance records, and past fault data. By combining the dynamic risk assessment results with the physical structural information of the high-voltage electrical equipment, a precise and concrete diagnostic conclusion regarding the characteristics of electrical malfunctions can be generated.

[0035] For example, the 75pC partial discharge signal detected in the previous step was processed. Through phase-resolved partial discharge spectrum analysis, the discharge characteristics of the signal were determined to be: the discharge phase is concentrated in the 30-60 degree and 210-240 degree ranges of the power frequency cycle, exhibiting typical gap discharge characteristics. The 4.09 million-pixel high-definition lens of the all-area device is pointed directly at the work area. Its built-in visual algorithm identifies the electronic thermometer and hygrometer hanging on the nearby wall. The environmental monitoring module reads and identifies the current environmental parameters as: temperature 38℃, relative humidity 92%.

[0036] In detail, the analyzed void discharge characteristics, 75pC, specific phase, and environmental parameters of 38℃ temperature and 92% humidity were input into a dynamic mapping model. Based on its built-in algorithm library for the aging mechanism of oil-paper insulation, the model determined that under this high-temperature and high-humidity environment, the dielectric constant of the insulating paperboard would change, causing the electric field strength of the internal voids to increase by nearly 30% compared to normal conditions. Therefore, the model constructed a mapping relationship, that is, the equivalent destructiveness of a 75pC discharge under the current environment is mapped to a 150pC discharge level in a dry environment, and output a dynamic risk vector, indicating that under this condition, the probability of accelerated insulation failure is as high as 95%.

[0037] Furthermore, the 95% probability of accelerated insulation failure was translated into a definitive diagnosis, and the physical structure information of the main transformer was immediately retrieved from the asset management database. The equipment was an oil-immersed transformer that had been in operation for 20 years, using cellulose paper as insulation material, and its maintenance records showed a record of handling aging and leakage of the sealing rings three years ago. Combining the physical structure with the dynamic mapping relationship of the sharply increased risk of gap discharge under high humidity conditions, the electrical symptoms of the equipment were finally determined. This diagnostic conclusion was immediately pushed to the remote monitoring center, and a high-brightness alarm was displayed on the 13.4-inch OLED display of the on-site all-area device: the oil-paper insulation layer of the A-phase high-voltage winding has been severely deteriorated due to moisture, and a stable discharge gap has formed inside, posing a high risk of inter-turn breakdown. Immediate power outage and maintenance are recommended.

[0038] S3: Based on the historical video data and the electrical fault characteristics, perform time-series extrapolation on the real-time panoramic video data to obtain the operation prediction information of the real-time panoramic video.

[0039] In this embodiment of the invention, the step of performing time-series extrapolation on the real-time panoramic video data based on the historical video data and the electrical malfunction characteristics to obtain the operation prediction information of the real-time panoramic video includes: Analyze the dynamic change characteristics of the historical video data; The dynamic change features and the electrical malfunction features are combined to form the scene state features of the work scenario; The state characteristics of the scenario are predicted in time series based on a pre-trained long short-term memory network to obtain the state evolution trend of the task scenario. Based on the state evolution trend, a forward-looking extrapolation is performed on the real-time panoramic video to obtain the operation prediction information of the real-time panoramic video.

[0040] Specifically, artificial intelligence algorithms are used to extract and quantify the regular behavioral patterns, i.e., dynamic change characteristics, of the work scenarios. Object detection and multi-object tracking algorithms are used to analyze the movement trajectories, average dwell time, and activity hotspots of personnel in historical video recordings, as well as the travel paths and operational patterns of machinery and equipment, such as vehicles. Through statistical analysis of this long-term data, a digital baseline of normal conditions can be constructed, providing a reference for subsequent identification of abnormal behavior.

[0041] In detail, the dynamic change characteristics describing the regular behavior patterns obtained from the previous step, such as the probability distribution of personnel positions and equipment speed limits, are combined with the electrical symptom characteristics describing the health status of the equipment, such as insulation aging and internal discharge, determined in the previous steps.

[0042] Furthermore, a Long Short-Term Memory (LSTM) network is used to infer the future evolution trend of the scene. The pre-trained LTM network receives the scene state features generated in the previous stage as input. Due to its unique gating mechanism, the LTM network can learn the long-term dependencies between scene states over time. By analyzing the current state of the input and combining it with the historical evolution patterns in its memory, the network can output a predicted sequence of scene state features for a future period, i.e., the state evolution trend.

[0043] Furthermore, a video prediction model based on generative adversarial networks is employed. This model takes the current real-time panoramic video frame as conditional input and the state evolution trend output by the Long Short-Term Memory network as guiding information. Based on the guiding information, the model makes forward-looking inferences about the current scene and generates a virtual video that is most likely to occur in the next few minutes.

[0044] For example, historical video data from the past 30 days for the main transformer area was retrieved and analyzed. Target detection algorithms were used to identify two key targets: workers and cranes. A multi-target tracking algorithm then generated their historical movement trajectories. The analysis showed that 85% of the workers' activity time was concentrated within the Class A safety zone, 15 meters away from the energized equipment; the crane's boom never entered the 10-meter restricted area above the high-voltage busbar.

[0045] In detail, the dynamic change characteristics, personnel activity heatmap, crane restricted area model, and the electrical malfunction feature diagnosed in S2—namely, the severe deterioration of the oil-paper insulation layer of the A-phase high-voltage winding due to moisture, resulting in a stable discharge gap inside—are spliced ​​together to generate a completely new scene state feature.

[0046] Specifically, the previously generated scene state features are input into a pre-trained Long Short-Term Memory (LSTM) network. Including features of severe insulation degradation, the LTM network's training data contains past signs preceding similar faults, which are used to make time-series predictions and output the state evolution trend for the next 10 minutes: Trend 1, the probability distribution of personnel locations should remain unchanged within the Class A safety zone; Trend 2, the characteristic value representing the temperature of the transformer's A-phase winding is expected to increase linearly at a rate of 0.5°C per minute.

[0047] Furthermore, real-time panoramic video captured by the all-domain device, along with the state evolution trend predicted by the Long Short-Term Memory Network (LSTM) as the personnel's position remains unchanged while the temperature in a specific area rises, are input into the video prediction GAN model. Based on this, the model performs forward-looking extrapolation and generates a 5-minute predictive video of the operation.

[0048] In this embodiment of the invention, the calculation formula for the scene state features includes:

[0049] in, These are the scene state features. yes The dynamic change characteristics described at any given time. It refers to the physical structure of the high-voltage electrical equipment. These are the environmental parameters. It is the temperature change rate of the historical video data. It is the discharge change rate of the historical video data. yes The time complexity coefficient, It is a time marker.

[0050] Specifically, These are the scene state features, at a specific moment. This is a quantitative score of the overall status of the entire work scenario. The higher the score, the higher the risk, importance, or uncertainty of the scenario.

[0051] In detail, yes The dynamic characteristics described at any given moment provide a quantitative description of real-time dynamic elements at the work site, such as the number of personnel, the intensity of activity, and the operating status of machinery and equipment. A target detection algorithm identifies the number of personnel and vehicles at the site, and a multi-target tracking algorithm tracks their speed and trajectory, comprehensively calculating a dynamic feature score.

[0052] Furthermore, This refers to the physical structure of the high-voltage electrical equipment, representing a quantitative value indicating the inherent risk and importance of the monitored equipment. This is related to factors such as the equipment's age, model, importance level, and historical failure rate.

[0053] Furthermore, This is a quantitative score of the impact of the environmental parameters on real-time on-site environmental conditions. Severe weather conditions, such as high temperature, high humidity, and strong winds, increase operational risks and therefore receive higher scores.

[0054] In detail, It is the temperature change rate of the historical video data, which is the average temperature change rate of the device under normal operating conditions, derived from the thermal imaging data of the historical video.

[0055] Furthermore, It is the discharge change rate of the historical video data, which is the average rate of change of the partial discharge signal of the equipment under normal operating conditions, derived from the analysis of historical monitoring data.

[0056] Furthermore, yes The complexity coefficient at any given moment is a dynamic coefficient that measures the complexity of the current operational scene. The complexity coefficient of the panoramic video image is obtained by dynamically analyzing the panoramic video footage.

[0057] In detail, the quantified values ​​of all the above parameters are calculated in real time or retrieved from the database. These parameters are then substituted into the formula to calculate a single value S. The underlying risk is determined by both real-time dynamic risks and static risks of the equipment environment.

[0058] Specifically, this underlying risk is influenced by historical stability; the more stable the historical rate of change, the more appropriately the weight of the current state will be adjusted. The formula also incorporates an independent risk term determined by real-time interaction, comprised of complexity and device environment factors. The calculated... The value, as a highly condensed information point, is fed into the Long Short-Term Memory network to predict the value at the next time point, thereby enabling the prediction of the evolution trend of the scene state.

[0059] S4: Calculate the deviation between the job prediction information and the real-time panoramic video data to obtain the deviation information of the real-time panoramic video.

[0060] In this embodiment of the invention, the step of calculating the deviation between the job prediction information and the real-time panoramic video data to obtain the deviation information of the real-time panoramic video includes: Specifically, anomalies are accurately identified by comparing the actual scene with the scene predicted by the model. Instead of directly comparing video pixels, the model compares structured feature data extracted from both video streams. Existing feature extraction algorithms are simultaneously applied to both real-time panoramic video and task prediction video.

[0061] In detail, two parallel sets of scene description data are generated for each time point, including the number of people, their location coordinates, equipment speed, and area temperature. Difference analysis is performed on these two sets of data, mainly including: entity deviation: comparing whether the number and category of targets are consistent; location deviation: calculating whether the spatial difference between the actual and predicted locations of the same target exceeds a threshold; and state deviation: comparing whether the target's attribute parameters, such as temperature and speed, differ significantly from the predicted values. All calculated differences are integrated into a structured deviation information log.

[0062] For example: Position deviation: The operation prediction video showed that all four workers should be within the Class A safety zone. However, the target tracking algorithm in the real-time video showed that one worker had left the safety zone and entered a position only 13 meters away from the main transformer, with a position deviation exceeding the safety threshold of 2 meters.

[0063] Condition Deviation: The operation prediction video showed that the temperature of the transformer's A-phase winding should be 55℃. However, the readings from the real-time thermal imaging analysis module showed that the actual temperature in this area had soared to 75℃, with a condition deviation of +20℃, far exceeding the expected rate of temperature increase.

[0064] Entity Deviation: The operation prediction video indicated that there should only be 4 workers on site. However, a fifth person was detected in the edge area of ​​the real-time video frame; this person was not wearing standard work clothes. This is a serious deviation in the number of entities. Immediately package these three deviation pieces of information, along with their occurrence time, location coordinates, and deviation values, into a complete deviation information log, and send it to the next step for risk level determination.

[0065] S5: Based on the deviation information, locate the risk target area of ​​the work scenario, and perform risk judgment on the deviation information based on the preset risk rule base to obtain the risk warning signal of the work scenario.

[0066] In this embodiment of the invention, the step of locating the risk target area of ​​the work scenario based on the deviation information, and determining the risk of the deviation information based on a preset risk rule base to obtain a risk warning signal for the work scenario includes: Based on the deviation information, a deviation heatmap of the work scenario is drawn, and the risk target area of ​​the work scenario in the deviation heatmap is located. The overall risk level of the work scenario is matched based on the deviation information and the preset risk rule base; A risk warning signal for the work scenario is generated based on the comprehensive risk level.

[0067] Specifically, for each entry in the deviation information log, a semi-transparent heatmap layer is overlaid on the real-time panoramic video feed. The heatmap's intensity is typically represented by color and brightness, with blue to red indicating a proportional severity of the deviation. In this way, all deviations from expectations are highlighted in the overall view. The area with the highest brightness in the heatmap is then automatically identified, and its coordinates and associated objects are defined as risk target areas.

[0068] In detail, the identified risks are classified. Each entry in the deviation information log is analyzed and matched against a pre-defined risk rule base. The rule base's judgment logic considers the type and magnitude of a single deviation, and may also consider the combined risks of multiple concurrent deviations. After matching each deviation event to an independent risk level, the overall risk level at that moment is given based on strategies such as taking the highest level or risk escalation.

[0069] Furthermore, once the overall risk level is determined, a detailed risk warning signal is immediately generated. This signal is typically a data packet, not just a simple sound or light alert. It is quite comprehensive, including at least: the final overall risk level, a list of specific deviation events that triggered that level, the precise coordinates of each risk target area, the timestamp of the alarm generation, and a human-readable risk summary text.

[0070] In this embodiment of the invention, the construction of the preset risk rule base includes: The safety regulations in the aforementioned work scenario will be used as the initial risk rule base for that work scenario. Collect risk events associated with the risk warning signals as feedback samples; The feedback samples are updated to the initial risk rule base to obtain the risk rule base for the work scenario.

[0071] Specifically, the process involves parsing a massive amount of safety regulations and documents, including national, industry, and enterprise-level power safety work procedures. Using technologies such as natural language processing and knowledge engineering, textual rules and regulations, such as the requirement to wear safety belts when working at heights and the prohibition of non-working personnel from entering fenced areas, are transformed into structured logical statements. An initial risk level is pre-assigned to each rule, forming an initial risk rule base.

[0072] In detail, after a risk warning signal is issued during actual operation and it is confirmed to be a real risk event, all data associated with the warning signal is automatically packaged and collected as a feedback sample.

[0073] Furthermore, upon receiving new feedback samples, the offline training module is activated to update the initial risk rule base with the collected feedback samples. These new samples are used as incremental training data to optimize model parameters and improve the accuracy of identifying similar risks in the future. New risk patterns not covered by the initial rules are discovered from the feedback samples, and new rule entries are automatically generated, resulting in a more powerful risk rule base that is closer to actual operational scenarios.

[0074] S6: Send the risk warning signal to the PTZ camera in the work scene, and drive the PTZ camera to monitor the close-up monitoring screen of the risk target area in real time.

[0075] In this embodiment of the invention, the step of sending the risk warning signal to the PTZ camera in the work scene and driving the PTZ camera to monitor the close-up surveillance footage of the risk target area in real time includes: Based on the risk warning signal, extract the risk target information of the work scenario; Control commands for the PTZ camera are generated based on the risk target information and the risk target area. The control command is sent to the PTZ camera to monitor a close-up view of the risk target area.

[0076] In detail, the structured early warning signal is received and parsed. The most critical risk target information is extracted, mainly including the target ID that needs to be focused on, the risk level of each target, and the precise coordinates of each target in the panoramic view. These targets are then sorted according to their risk levels to determine the priority order of response.

[0077] Specifically, based on the extracted risk target information and risk target area, control commands for the PTZ camera are generated. According to pre-calibrated parameters, the pixel coordinates of the target on the 2D panoramic image are converted into three-dimensional spatial control commands for the PTZ camera, namely horizontal rotation angle, vertical tilt angle, and lens zoom ratio. The zoom ratio is automatically calculated based on the size and distance of the target to ensure clear, full-frame close-ups. These parameters are ultimately encapsulated into control commands conforming to standard protocols.

[0078] Furthermore, control commands are sent to the PTZ camera via the internal bus. Upon receiving the commands, the camera hardware immediately drives the motors to precisely perform rotation, tilt, and zoom movements. At this time, the video stream transmitted to the monitoring center automatically switches from a panoramic view to a close-up view from the camera. This provides managers with crucial visual evidence for rapid confirmation of risk events, detailed evidence collection, and process locking, improving the efficiency and accuracy of risk management.

[0079] For example, when the PTZ control module of a global-area device receives a Level 1 emergency risk warning signal, it immediately parses the data packets. The module extracts information about the risk targets and sorts them according to risk level.

[0080] Furthermore, the primary target and its coordinates are locked, and gimbal control commands are generated accordingly. Calculations indicate a need for 35 degrees of horizontal rotation and -15 degrees of vertical tilt. A 25x zoom command is set, which is within the device's 30x optical zoom capability. The entire set of commands is encapsulated into a protocol data packet, while control commands for the next target are pre-generated and queued.

[0081] Furthermore, the packaged instructions were sent to the pan-tilt camera of the all-area device. Thanks to its 3D PTZ control capabilities, the camera precisely rotated and locked onto the target. Its high-performance 5.4-108mm lens group began zooming, quickly reaching a 25x magnification effect. On the large screen in the monitoring center, the original panoramic view was replaced by an ultra-high-definition close-up image from a 4.09 million-pixel sensor: the image clearly showed that the connecting bolts of the A-phase winding of the main transformer were glowing red-hot, accompanied by wisps of smoke. After locking onto the image for 3 seconds to collect evidence, the next instruction in the queue was automatically executed, and the camera instantly turned to capture a clear close-up of the person working in the hazardous area, completing a comprehensive, dynamic, and in-depth visual verification of the entire complex risk event.

[0082] In this embodiment of the invention, the step of sending the risk warning signal to the PTZ camera in the work scene and driving the PTZ camera to monitor the close-up surveillance footage of the risk target area in real time further includes: A risk assessment is performed on the close-up surveillance footage to determine its risk status. The risk status is compared with the risk warning signal to obtain the feedback information of the close-up monitoring screen; The control commands for the gimbal camera are regenerated based on the feedback information.

[0083] Specifically, unlike the general risk identification performed on a panoramic view, this stage invokes specialized analysis models for specific objects. For example, for a close-up of equipment, a model might be run to identify specific fault types, such as loose bolts, oil leaks, or damaged insulators; for a close-up of personnel, facial recognition, micro-expression analysis, or more sophisticated motion recognition, such as whether tools are being used improperly, might be used.

[0084] In detail, the risk status obtained from the secondary assessment is compared with the initial risk warning signal to obtain feedback information. The comparison results may be: confirmation and escalation, close-up footage confirms the risk and finds the situation to be more serious; confirmation and specification, confirmation of the risk and clarification of the specific cause; correction, finding that the actual risk does not match the initial judgment; or resolution, close-up footage confirms it is a false alarm.

[0085] Furthermore, based on the feedback information, a new set of more targeted control commands for the PTZ camera is generated. This upgrades the camera's behavior from simply arriving and shooting to intelligently observing and adapting.

[0086] For example, after the 30x optical zoom lens of the all-area device locked onto the overheated area of ​​phase A of the main transformer, a secondary risk assessment was immediately conducted on the high-definition close-up footage captured by a 4.09 million-pixel sensor. A specialized thermal fault diagnosis model was invoked, which, by analyzing the shape and gradient of the hot spot, determined a new risk status: it was confirmed that the contact bolts between the connecting clamp and the equipment bushing were loose, resulting in excessive contact resistance and overheating, accompanied by visible smoke.

[0087] Furthermore, the risk status of the secondary assessment—loose bolts causing heat and smoke—was compared with the initial risk warning signal of severe overheating of the core equipment. The feedback from this comparison was: Level 1 risk confirmed, the cause of the risk specified as loose bolts, and the risk level increased to extremely critical due to the presence of smoke. This feedback clearly indicates that the initial judgment was correct, but the degree of danger was higher than expected.

[0088] In detail, if the feedback is an escalation of the risk, the new instruction may be to keep the camera locked and continue recording evidence; if the feedback is a risk correction, the new instruction may be to fine-tune the gimbal focus to the new risk point; if the risk of one target has been dealt with, the new instruction is to move on to the next risk target.

[0089] like Figure 2The diagram shown is a functional module diagram of a work scene monitoring system based on panoramic stitching and PTZ linkage provided in an embodiment of the present invention.

[0090] The work scene monitoring system 100 based on panoramic stitching and PTZ linkage described in this invention can be installed in an electronic device. Depending on the functions implemented, the work scene monitoring system 100 based on panoramic stitching and PTZ linkage may include a data acquisition module 101, an electrical malfunction characteristic simulation module 102, a time-series deduction module 103, a deviation information calculation module 104, a risk warning module 105, and a PTZ control module 106. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0091] In this embodiment, the functions of each module / unit are as follows: The data acquisition module 101 is used to acquire real-time panoramic video data and historical video data of the work scene, as well as partial discharge signals of high-voltage electrical equipment. The electrical malfunction feature simulation module 102 is used to simulate the electrical malfunction features of the high-voltage electrical equipment based on the environmental parameters of the partial discharge signal and the real-time panoramic video data. The time-series extrapolation module 103 is used to perform time-series extrapolation on the real-time panoramic video data based on the historical video data and the electrical malfunction characteristics to obtain the operation prediction information of the real-time panoramic video. The deviation information calculation module 104 is used to perform deviation calculation on the job prediction information and the real-time panoramic video data to obtain the deviation information of the real-time panoramic video. The risk warning module 105 is used to locate the risk target area of ​​the work scenario based on the deviation information, perform risk judgment on the deviation information based on a preset risk rule base, and obtain a risk warning signal for the work scenario. The PTZ control module 106 is used to send the risk warning signal to the PTZ camera in the work scene and drive the PTZ camera to monitor the close-up monitoring screen of the risk target area in real time.

[0092] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0093] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0095] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0096] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for monitoring work scenarios based on panoramic stitching and PTZ linkage, characterized in that, The method includes: S1: Acquire real-time panoramic video data and historical video data of the work scene, as well as partial discharge signals of high-voltage electrical equipment; S2: Simulate the electrical malfunction characteristics of the high-voltage electrical equipment based on the environmental parameters of the partial discharge signal and the real-time panoramic video data. The environmental parameters of the real-time panoramic video data include the real-time temperature and humidity of the work site obtained by reading the readings of physical instruments in the picture. S3: Based on the historical video data and the electrical fault characteristics, perform time-series extrapolation on the real-time panoramic video data to obtain the operation prediction information of the real-time panoramic video; S4: Calculate the deviation between the job prediction information and the real-time panoramic video data to obtain the deviation information of the real-time panoramic video; S5: Based on the deviation information, locate the risk target area of ​​the work scenario, and perform risk judgment on the deviation information based on the preset risk rule base to obtain the risk warning signal of the work scenario; S6: Send the risk warning signal to the PTZ camera in the work scene, and drive the PTZ camera to monitor the close-up monitoring screen of the risk target area in real time; The simulation of electrical malfunction characteristics of the high-voltage electrical equipment based on environmental parameters derived from the partial discharge signal and the real-time panoramic video data includes: The partial discharge characteristics of the partial discharge signal are analyzed, and the environmental parameters of the real-time panoramic video data are identified. A dynamic mapping relationship for the high-voltage electrical equipment is constructed based on the partial discharge characteristics and the environmental parameters. The electrical malfunction characteristics of the high-voltage electrical equipment are determined based on the dynamic mapping relationship and the physical structure of the high-voltage electrical equipment.

2. The method for monitoring work scenarios based on panoramic stitching and PTZ linkage as described in claim 1, characterized in that, The acquisition of real-time panoramic video data and historical video data of the work scene, as well as partial discharge signals of high-voltage electrical equipment, includes: The real-time video streams from multiple different angles in the work scene are collected and stitched together to form real-time panoramic video data of the work scene; Retrieve historical video data of the aforementioned work scenario; The non-intrusive signals of the high-voltage electrical equipment are monitored, and the non-intrusive signals are identified as partial discharge signals of the high-voltage electrical equipment.

3. The method for monitoring work scenarios based on panoramic stitching and PTZ linkage as described in claim 1, characterized in that, The step of performing time-series extrapolation on the real-time panoramic video data based on the historical video data and the electrical fault characteristics to obtain the operation prediction information of the real-time panoramic video includes: Analyze the dynamic change characteristics of the historical video data; The dynamic change features and the electrical malfunction features are combined to form the scene state features of the work scenario; The state characteristics of the scenario are predicted in time series based on a pre-trained long short-term memory network to obtain the state evolution trend of the task scenario. Based on the state evolution trend, a forward-looking extrapolation is performed on the real-time panoramic video to obtain the operation prediction information of the real-time panoramic video.

4. The operation scene monitoring method based on panoramic stitching and PTZ linkage as described in claim 3, characterized in that, The calculation formula for the scene state features includes: in, These are the scene state features. yes The dynamic change characteristics described at any given time. It refers to the physical structure of the high-voltage electrical equipment. These are the environmental parameters. It is the temperature change rate of the historical video data. It is the discharge change rate of the historical video data. yes The time complexity coefficient, It is a time marker.

5. The method for monitoring work scenarios based on panoramic stitching and PTZ linkage as described in claim 1, characterized in that, The process of locating the risk target area of ​​the work scenario based on the deviation information, determining the risk of the deviation information based on a preset risk rule base, and obtaining a risk warning signal for the work scenario includes: Based on the deviation information, a deviation heatmap of the work scenario is drawn, and the risk target area of ​​the work scenario in the deviation heatmap is located. The overall risk level of the work scenario is matched based on the deviation information and the preset risk rule base; A risk warning signal for the work scenario is generated based on the comprehensive risk level.

6. The method for monitoring work scenarios based on panoramic stitching and PTZ linkage as described in claim 5, characterized in that, The construction of the preset risk rule base includes: The safety regulations in the aforementioned work scenario will be used as the initial risk rule base for that work scenario. Collect risk events associated with the risk warning signals as feedback samples; The feedback samples are updated to the initial risk rule base to obtain the risk rule base for the work scenario.

7. The method for monitoring work scenarios based on panoramic stitching and PTZ linkage as described in claim 1, characterized in that, The step of sending the risk warning signal to the PTZ camera in the work scene and driving the PTZ camera to monitor the close-up surveillance footage of the risk target area in real time includes: Based on the risk warning signal, extract the risk target information of the work scenario; Control commands for the PTZ camera are generated based on the risk target information and the risk target area. The control command is sent to the PTZ camera to monitor a close-up view of the risk target area.

8. The method for monitoring work scenarios based on panoramic stitching and PTZ linkage as described in claim 7, characterized in that, The step of sending the risk warning signal to the PTZ camera in the work scene and driving the PTZ camera to monitor the close-up surveillance footage of the risk target area in real time also includes: A risk assessment is performed on the close-up surveillance footage to determine its risk status. The risk status is compared with the risk warning signal to obtain the feedback information of the close-up monitoring screen; The control commands for the gimbal camera are regenerated based on the feedback information.

9. A work scene monitoring system based on panoramic stitching and PTZ linkage, characterized in that, The system includes: The data acquisition module is used to acquire real-time panoramic video data and historical video data of the work scene, as well as partial discharge signals of high-voltage electrical equipment. The electrical malfunction characteristic simulation module is used to simulate the electrical malfunction characteristics of the high-voltage electrical equipment based on the partial discharge signal and the environmental parameters of the real-time panoramic video data. The environmental parameters of the real-time panoramic video data include the real-time temperature and humidity of the work site obtained by reading the readings of physical instruments in the picture. The time-series extrapolation module is used to perform time-series extrapolation on the real-time panoramic video data based on the historical video data and the electrical malfunction characteristics to obtain the operation prediction information of the real-time panoramic video. The deviation information calculation module is used to calculate the deviation between the job prediction information and the real-time panoramic video data to obtain the deviation information of the real-time panoramic video. The risk warning module is used to locate the risk target area of ​​the work scenario based on the deviation information, determine the risk of the deviation information based on a preset risk rule base, and obtain a risk warning signal for the work scenario. The PTZ control module is used to send the risk warning signal to the PTZ camera in the work scene and drive the PTZ camera to monitor the close-up monitoring screen of the risk target area in real time. The simulation of electrical malfunction characteristics of the high-voltage electrical equipment based on environmental parameters derived from the partial discharge signal and the real-time panoramic video data includes: The partial discharge characteristics of the partial discharge signal are analyzed, and the environmental parameters of the real-time panoramic video data are identified. A dynamic mapping relationship for the high-voltage electrical equipment is constructed based on the partial discharge characteristics and the environmental parameters. The electrical malfunction characteristics of the high-voltage electrical equipment are determined based on the dynamic mapping relationship and the physical structure of the high-voltage electrical equipment.

Images