A substation construction site collaborative supervision method and system based on space-time prediction and viewpoint complementation

Abstract

A substation construction site collaborative supervision method and system based on space-time prediction and viewpoint complementation are disclosed, which uses a high-positioned fixed panoramic camera to collect video streams, identifies dynamic personnel targets and extracts their panoramic positions and appearance features; a homography matrix mapping and Kalman filtering algorithm are used to construct the motion state vector of the target and predict its future trajectory; the optimal space-time interception point is solved in combination with the motion capability of a quadruped robot, and the robot is dispatched to implement dynamic interception; after the robot arrives, images are collected by a local camera, and cross-view identity locking is completed by calculating the cosine similarity of the site features and the panoramic appearance features; fine compliance detection is performed on the locked target. Through dynamic and static collaboration and feature relay mechanism, the invention effectively solves the problems of blind area in fixed monitoring, robot response lag and easy loss of cross-view targets, and realizes full-time, dead-angle-free accurate supervision in the complex environment of a substation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robot control technology, and in particular to a collaborative monitoring method and system for substation construction sites based on spatiotemporal prediction and viewpoint complementarity. Background Technology

[0002] Substation renovation and expansion projects involve complex on-site environments, including high-voltage equipment areas, high-altitude operations, and cross-disciplinary construction, resulting in immense pressure for safety supervision. Traditional supervision methods, primarily relying on manual inspections and fixed-point video surveillance, have significant shortcomings. Limitations of fixed surveillance: Although fixed cameras have a wide coverage area, they have blind spots and obstruction problems. Furthermore, due to their high installation position, it is difficult to obtain construction details (such as whether the safety buckles are properly attached, instrument readings, etc.), which cannot meet the needs of refined supervision.

[0003] Limitations of mobile robots: Although quadruped robots are flexible and can get close to observe, they are limited by battery life and cannot achieve 24-hour full coverage patrol; moreover, the robots have a narrow field of vision and it is difficult to detect sudden violations on a macro level.

[0004] The pain points of existing collaborative technologies: Existing "fixed + mobile" linkage solutions typically employ a simple coordinate guidance mode (i.e., after a fixed camera detects a target, it sends its current coordinates to the robot). However, in large substation scenarios, there is a time lag between the robot receiving the instruction and moving to the target location, and the dynamic target often has already moved from its original position, causing the robot to "miss its target".

[0005] The challenge of cross-view recognition: Fixed cameras typically have a high, downward-looking perspective, while robots have a level or upward-looking perspective. The appearance of the same person varies greatly from different perspectives, making it difficult for the robot to identify the target to be monitored once it arrives on site. This can easily lead to target tracking loss or mismatch of identities.

[0006] Therefore, there is an urgent need for a collaborative regulatory method that can predict the trajectory of dynamic targets and solve the problem of cross-perspective identity verification. Summary of the Invention

[0007] To address the aforementioned technical problems in the existing technology, this invention proposes a collaborative monitoring method and system for substation construction sites based on spatiotemporal prediction and viewpoint complementarity, in order to solve the above-mentioned technical problems.

[0008] According to a first aspect of the present invention, a collaborative monitoring method for substation construction sites based on spatiotemporal prediction and viewpoint complementarity is proposed, comprising: S1: Real-time monitoring video streams are collected using fixed panoramic cameras deployed high up in the substation. A lightweight target detection algorithm is used to identify dynamic personnel targets in the construction area. The position information of the dynamic personnel targets in the panoramic image coordinate system is extracted, and the visual appearance feature vector of the target is extracted simultaneously. S2: Based on the pre-calibrated homography matrix, the position information in the panoramic image coordinate system is mapped to the geographic coordinates in the substation world coordinate system; the Kalman filter algorithm is used to process the time series of geographic coordinates to construct the motion state vector of dynamic personnel targets. The motion state vector includes at least the current position, motion velocity vector and acceleration vector. S3: Obtain the current position and maximum speed of the quadruped robot in standby or patrol state; based on the motion state vector of the dynamic human target and the motion capability of the quadruped robot, solve for the optimal spatiotemporal interception point, and generate a navigation path command from the current position of the quadruped robot to the optimal spatiotemporal interception point and send it to the quadruped robot. S4: After the quadruped robot reaches the optimal spatiotemporal interception point, it uses its onboard local detail camera to collect images of people on site and extract site feature vectors; it calculates the cosine similarity between the site feature vectors and the visual appearance feature vectors issued in step S1. S5: If the cosine similarity is greater than the preset identity verification threshold, the target is successfully locked. The quadruped robot adjusts the focal length and gimbal angle of the local detail camera according to the preset violation detection logic to perform high-resolution acquisition and compliance analysis of the construction details of the locked target.

[0009] In some specific embodiments, the process of constructing the motion state vector specifically includes: establishing state equations. ,in, for Target state vector at time 1 , Here is the state transition matrix. The process noise vector is defined; the observation equation is established. ,in, These are the geographic coordinate observations obtained by mapping in S2. Represents the observation matrix. The observation noise vector is represented; the target's velocity vector is corrected in real time using the prediction and update steps of Kalman filtering. This is to eliminate positioning noise caused by wide-angle distortion in fixed panoramic cameras.

[0010] In some specific embodiments, in S3, the method for solving the optimal spatiotemporal interception point is as follows: constructing an encounter time function. ,in, As candidate interception points, This is the robot's current position. The target's current position, and Let be the scalar velocities of the robot and the target, respectively; under the constraints of the passable road network of the substation, solve for... Minimized position point As the optimal spatiotemporal interception point, it ensures that the quadruped robot completes the interception before the target leaves the current monitoring area.

[0011] In some specific embodiments, in S1 and S4, the extraction of visual appearance feature vectors adopts a deep twin network with shared weights: a fixed panoramic camera acquires images from a high top-down angle, and a local detail camera acquires images from a level or bottom-up angle; the deep twin network introduces a viewpoint transfer loss function during the training phase, and maps pedestrian images from different viewpoints to a unified feature space through an adversarial generative network, so that the feature vector distance of the same target under top-down and level-up views is less than the feature vector distance of different targets under the same viewpoint.

[0012] In some specific embodiments, a priority scheduling strategy based on the risk potential energy field is also included: a static risk map of the substation is pre-constructed, and the energized area and high-altitude work area are marked as high potential energy areas; the probability of a dynamic personnel target entering the high potential energy area at a future time is calculated by combining the motion state vector in S2; when there are multiple dynamic personnel targets, the quadruped robot is prioritized to intercept the target with the highest probability of entering the high potential energy area.

[0013] In some specific embodiments, an active compensation mechanism based on line-of-sight occlusion is also included: when a fixed panoramic camera detects that a dynamic human target is occluded by on-site equipment, resulting in a decrease in the confidence of feature extraction, the spatial geometric relationship between the occluder and the target is calculated; the optimal compensation observation point instruction is generated and sent to the quadruped robot, instructing the quadruped robot to move to a side-rear position that can avoid the occluder and maintain a safe distance from the target for feature acquisition.

[0014] In some specific embodiments, in S5, the compliance analysis specifically includes: the quadruped robot using the high-magnification zoom function of the local detail camera to crop images of specific body parts of the locked target; inputting the cropped image into a classification model trained for fine-grained violations to identify behaviors including not fastening seat belt buckles, not wearing helmets correctly, and using mobile phones in violation of regulations; if a violation is identified, uploading the combined evidence chain containing panoramic and local detail images to the management platform.

[0015] In some specific embodiments, the quadruped robot receives motion state vectors updated by S2 in real time during its movement and uses model predictive control algorithms to dynamically adjust its own trajectory in order to cope with the random turning or speed change behavior of dynamic human targets.

[0016] According to a second aspect of the invention, a computer-readable storage medium is provided on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0017] According to a third aspect of the present invention, a collaborative monitoring system for substation construction sites based on spatiotemporal prediction and viewpoint complementarity is proposed to implement the method described above, comprising: The panoramic perception subsystem includes multiple low-power fixed cameras distributed at the highest point of the substation, and is equipped with an edge computing module for performing target detection and feature fingerprint extraction. The collaborative management and control cloud platform is equipped with a coordinate mapping module, a Kalman filter motion prediction module, and an interception scheduling engine, which are used to calculate the optimal spatiotemporal interception point and issue scheduling instructions. The mobile monitoring subsystem includes at least one quadruped robot equipped with a dual-light gimbal camera and an onboard AI computing unit, used to perform cross-view feature comparison and refined compliance inspection.

[0018] This invention proposes a collaborative monitoring method and system for substation construction sites based on spatiotemporal prediction and viewpoint complementarity, which has the following advantages compared with existing technologies: This invention introduces Kalman filtering to construct the target's motion state vector (velocity, acceleration) and calculates the "optimal spatiotemporal interception point" instead of simply the current position. This enables the robot to predict the personnel's movement path and intercept them, significantly improving the success rate of tracking and monitoring mobile construction workers, and avoiding the passive situation of the robot chasing after the target.

[0019] To address the significant difference in perspective between a fixed camera (top-down view) and a robotic camera (eye-level view), this invention employs a deep Siamese network with a viewpoint transfer loss function to extract features, and uses cosine similarity for feature relay. This ensures that the person captured by the robot is indeed a suspicious person detected by the fixed camera, forming a tight logical loop and effectively preventing false alarms and missed alarms.

[0020] By introducing a risk potential energy field strategy, intrusion events in high-risk areas (such as electrified areas) can be prioritized, enabling intelligent allocation of regulatory resources. At the same time, based on the active compensation mechanism for line-of-sight obstruction, the robot can intelligently detour to the side and rear of the obstruction for observation, solving the pain point of fixed cameras not being able to see the whole picture or see clearly.

[0021] By combining global trajectory information from a fixed camera with high-resolution local details of the robot (such as close-ups of unfastened seatbelts), a combined chain of evidence is generated. This not only enables accurate determination of violations but also provides irrefutable video evidence for subsequent accountability.

[0022] By using model predictive control algorithms, the robot can dynamically respond to changes in the target's speed and direction during movement, ensuring smooth tracking. At the same time, the event-triggered scheduling mode avoids ineffective patrols by the robot, significantly extending the effective duration of a single operation. Attached Figure Description

[0023] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Other features, objects, and advantages of this application will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of a method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity, which is an embodiment of this application; Figure 3 This is an architecture diagram of a substation construction site collaborative monitoring system based on spatiotemporal prediction and viewpoint complementarity, which is a specific embodiment of this application. Figure 4 This is a schematic diagram of the structure of a computer system used to implement the electronic device of the present application. Detailed Implementation

[0024] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Figure 1 An exemplary system architecture 100 is shown, which can be applied to a collaborative monitoring method for substation construction sites based on spatiotemporal prediction and viewpoint complementarity according to embodiments of this application.

[0027] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0028] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications can be installed on terminal devices 101, 102, and 103, such as data processing applications, data visualization applications, and web browser applications.

[0029] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0030] Server 105 can be a server that provides various services, such as a background information processing server that supports the mapping table data displayed on terminal devices 101, 102, and 103. The background information processing server can process the obtained logical address and generate processing results.

[0031] It should be noted that the method provided in this application embodiment can be executed by server 105 or by terminal devices 101, 102, and 103. The corresponding device is generally set in server 105 or can be set in terminal devices 101, 102, and 103.

[0032] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0033] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0034] Figure 2 A flowchart illustrating a collaborative monitoring method for substation construction sites based on spatiotemporal prediction and viewpoint complementarity, according to an embodiment of this application, is shown. Figure 2 As shown, the method includes the following steps: S1: Real-time monitoring video streams are collected using fixed panoramic cameras deployed high up in the substation. Dynamic personnel targets in the construction area are identified through a lightweight target detection algorithm. The position information of the dynamic personnel targets in the panoramic image coordinate system is extracted, and the visual appearance feature vector of the target is extracted simultaneously.

[0035] In a specific embodiment, a fixed panoramic camera (such as a high-definition monitoring PTZ camera with a 45° tilt angle) deployed high up in the substation is used to acquire real-time monitoring video streams. The video streams are processed by an edge computing unit, which runs a lightweight target detection algorithm (such as YOLOv8-Nano) to identify dynamic personnel targets within the construction area and outputs the coordinates of the center point of the bottom edge of the detection frame in the pixel coordinate system. .

[0036] In a specific embodiment, the visual appearance feature vector of the target is extracted. This process employs a panoramic branch of a deep Siamese Network with shared weights. The network architecture uses ResNet-50 as the backbone. Training strategy: During offline training, the network introduces a viewpoint transfer loss function. This is specifically implemented through an adversarial generative network architecture, where the generator attempts to transfer different viewpoints (top-down view, bottom-up view, top-down ... eye level The image is mapped to a unified feature space, and the discriminator attempts to distinguish the sources of the features. Loss function: ,in, The symbol represents the expected value, which is usually calculated by averaging the samples in a batch during actual training. The mapping function represents the feature encoder (i.e., the forward propagation process of the deep Siamese network, with the output being a feature vector). This represents the output probability of the viewpoint discriminator, with a value range of [0, 1]. By minimizing this loss, we ensure that the extracted... It is invariant to changes in viewpoint and serves as a digital fingerprint for subsequent cross-viewpoint comparisons.

[0037] S2: Based on the pre-calibrated homography matrix, the position information in the panoramic image coordinate system is mapped to the geographic coordinates in the substation world coordinate system; the Kalman filter algorithm is used to process the time series of geographic coordinates to construct the motion state vector of dynamic personnel targets. The motion state vector includes at least the current position, motion velocity vector and acceleration vector.

[0038] In a specific embodiment, Kalman filtering is used to eliminate wide-angle distortion noise from the panoramic camera and obtain an accurate velocity vector. A pre-calibrated homography matrix is ​​then used. , pixel coordinates Mapped to geographic coordinate observations in the substation's world coordinate system ; Set goals in Motion state vector at time 1 State equation: ,in, Here is the state transition matrix. Sampling time, The noise is the process noise, which follows a mean of 0 and a covariance of . The multivariate Gaussian distribution, i.e. This reflects the random perturbations experienced by the target during its motion. Observation equation: ,in, For the observation matrix, To observe the noise, it follows a pattern with a mean of 0 and a covariance of . The multivariate Gaussian distribution, i.e. This reflects the positioning and mapping errors of the panoramic camera. A Kalman filter prediction-update loop is used to output the corrected motion velocity vector in real time. .

[0039] S3: Obtain the current position and maximum speed of the quadruped robot in standby or patrol mode; based on the motion state vector of the dynamic human target and the quadruped robot's motion capabilities, solve for the optimal spatiotemporal interception point, and generate a navigation path command from the quadruped robot's current position to the optimal spatiotemporal interception point, which is then sent to the quadruped robot. This step is the decision-making center of the system, integrating risk assessment, occlusion handling, and interception calculation.

[0040] In a specific implementation, regulatory priorities are determined before calculating interception points. A static risk map of the substation is pre-constructed, marking energized areas and high-altitude work areas as high-potential zones. Combining the state vector in S2 (Position and velocity), calculate the dynamic target's entry at future moments using Gaussian distribution integration. probability When multiple targets are within the field of view, prioritize locking onto them. The highest target is then used for subsequent interception calculations.

[0041] In specific embodiments, active compensation for line-of-sight occlusion is also included. It determines whether the target is obstructed. If the fixed camera detects that the target is obstructed by a transformer or wall (manifested as a sudden drop in detection confidence or an incomplete target frame), the spatial geometric relationship between the obstruction and the target is calculated. Compensation instructions are generated: instead of calculating a conventional interception point, a position approximately 3-5 meters to the side and rear of the target that avoids the obstruction is calculated as the optimal compensation observation point. If there is no obstruction, the optimal spatiotemporal interception point is calculated. Get the robot's current position. Maximum speed and target location ,speed Construct the meeting time function: Solve the optimization problem: The system generates from arrive The navigation task (or the best replacement observation point) is issued to the robot, whereby... This represents any candidate location point within the substation space. This represents the set of accessible networks within the substation, which excludes static obstacles such as transformers and fences, ensuring that the solved interception points are physically reachable by the robot. This represents the variable value that minimizes the objective function, i.e., finding a specific point. This makes the meeting time Minimum.

[0042] In a specific embodiment, if the calculated minimum interception time If the time it takes for the target to leave the monitored area is greater than the expected time, or if the energy consumption of the interception path exceeds the robot's remaining battery power, the system will automatically switch to long-distance zoom tracking mode. In this mode, the robot does not move and directly uses the maximum magnification of the gimbal camera to capture images from a distance, marking the target as an unverified risk.

[0043] S4: After the quadruped robot reaches the optimal spatiotemporal interception point, it uses its onboard local detail camera to collect images of people on site and extract site feature vectors; it calculates the cosine similarity between the site feature vectors and the visual appearance feature vectors issued in step S1.

[0044] In a specific embodiment, the quadruped robot travels to... During the process, a rigid path is not followed. The target status is received in real-time from the cloud. Establish the discrete kinematic equations of the robot. Construct a quadratic programming function that includes an interception error term (approaching the interception point during movement), a control smoothing term (reducing sudden stops and starts), and an energy consumption term. In each control cycle (e.g., 100ms), solve for the optimal velocity sequence for the next N steps, executing only the first step. This enables the robot to flexibly respond to sudden turns or speed changes by personnel.

[0045] S5: If the cosine similarity is greater than the preset identity verification threshold, the target is successfully locked. The quadruped robot adjusts the focal length and gimbal angle of the local detail camera according to the preset violation detection logic to perform high-resolution acquisition and compliance analysis of the construction details of the locked target.

[0046] In a specific embodiment, after the robot reaches the designated location, it uses a local detail camera (dual-light pan-tilt head) to capture a planar image of the personnel on site. The same deep Siamese network as S1 is used to extract the on-site feature vector. Calculate cosine similarity: ,in, Represents the on-site feature vector, if The identity verification threshold (e.g., 0.75) determines that the identity uncertainty caused by the large difference in perspective has been eliminated, and the target has been successfully locked.

[0047] In a specific implementation, compliance analysis includes: after target locking, the robot automatically adjusts the camera's optical zoom based on the target distance and adjusts the gimbal to center the target in the image. Specific parts of the target (head, waist, hands) are located based on human pose estimation. The cropped image is input into a dedicated classification model to identify the following behaviors: Seatbelt detection: identifying whether the buckle is closed or suspended in the air. Helmet detection: identifying whether it is worn and whether the chin strap is fastened. Violation behavior: identifying whether smoking or using a mobile phone. Once a violation is identified, a panoramic image (showing the environmental location) and a close-up image (showing the details of the violation) are packaged and uploaded to the management platform to complete closed-loop supervision.

[0048] Figure 3 This paper illustrates an architecture diagram of a substation construction site collaborative monitoring system based on spatiotemporal prediction and viewpoint complementarity, according to a specific embodiment of this application. Figure 3 As shown, the system adopts a layered distributed architecture design, mainly consisting of three parts: panoramic perception subsystem 310, collaborative management and control cloud platform 320, and mobile monitoring subsystem 330. Each part interacts with data through a high-speed industrial network (such as gigabit Ethernet, 5G or WIFI-6 private network).

[0049] In a specific embodiment, the panoramic perception subsystem 310 serves as the perception layer, deployed at the physical site of the substation, primarily responsible for all-weather monitoring and primary data processing over a wide area. It includes a fixed panoramic camera 311: deployed at a high point within the substation (such as the top of a structure, lightning rod tower, or dedicated monitoring pole), employing a wide-angle or fisheye lens to capture real-time video streams covering the construction area. Its installation height is typically 8-15 meters to obtain a high-angle, macroscopic view and reduce blind spots. An edge computing module 312: directly connected to or integrated within the fixed panoramic camera 311. This module incorporates a lightweight AI acceleration chip (such as an NPU or GPU) and runs a lightweight target detection algorithm (such as YOLO-Nano). Its functions include: real-time identification of dynamic personnel targets in the video; and extraction of the target's position information in the image coordinate system. Extract the visual appearance feature vector of the target (i.e., ReID fingerprint) and upload the structured data to the cloud platform.

[0050] In a specific embodiment, the collaborative management and control cloud platform 320 serves as the decision-making layer. Deployed on the substation's main control room server or the private cloud of the centralized control center, this platform is the brain of the entire system, responsible for multi-source data fusion and core scheduling decisions. It includes: a coordinate mapping module 321: storing a pre-calibrated homography matrix, receiving image pixel coordinates from the edge computing module 312, and mapping them into unified world geographic coordinates for the substation. The Kalman filter motion prediction module 322 filters the time series of geographic coordinates to eliminate positioning noise and constructs a target motion state vector containing position, velocity, and acceleration vectors to predict the target's future trajectory. The interception scheduling engine 323 calculates the "optimal spatiotemporal interception point" based on the target's motion state and the robot's motion capabilities by solving the encounter time function. Subsequently, it generates navigation path instructions and the target's feature fingerprint and sends them to the mobile monitoring subsystem. The collaborative management cloud platform 320 may also include a risk potential energy field analysis module, used to store static risk maps of substations (marking energized areas, high-altitude work areas, etc.), and, combined with the target's motion prediction results, calculates the probability of dynamic personnel entering high-risk areas, thereby determining the priority of monitoring tasks; and a line-of-sight occlusion analysis module, which calculates the coordinates of the best replacement observation point to avoid obstacles when a target is detected to be occluded, assisting the scheduling engine in generating detour instructions.

[0051] In a specific embodiment, the mobile surveillance subsystem 330 serves as the execution layer. This subsystem mainly consists of at least one quadruped robot 331, responsible for performing close-range reconnaissance and refined surveillance tasks. The quadruped robot body 331, as a mobile carrier, possesses omnidirectional mobility and adaptability to complex terrain; it can receive navigation commands from the cloud and use its own navigation algorithm to plan a local path to the interception point. The robot's back is equipped with a dual-light gimbal camera, integrating a visible light zoom camera and an infrared thermal imaging camera, used to acquire images of personnel at eye level or from below after the robot arrives at the scene. The robot is equipped with an onboard AI computing unit, specifically a high-performance computing unit (such as NVIDIA Jetson Xavier), which runs the following core algorithms: MPC motion control algorithm, which receives real-time updates of the target status from the cloud and dynamically adjusts the robot's speed and trajectory to cope with changes in the target's speed or direction; cross-view feature comparison module, which uses a deep twin network to extract features from the scene image and compares them with the panoramic appearance features sent from the cloud using cosine similarity to confirm the target's identity; and a refined compliance detection module, which performs fine-grained analysis on high-definition close-up images after the target is locked to identify specific violations such as not wearing a seat belt or helmet.

[0052] In a specific embodiment, the system workflow forms the following closed loop: the panoramic perception subsystem 310 detects anomalies through video streams and extracts features for uploading; the collaborative management and control cloud platform 320 performs spatiotemporal prediction and issues interception commands and feature fingerprints; the mobile monitoring subsystem 330 performs interception, identity verification and evidence collection, and finally sends the combined evidence chain containing panoramic distant view and local close-up view back to the cloud platform for archiving and alarm.

[0053] To more intuitively illustrate the workflow and technical effects of this invention, the following description uses a specific substation construction monitoring scenario. Scenario Background: A 500kV substation is undergoing renovation and expansion. Fixed panoramic cameras (installed at a height of 12 meters) are deployed throughout the station, and a quadruped robot (located at the charging pile) is deployed on the ground in standby mode. The key monitoring area is the energized bay area near the No. 1 main transformer. The specific implementation process is as follows: Anomaly Detection: At 10:15 AM, a fixed panoramic camera detected that a construction worker, A, suddenly deviated from the designated path and ran rapidly towards the energized area of ​​the No. 1 main transformer. The edge computing module immediately extracted the panoramic visual feature vector of worker A. (Feature fingerprint).

[0054] Intent prediction: The system maps pixel coordinates to geographic coordinates and calculates the current speed of person A as 3 m / s through Kalman filtering, pointing directly at the gap in the electrified area fence.

[0055] Risk assessment: The system calculation found that the probability of person A entering the high potential energy region (charged area) is over 90%, triggering the highest priority scheduling.

[0056] Interception Calculation: The system did not instruct the robot to go to person A's current coordinates (because people run very fast, going to the current coordinates would result in missing), but instead calculated the optimal spatiotemporal interception between person A and the robot's trajectory. It is located 5 meters in front of the gap in the fence of the electrified area.

[0057] Dynamic interception and backup: The system schedules a quadruped robot to run at full speed (e.g., 2.5 m / s) to... .

[0058] Active positioning: At this moment, the system detects that construction worker A is about to be obstructed by a circuit breaker cabinet from the panoramic camera's perspective. The system automatically fine-tunes the robot's positioning commands, instructing the robot to move to a position with a clear view behind and to the side of the circuit breaker.

[0059] Identity verification: After the robot arrives at the location, the dual-light gimbal captures the person in front of it. Feature vectors are extracted from the scene. Calculations revealed With cloud delivery The cosine similarity is 0.88 (greater than the threshold of 0.75). Judgment: It is confirmed that the person in front of us is indeed the intruder A detected by the panoramic camera, and not another worker B passing by.

[0060] Compliance Response: The robot locks onto the target, and the gimbal automatically zooms in. A fine-grained detection algorithm identifies that person A is not wearing a safety helmet or insulated clothing. The robot issues an audible and visual alarm: "Danger area, please leave immediately!" and uploads a chain of evidence, including a panoramic trajectory map and close-up photos, to the control center, notifying safety inspectors to intervene.

[0061] Through the above process, the system successfully intercepted and drove away personnel A within 30 seconds before A actually touched the live conductor, preventing a potential personal safety accident. Compared to traditional fixed monitoring that can only record footage afterward or traditional inspection robots that follow people, the spatiotemporal interception mechanism of this invention significantly improves the initiative and success rate of supervision.

[0062] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0063] like Figure 4As shown, the computer system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of system 400. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0064] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a liquid crystal display (LCD) and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card and a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0065] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0066] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0068] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0069] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire real-time monitoring video streams using a fixed panoramic camera deployed high in the substation; identify dynamic personnel targets within the construction area using a lightweight target detection algorithm; extract the position information of the dynamic personnel targets in the panoramic image coordinate system, and simultaneously extract the visual appearance feature vector of the target; map the position information in the panoramic image coordinate system to geographical coordinates in the substation's world coordinate system based on a pre-calibrated homography matrix; process the time series of the geographical coordinates using a Kalman filter algorithm to construct the motion state vector of the dynamic personnel targets, the motion state vector including at least the current position, motion velocity vector, and acceleration vector; and obtain the position of the target in the waiting area. The system calculates the current position and maximum speed of the quadruped robot in patrol or operational status; based on the motion state vector of the dynamic personnel target and the quadruped robot's motion capability, it solves for the optimal spatiotemporal interception point and generates a navigation path command from the quadruped robot's current position to the optimal spatiotemporal interception point, which is then sent to the quadruped robot; after the quadruped robot reaches the optimal spatiotemporal interception point, it uses its onboard local detail camera to acquire images of the personnel on site and extracts the site feature vector; it calculates the cosine similarity between the site feature vector and the visual appearance feature vector issued in step S1; if the cosine similarity is greater than a preset identity verification threshold, the target is determined to be successfully locked, and the quadruped robot adjusts the focal length and gimbal angle of the local detail camera according to the preset violation detection logic to perform high-resolution acquisition and compliance analysis of the construction details of the locked target.

[0070] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A collaborative monitoring method for substation construction sites based on spatiotemporal prediction and viewpoint complementarity, characterized in that, include: S1: Real-time monitoring video streams are collected using fixed panoramic cameras deployed high up in the substation. Dynamic personnel targets in the construction area are identified using a lightweight target detection algorithm. The position information of the dynamic personnel targets in the panoramic image coordinate system is extracted, and the visual appearance feature vector of the targets is extracted simultaneously. S2: Based on the pre-calibrated homography matrix, the position information in the panoramic image coordinate system is mapped to the geographic coordinates in the substation world coordinate system; the time series of the geographic coordinates is processed using the Kalman filter algorithm to construct the motion state vector of the dynamic personnel target, the motion state vector including at least the current position, the motion velocity vector and the acceleration vector; S3: Obtain the current position and maximum speed of the quadruped robot in standby or patrol state; based on the motion state vector of the dynamic human target and the motion capability of the quadruped robot, solve for the optimal spatiotemporal interception point, and generate a navigation path command from the current position of the quadruped robot to the optimal spatiotemporal interception point and send it to the quadruped robot. S4: After the quadruped robot reaches the optimal spatiotemporal interception point, it uses its onboard local detail camera to capture images of people on site and extract site feature vectors; it calculates the cosine similarity between the site feature vectors and the visual appearance feature vectors issued in step S1. S5: If the cosine similarity is greater than the preset identity verification threshold, the target is determined to be successfully locked. The quadruped robot adjusts the focal length and gimbal angle of the local detail camera according to the preset violation detection logic to perform high-resolution acquisition and compliance analysis of the construction details of the locked target.

2. The method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity as described in claim 1, characterized in that, The process of constructing the motion state vector specifically includes: establishing state equations. ,in, for Target state vector at time 1 , Here is the state transition matrix. The process noise vector is defined; the observation equation is established. ,in, These are the geographic coordinate observations obtained by mapping in S2. Represents the observation matrix. The observation noise vector is represented; the target's velocity vector is corrected in real time using the prediction and update steps of Kalman filtering. This is to eliminate positioning noise caused by wide-angle distortion in fixed panoramic cameras.

3. The method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity as described in claim 1, characterized in that, In S3, the method for finding the optimal spatiotemporal interception point is as follows: constructing an encounter time function. ,in, As candidate interception points, This is the robot's current position. The target's current position, and Let be the scalar velocities of the robot and the target, respectively; under the constraints of the passable road network of the substation, solve for such that Minimized position point As the optimal spatiotemporal interception point, it ensures that the quadruped robot completes the interception before the target leaves the current monitoring area.

4. The method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity as described in claim 1, characterized in that, In S1 and S4, the extraction of the visual appearance feature vector adopts a deep twin network with shared weights: the fixed panoramic camera acquires images from a high top-down angle, and the local detail camera acquires images from a level or upward angle; the deep twin network introduces a viewpoint transfer loss function during the training phase, and maps pedestrian images from different viewpoints to a unified feature space through an adversarial generative network, so that the feature vector distance of the same target under top-down and level-up views is less than the feature vector distance of different targets under the same viewpoint.

5. The method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity as described in claim 1, characterized in that, It also includes a priority scheduling strategy based on the risk potential energy field: a static risk map of the substation is pre-constructed, and the energized area and high-altitude work area are marked as high potential energy areas; combined with the motion state vector in S2, the probability of a dynamic personnel target entering the high potential energy area at a future time is calculated; when there are multiple dynamic personnel targets, the quadruped robot is prioritized to intercept the target with the highest probability of entering the high potential energy area.

6. The method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity as described in claim 1, characterized in that, It also includes an active replacement mechanism based on line-of-sight occlusion: when a fixed panoramic camera detects a dynamic human target being occluded by on-site equipment, causing a decrease in the confidence level of feature extraction, the spatial geometric relationship between the occluder and the target is calculated; The command to generate the best replacement observation point is sent to the quadruped robot, instructing the quadruped robot to move to a side-rear position that can avoid obstructions and maintain a safe distance from the target to collect features.

7. The method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity as described in claim 1, characterized in that, In S5, the compliance analysis specifically includes: the quadruped robot using the high-magnification zoom function of the local detail camera to crop images of specific body parts of the locked target; inputting the cropped images into a classification model trained for fine-grained violations to identify behaviors including not fastening seat belt buckles, not wearing helmets correctly, and using mobile phones in violation of regulations; if a violation is identified, uploading a combined evidence chain containing panoramic and local detail images to the management platform.

8. The method for collaborative supervision of substation construction sites based on spatiotemporal prediction and viewpoint complementarity as described in claim 1, characterized in that, During its movement, the quadruped robot receives the motion state vector updated by S2 in real time and uses a model predictive control algorithm to dynamically adjust its own trajectory in order to cope with the random turning or speed change behavior of dynamic human targets.

9. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that, When the one or more computer programs are executed by a computer processor, they perform the method according to any one of claims 1-8.

10. A substation construction site collaborative monitoring system based on spatiotemporal prediction and viewpoint complementarity, used to implement the method as described in any one of claims 1-8, characterized in that, include: The panoramic perception subsystem includes multiple low-power fixed cameras distributed at the highest point of the substation, and is equipped with an edge computing module for performing target detection and feature fingerprint extraction. The collaborative management and control cloud platform is equipped with a coordinate mapping module, a Kalman filter motion prediction module, and an interception scheduling engine, which are used to calculate the optimal spatiotemporal interception point and issue scheduling instructions. The mobile monitoring subsystem includes at least one quadruped robot equipped with a dual-light gimbal camera and an onboard AI computing unit for performing cross-view feature comparison and refined compliance inspection.

Images