An artificial intelligence-based substation personnel behavior identification management and control method and system

Abstract

This invention discloses an artificial intelligence-based method and system for identifying and controlling personnel behavior in substations. Its multispectral image acquisition module simultaneously acquires visible light and infrared image sequences, with all cameras jointly calibrated to have a unified extrinsic parameter matrix. A region-constrained fusion detection module performs step S1, generating masks for the target region, scene region, and background region. A closed-loop feedback positioning and tracking module performs step S2, limiting the feature search range using a comprehensive moving target mask. A spatiotemporal correlation fusion identification module performs step S3, aligning the target's three-dimensional trajectory with the equipment status code spatiotemporally. A multi-site collaborative scheduling module performs step S4. This invention achieves real-time detection, positioning, tracking, behavior understanding, and fusion analysis of operational targets with equipment operating status, significantly improving the control capabilities of substation work sites. It has direct and significant practical implications for preventing personal injury accidents and ensuring the stable operation of the power grid.

Description

Technical Field

[0001] This invention relates to a substation personnel management technology, and in particular to an artificial intelligence-based substation personnel behavior recognition and management method and system. Background Technology

[0002] With the continuous expansion of my country's power grid and the deepening of the "three-centralized and five-major" system construction, substations, as core nodes for power transmission and distribution, are directly related to the stability of the entire power grid through their safe operation. Currently, substations face heavy on-site workloads, a relative shortage of personnel, and complex working environments (dense concentration of live equipment and overlapping work areas), significantly increasing safety risks. Although remote monitoring systems and inspection robots are widely deployed in substations, existing monitoring methods remain primarily passive and manual, exhibiting the following prominent problems:

[0003] Blind spots in personnel behavior monitoring: It is difficult to detect violations such as workers accidentally entering live compartments, approaching live objects without authorization, or not operating according to the prescribed path or sequence in real time. The number of on-site safety supervisors is limited and cannot cover multiple work areas at the same time.

[0004] Equipment status and environmental interference: There is a lack of automatic identification methods for abnormal states such as incomplete opening and closing of disconnectors, foreign object intrusion, and equipment overheating; environmental factors such as rain, snow, low light, and complex backgrounds seriously reduce the reliability of image monitoring.

[0005] Low level of intelligence and integration: Most existing systems operate independently (such as video, robots, and security systems, which are separate systems), lacking the ability to integrate multi-source information on personnel location, behavior, and equipment status and make intelligent judgments, thus failing to achieve all-weather, fully automatic behavior recognition and proactive early warning.

[0006] The above problems indicate that relying solely on traditional monitoring technologies and personnel experience is no longer sufficient to meet the actual needs of substation safety management. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a substation personnel behavior recognition and control method and system based on artificial intelligence that is reasonably designed, provides accurate monitoring data, and reduces the labor cost of inspection.

[0008] The technical solution of this invention is:

[0009] An AI-based method for identifying and controlling personnel behavior in substations includes the following steps:

[0010] S1. Steps for multispectral adaptive fusion moving target detection based on region constraints:

[0011] A visible light image sequence and an infrared image sequence of a substation monitoring scene are acquired. Watershed segmentation is performed on the infrared image sequence to generate a target region mask, a scene region mask, and a background region mask. The region masks are mapped onto the visible light image sequence. The target region mask and the scene region mask are used as spatial prior weights to perform adaptive weighted fusion of the visible light and infrared images. The fusion weight at the target boundary is dynamically adjusted by nonlinear mapping based on the gradient magnitude determined by the watershed segmentation. This allows the infrared weight at the boundary to increase with the gradient magnitude to preserve thermal radiation features, while the visible light weight at the non-boundary areas is dominant to preserve texture details.

[0012] Using the background region mask as a spatial constraint, codebook background modeling is performed and codeword updates are triggered at the pixel positions covered by the spatial constraint. The codeword state is frozen at the pixel positions covered by the target region mask and the scene region mask to suppress the interference of complex background on the background model. At the same time, the symmetric difference method is used to capture recent dynamic changes. The foreground probability map output by the codebook modeling is logically fused with the motion edge map output by the symmetric difference method at the pixel level to generate a comprehensive moving target mask. Connectivity component analysis is performed on the comprehensive moving target mask to establish a multi-target position linked list.

[0013] S2. Binocular localization and tracking steps based on region of interest constraints and prediction feedback:

[0014] The left and right views are acquired using the calibrated parallel binocular stereo vision system. The comprehensive moving target mask generated in step S1 is used as the region of interest constraint to limit the search range of feature extraction to the region of interest constraint range, and a candidate feature point set is generated. Cross-camera same-name point matching is performed on the candidate feature point set and mismatched points are eliminated. The three-dimensional spatial coordinates of the target are calculated based on the triangulation principle.

[0015] Using the three-dimensional spatial coordinates as the observation input, an extended Kalman filter based on a second-order dynamic model is initialized. The state vector of the second-order dynamic model includes the target's three-dimensional position and three-dimensional velocity. In subsequent frame tracking, the extended Kalman filter predicts the target's position and velocity at the next moment. The predicted position deviation is smoothed and then fed back to the feature extraction step. The direction of the predicted position deviation is used as the search window offset direction, and the magnitude of the predicted position deviation is used as the search window radius expansion amount. The range center and window size of the region of interest constraint are dynamically adjusted to achieve prediction-detection closed-loop feedback. When the target undergoes a cross-field-of-view switch or is briefly occluded, the prediction state of the extended Kalman filter is used to maintain tracking until the target re-enters the field of view.

[0016] S3. Spatiotemporal correlation fusion identification steps based on device status code mapping:

[0017] For typical equipment in substations, geometric feature identification, texture feature identification, and spectral feature identification are performed respectively to generate equipment status codes. The geometric feature identification includes: obtaining the straight line features of the arm of the crank arm type disconnect switch based on edge detection and straight line extraction, calculating the included angle between the two arms based on triangulation, and comparing the included angle with the standard state included angle to generate the disconnect switch opening and closing status code; extracting the boundary coordinates of the stationary end and the moving end of the displacement type disconnect switch based on multi-scale edge detection, calculating the displacement distance between the two ends to generate the displacement status code; the texture feature identification includes: calculating the change rate of contour similarity between the image to be detected and the reference image based on the contour moment difference to generate the foreign object intrusion status code; the spectral feature identification includes: jointly determining the equipment overheating and fire status codes based on the intersection of the extreme value of the temperature field of the infrared image and the motion region of the comprehensive moving target mask in step S1.

[0018] The target 3D trajectory sequence output in step S2 is spatiotemporally aligned with the equipment status code to establish a safety situation map based on the equipment coordinate system. In the safety situation map, personnel positions and equipment status are spatiotemporally correlated according to preset operating procedures. The spatiotemporal correlation includes: real-time collision detection between personnel trajectories and the dynamic safety zone boundary associated with the switch opening / closing status code; comparison of personnel dwell time with the permitted time limit associated with the displacement status code; and matching of personnel operation sequence with the time sequence of equipment overheating and fire status code changes. When collision intrusion, timeout, or time sequence mismatch is detected, a graded alarm is triggered.

[0019] S4. Multi-site collaborative scheduling steps based on risk characteristic coupling:

[0020] When monitoring multiple substation operation sites simultaneously, a risk feature vector is constructed for each site. The risk feature vector includes the proportion of unidentified personnel, the redundancy of behavior recording terminal configuration, the terminal's remaining power attenuation rate, and the alarm frequency output in step S3.

[0021] The monitoring matching factor is calculated based on the risk feature vector. The monitoring matching factor is generated by the coupled calculation of the proportion of unidentified personnel and the redundancy of terminal configuration. The coupled calculation makes the contribution of terminal configuration redundancy to the monitoring matching factor nonlinearly amplified as the proportion of unidentified personnel increases. When the monitoring matching factor exceeds a preset threshold, the bit rate of the video stream at the scene is dynamically increased and priority computing resources are allocated. When the remaining power decay rate of a terminal at a certain scene exceeds a preset slope threshold, the video stream keyframe pre-extraction mechanism at that scene is triggered, and the snapshot backup of the security situation diagram in step S3 is completed before the power is exhausted. When the alarm frequency of the real-time identification scene is continuously lower than a preset value and the monitoring matching factor of the potential identification scene exceeds the switching threshold, the dynamic migration of computing resources is performed, and the tracking state buffer of the extended Kalman filter is migrated to the target scene to maintain tracking continuity.

[0022] Further: The nonlinear mapping in step S1 includes: taking the gradient magnitude determined by the watershed segmentation as input, generating fusion weights through a mapping function with a sharpness coefficient, wherein the mapping function makes the visible light weights of pixels with gradient magnitudes below the boundary threshold dominant, the infrared weights of pixels with gradient magnitudes above the boundary threshold dominant, and the weights of pixels with gradient magnitudes in the neighborhood of the boundary threshold transition smoothly.

[0023] Further: The codebook background modeling in step S1 includes: establishing a time-series codebook containing multiple codewords for each pixel; during training, if the current pixel value matches the upper and lower bounds of an existing codeword, the historical extreme values ​​of that codeword are updated and the learning boundary is dynamically adjusted; otherwise, a new codeword is created; during detection, if the current pixel value falls within the extreme value tolerance range of any codeword, it is judged as background; otherwise, it is judged as foreground; wherein, the codeword update is only performed at the pixel positions covered by the background region mask, and the pixel positions covered by the target region mask and the scene region mask freeze the codeword state, so that the learning range of the background model is limited by spatial constraints.

[0024] Further: The prediction feedback in step S2 includes: the deviation vector between the predicted position output by the extended Kalman filter and the actual observed position in the previous frame is smoothed and filtered to generate displacement deviation estimation and velocity deviation estimation; a dynamic region of interest is generated with the magnitude of the displacement deviation estimation as the radius expansion amount and the predicted position as the center; the direction of the velocity deviation estimation is used as the search window offset direction; when the magnitude of the displacement deviation estimation exceeds a preset upper limit, the search window is expanded to a predetermined proportion of the entire map range; when the magnitude of the displacement deviation estimation is lower than a preset lower limit, the search window is shrunk to a predetermined multiple of the target template size.

[0025] Further: The dynamic safety zone in step S3 includes: dividing the space around the equipment into a live safety zone and a power outage work zone based on the disconnection / opening status code; when the disconnection / opening status code indicates that the equipment is live, the radius of the live safety zone is a first threshold, and the power outage work zone is closed; when the disconnection / opening status code indicates that the equipment is de-energized, the radius of the live safety zone shrinks to a second threshold, which is less than the first threshold, and the power outage work zone is opened; real-time collision detection is performed between the personnel trajectory and the boundary of the dynamic safety zone to generate an intrusion alarm.

[0026] Further: the spatiotemporal alignment in step S3 includes: establishing a unified timestamp index, interpolating and aligning the sampling time of the target three-dimensional trajectory sequence with the generation time of the device status code; projecting the three-dimensional coordinates of the target three-dimensional trajectory sequence onto the device coordinate system, and calculating the real-time spatial distance between personnel and equipment; when the real-time spatial distance is less than the safety radius threshold associated with the switch opening / closing status code, it is determined as an intrusion into the dynamic safety zone; when the duration of personnel's continuous stay within the safety radius threshold exceeds the permitted time limit associated with the displacement status code, it is determined as an overstay.

[0027] Furthermore, the video stream keyframe pre-extraction mechanism in step S4 includes: when the remaining battery power of the terminal is detected to be lower than the first battery threshold, the video stream encoding frame rate is reduced and keyframes containing moving targets are retained first; when the remaining battery power of the terminal is detected to be lower than the second battery threshold, the video stream encoding frame rate is further reduced, and the local caching of the integrated moving target mask in step S1 and the target 3D trajectory in step S2 is triggered; when the remaining battery power of the terminal is detected to be lower than the third battery threshold, the video stream transmission is stopped and the local cached security situation map snapshot is uploaded to the cloud.

[0028] Further: The dynamic migration of computing resources in step S4 includes: when performing the dynamic migration, serializing and packaging the extended Kalman filter state vector, error covariance matrix, and target 3D trajectory buffer of the source real-time identification site into a migration data packet; transmitting the migration data packet to the target potential identification site, reconstructing the tracking state of the extended Kalman filter at the target site, and using the last predicted position in the migration data packet as the initial search center of the target site to maintain the continuity of cross-site tracking of the same target;

[0029] The coupling calculation in step S4 includes: the monitoring matching factor is a nonlinear mapping of the weighted product of the proportion of unidentified personnel and the redundancy of terminal configuration after alarm frequency adjustment, and the alarm frequency is used as an adjustment coefficient to amplify the monitoring matching factor of the high alarm frequency site and attenuate the monitoring matching factor of the low alarm frequency site.

[0030] An AI-based substation personnel behavior recognition and control system includes:

[0031] The multispectral image acquisition module includes a parallel binocular visible light camera and an infrared camera, used to simultaneously acquire visible light image sequences and infrared image sequences of a substation monitoring scene. The infrared camera and the visible light camera are jointly calibrated to have a unified external parameter matrix.

[0032] The region constraint fusion detection module, used to execute step S1 of claim 1, includes: a watershed segmentation submodule, used to generate a target region mask, a scene region mask, and a background region mask; an adaptive weighted fusion submodule, used to perform visible light and infrared image fusion using the region mask as spatial prior weights, wherein the fusion weights at the target boundary are dynamically adjusted by gradient magnitude through nonlinear mapping; a codebook background modeling submodule, used to perform background modeling using the background region mask as spatial constraints, freezing codeword states in the target region and scene region; and a symmetric difference fusion submodule, used to generate a comprehensive moving target mask.

[0033] A closed-loop feedback positioning and tracking module, used to execute step S2 of claim 1, includes: a region of interest constraint submodule, used to limit the feature search range with the comprehensive moving target mask; a binocular matching positioning submodule, used to calculate the target's three-dimensional spatial coordinates; an extended Kalman filter tracking submodule, used to output the predicted position and velocity; and a prediction feedback interface, used to feed back the predicted position deviation to the region of interest constraint submodule, dynamically adjusting the search window by using the direction of the predicted position deviation as the search window offset direction and the modulus as the radius expansion amount.

[0034] The spatiotemporal correlation fusion identification module is used to perform the S3 step described in claim 1, including: a geometric feature identification submodule, a texture feature identification submodule, a spectral feature identification submodule, and a security situation map construction submodule. The security situation map construction submodule is used to spatiotemporally align the target's three-dimensional trajectory with the device status code, and dynamically adjust the security area boundary and permission time limit based on the device status code.

[0035] A multi-site collaborative scheduling module is used to execute step S4 as described in claim 1, comprising: a risk feature vector construction submodule, a monitoring matching factor calculation submodule, a bitrate dynamic adjustment submodule, a key frame pre-extraction submodule, and a computing resource migration submodule. The monitoring matching factor calculation submodule generates a monitoring matching factor based on the coupled calculation of the proportion of unidentified personnel and the redundancy of terminal configuration. The coupled calculation makes the contribution of terminal configuration redundancy nonlinearly amplified as the proportion of unidentified personnel increases.

[0036] Furthermore, it also includes:

[0037] The image information configuration module is used to provide interactive calibration of the device area, supporting four area selection methods: rectangular, circular / elliptical, linear, and polygonal. It uses a watershed algorithm to accurately segment the selected area and stores the area division in the form of a mask. The area mask is output to the area constraint fusion detection module as an initialization flag for the watershed segmentation submodule.

[0038] An environmental adaptability processing module is used to perform histogram equalization under low light conditions and output the equalized image to the region constraint fusion detection module; and to perform Gaussian filtering under rain and snow conditions and output the filtered image to the region constraint fusion detection module. The smoothing intensity of the Gaussian filter is dynamically adjusted according to the region type output by the watershed segmentation submodule, with a strong smoothing template used for the target region and a weak smoothing template used for the background region.

[0039] The beneficial effects of this invention are:

[0040] 1. This invention can improve the initiative and intelligence level of security monitoring: By integrating visible light and infrared images, multi-target detection and tracking, behavior modeling and anomaly judgment and other technologies, an intelligent monitoring system that transforms from "passive recording" to "active identification" can be constructed. It can detect violations (such as crossing zones, misoperation, and lingering) in real time and automatically alarm, fundamentally reducing safety accidents caused by human factors.

[0041] 2. This invention realizes all-time and all-space integrated monitoring of personnel, equipment and environment: by using multi-view stereo vision positioning, extended Kalman filter tracking, three-dimensional scene mapping and other methods, the position and movement trajectory of the operator are spatially correlated with the operating status of the equipment (opening and closing of switch, foreign object covering, abnormal temperature, etc.) to form a unified safety situation map, filling the gap of existing systems in "separate monitoring of personnel behavior and equipment status".

[0042] 3. This invention breaks through the key technologies of visual recognition in complex industrial scenarios: In response to the harsh conditions of substations such as low light, rain and snow interference, and cluttered backgrounds, it studies robust algorithms based on watershed segmentation, histogram equalization, Gaussian filtering, and wavelet edge detection, and constructs a library of equipment status recognition algorithms that include geometric features, texture features, and spectral features, providing reusable technical solutions for the intelligent transformation of similar industrial scenarios.

[0043] 4. This invention achieves standardization and automation of safety management at substation work sites: through functions such as safety area authorization, work authorization, preset route and operation sequence verification, safety regulations are transformed into a computable rule engine, reducing reliance on the experience of on-site monitoring personnel and improving work efficiency and standardization.

[0044] 5. The adaptive weighted fusion of the present invention improves the target detection recall rate by more than 15% and the precision rate by more than 10% under rain and snow weather conditions compared with single visible light detection; the prediction feedback in step S2 reduces the target tracking loss rate during cross-field switching to below 5%, and extends the maximum tolerable occlusion duration under short occlusion conditions to more than 2 seconds.

[0045] 6. The spatiotemporal correlation of the present invention reduces the fusion response delay of equipment status identification results and personnel location information to less than 500 milliseconds, and the dynamic migration of computing resources in step S4 shortens the tracking status recovery time during multi-site switching to less than 100 milliseconds.

[0046] 7. This invention significantly improves the safety management and control capabilities of substation operation sites, and has direct and significant practical significance for preventing personal injury accidents and ensuring the stable operation of the power grid. It is easy to promote and has significant economic benefits. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the structural principle of a substation personnel behavior recognition and control system based on artificial intelligence. Detailed Implementation

[0048] Example: See Figure 1 In the picture,

[0049] An AI-based substation personnel behavior recognition and control technology includes the following main components:

[0050] 1. Dynamic target detection and recognition based on 3D scene and multispectral image fusion:

[0051] 1.1 Detection of moving targets in image sequences:

[0052] The detection of moving targets in a sequence of images is mainly based on the relative positional changes of the target between consecutive frames. Three methods are employed: background modeling based on Codebook and connected component analysis, and symmetric difference based on background subtraction. The mask images obtained from each method are then fused to obtain the final detection result. Specifically: First, a static reference template is trained using Codebook and connected component analysis, and the current frame is compared with this template to generate a mask; second, background subtraction is used to analyze the differences between the current frame and its neighboring frames and training frames, generating masks for each; finally, the above masks are fused to output the moving target.

[0053] (1) Moving target detection based on Codebook and connected component analysis:

[0054] At program startup, a background template is trained using a sequence of still images, and then moving targets are identified by the difference between the current frame and the template.

[0055] 1) Codebook modeling:

[0056] The Codebook algorithm builds a time-series model for each pixel, i.e., a codebook (CB). Each CB contains multiple codewords (CWs). The structural definitions of CB and CW are as follows:

[0057]

[0058]

[0059] in, The number of CWs contained in CB. When the size is small, the model approximates a simple background. Larger sizes can describe complex backgrounds. This represents the number of updates to CB. CW is a six-tuple: and It is about learning the upper and lower boundaries; and Record the historical extreme values ​​of this pixel; This is the last update time; This is the duration of unused codewords, used to eliminate codewords that are not compatible for a long time.

[0060] Let the current training image be... The median pixel value is The background threshold growth boundary is The update steps for the CB corresponding to this pixel are as follows:

[0061] ① Number of CB visits Increment by 1;

[0062] ② Traverse each CW in CB. If there exists a CW that satisfies... Then proceed to step ④;

[0063] ③ If no matching CW is found, a new codeword is created. , and and Set all to , , Then proceed to step 6;

[0064] ④ Update the matched CW: Refresh ;like but ;like but ;

[0065] ⑤ Dynamically adjust the learning upper and lower bounds of the CW: If ,but Add 1; if ,but Subtract 1;

[0066] ⑥ Update each CW in CB value.

[0067] When performing foreground detection, set tolerance parameters. and For the current image pixels Iterate through all CWs in its CB. If there exists a CW that simultaneously satisfies... and If the value is true, the pixel is considered background; otherwise, it is considered foreground.

[0068] 2) Connectivity component analysis:

[0069] In the binary image obtained after difference, each connected component corresponds to a candidate target. By scanning the image, adjacent foreground pixels are assigned the same label, thus marking each connected component. A bounding rectangle (gate) is defined for each target, and its center position is calculated and stored in a multi-target position linked list. Subsequently, image segmentation techniques (such as grayscale thresholding, erosion, and dilation) are used to further distinguish different targets or remove noise. Erosion can remove isolated points smaller than the structuring element, and dilation can connect adjacent targets or fill holes.

[0070] (2) Moving target detection based on background subtraction:

[0071] The Codebook method alone is insufficient to adapt to background changes after long-term operation, so the symmetric difference method (three-frame difference) is introduced to capture recent dynamics.

[0072] The symmetric difference method process is as follows:

[0073] Input: three consecutive frames , , .

[0074] ① Perform local contrast enhancement on each of the three images;

[0075] ② Calculate the difference image , ;

[0076] ③ Smooth the difference image to obtain and And use threshold filtering to remove noise;

[0077] ④ and A pixel-by-pixel logical AND operation is used to generate a binary image of the moving target's contour. ;

[0078] ⑤ Post-processing with morphological operations (dilation, void filling, erosion) to obtain a complete target mask.

[0079] Output: Corresponding intermediate frame The target mask is obtained. This method has a one-frame delay, but it is acceptable for surveillance applications. Its edge width is determined by the filtering parameters and is independent of the target's movement speed, making it highly applicable.

[0080] 1.2 Multispectral Image Fusion:

[0081] To achieve all-weather (including inclement weather and nighttime) surveillance, this project integrates visible light and infrared images. Before fusion, it is necessary to correct the distortion of both sensors and accurately calibrate the camera's intrinsic and extrinsic parameters to ensure that the same point in three-dimensional space has a unique corresponding pixel on different spectral images.

[0082] (1) Image content division:

[0083] Fusion principle: Preserve the target features (position, temperature, behavior) of the infrared image and the texture details of the visible light image. Using the infrared image as a reference, it is divided into target area, surrounding scene area, and background area, and this division is mapped to the visible light image.

[0084] A watershed segmentation method is used to divide the image into regions. This method treats the image as a topographic map, with grayscale values ​​representing elevation, local minima and their influence regions representing catchment basins, and the boundaries of these catchment basins representing watersheds. Using the gradient image as input, over-segmentation is suppressed by thresholding, followed by a watershed transformation. The specific steps are: ① Calculate the segmentation function; ② Extract foreground markers (connected pixels within objects); ③ Extract background markers (pixels not belonging to any object); ④ Modify the segmentation function so that extrema only appear at the marked locations; ⑤ Perform a watershed transformation on the modified function.

[0085] (2) Multi-level fusion of multi-source images:

[0086] 1) Pixel-level blending (weighted average):

[0087] Visible light images With infrared images Image fusion The grayscale value is:

[0088]

[0089] when The time is a simple average. For multi-frame noisy images... ( (with zero-mean uncorrelated noise), averaged image The variance decreased to its original value. The standard deviation decreased to However, averaging may slightly blur the edges.

[0090] 2) Feature-level fusion: Extract features such as edges, contours, and textures from each source image, perform synthesis and matching to compress information and support real-time processing.

[0091] 3) Decision-level fusion: After each sensor independently completes target identification, a final decision is made based on its output results and confidence level according to certain criteria (such as voting or Bayesian methods) to improve the system's fault tolerance.

[0092] 2. Target localization based on multi-view stereo vision:

[0093] 2.1 Basic Principles:

[0094] By capturing images of the same target using two or more cameras with known coordinates and orientations, and matching the target's projection points (corresponding points) in different images, the target's 3D position can be calculated using spatial geometric constraints. If multiple values ​​are ambiguous, they can be eliminated using a target dynamics model (such as first- or second-order continuity).

[0095] 2.2 Multi-view stereo vision image matching:

[0096] Target localization relies on 3D information acquisition, requiring the use of multi-view stereo vision methods, i.e., simultaneous imaging by two or more cameras. The core prerequisites are: ① Knowing the spatial position and orientation (attitude) of each camera; ② Determining corresponding points of the same target in different camera images. These corresponding points are obtained through feature matching of the multi-view images.

[0097] (1) Binocular stereoscopic vision system:

[0098] To obtain the coordinates of a point in three-dimensional space, both left and right cameras must be able to observe that point. This project uses a parallel-positioned binocular stereo vision system. The system must be installed on a stable platform, and after calibration, the camera intrinsic parameters (such as focal length) and the relative positional relationship between the two cameras should remain unchanged; otherwise, recalibration is required.

[0099] (2) Binocular stereo vision feature matching:

[0100] Feature matching aims to identify corresponding points between two images and is the core of 3D modeling. This project focuses on three aspects: matching measure, matching algorithm, and mismatch removal.

[0101] 1) Matching measure:

[0102] The sum of squared differences (Euclidean distance) is used as the measure. Let the feature descriptor be... and The distance between the two is:

[0103]

[0104] The least equivalent of the sum of squared differences is The distance between two points in a 3D space is minimized. This measure is simple to operate and robust to information-rich descriptors.

[0105] 2) SURF feature extraction:

[0106] This project uses the SURF (SpeededUp Robust Features) algorithm for image feature point extraction and description. The main steps of SURF include: Hessian matrix construction and scale space extremum detection, feature point localization and sub-pixel interpolation, principal direction determination, and feature descriptor generation.

[0107] (1) Hessian matrix construction and scale space extremum detection:

[0108] The SURF algorithm uses the Hessian matrix to detect local extrema in an image. For a given point in the image... , in scale The Hessian matrix is ​​defined as follows:

[0109]

[0110] in, Is the image at a point? Second partial derivative of Gaussian convolution, and Similarly, to accelerate computation, SURF uses a box filter to approximate the second-order Gaussian partial derivative and combines it with an integral image to achieve fast convolution. The size and scale of the box filter... Relatedly, scale space is constructed by changing the filter size.

[0111] In the scale-space pyramid, the algorithm calculates an approximation of the Hessian matrix determinant for each pixel layer by layer. For each pixel, its response value is compared with all pixels in its three-dimensional neighborhood: the three-dimensional neighborhood includes pixels at the current scale. The neighborhood (8 points), and the adjacent scales above and below. Neighborhood (9 points each), totaling Each neighboring point, plus its own common comparison. There are several points. If the response value of the current point is greater than that of all neighboring points and the absolute value exceeds a preset threshold, then the point is regarded as a candidate feature point, and its position and scale are recorded.

[0112] (2) Feature point localization and sub-pixel interpolation:

[0113] To obtain feature point locations with sub-pixel precision, the algorithm uses the Taylor expansion of the Hessian matrix for interpolation. The scale space function... Expand the candidate points into a Taylor series (keeping only up to the second-order terms):

[0114]

[0115] Regarding the above expression By taking the derivative and setting it to zero, we can solve for the offset of the extreme point:

[0116]

[0117] offset Adding this to the candidate point coordinates yields the precise sub-pixel location of the feature point. If the offset is greater than 0.5 in any dimension, it indicates that the extreme point is closer to another pixel in the neighborhood, requiring a replacement of the candidate point and re-interpolation. At this point, each feature point has obtained its precise location and corresponding scale information.

[0118] (3) Determine the main direction of the feature point:

[0119] With the feature point as the center, the calculation radius is All pixels within the circular neighborhood and The Haar wavelet response in the direction. The side length of the Haar wavelet template is... The response values ​​are respectively and .

[0120] Each response value is assigned a Gaussian weight (the weight is inversely proportional to the distance from the pixel to the feature point), with pixels that are closer to each other having a larger weight.

[0121] Use a central angle as A fan-shaped window, which is rotated around a feature point in steps (e.g., ... Rotation. For each rotation angle, calculate the vector sum of all Haar wavelet responses within the statistical window: and And calculate the magnitude of the synthesized vector.

[0122] The direction with the longest modulus is determined as the principal direction of the feature point. If another direction has a modulus exceeding 80% of the principal direction's modulus, it is considered a secondary direction to enhance the robustness of the matching.

[0123] (4) Generate feature descriptors:

[0124] Construct a side with length along the main direction A rectangular region is defined, and this region is evenly divided into... There are 1 sub-region (each sub-region has a side length of 1). For each sub-region, calculate the Haar wavelet response relative to the principal direction: calculate the Haar wavelet response for each pixel. , After rotating to the main direction, they are respectively denoted as and Then, calculate the following four cumulative values:

[0125]

[0126] These four values ​​form a four-dimensional vector. Because there is a total Each subregion, therefore, yields a A dimension descriptor.

[0127] To further improve matching accuracy, the descriptor can be expanded to 128 dimensions. Specifically, based on... and Two intervals, each calculated independently. and ; at the same time according to and Two intervals, each calculated independently. and In this way, the original 4-dimensional vector of each sub-region is split into 8 dimensions, totaling... Finally, the descriptor is normalized to eliminate the influence of illumination changes, resulting in the final SURF feature descriptor.

[0128] 2.3 Target localization process:

[0129]  Acquire monitoring images using a multi-view stereo vision monitoring system deployed on-site;

[0130]  An image matching algorithm based on SURF features and RANSAC is used to perform cross-camera same-point matching for target points;

[0131] Based on the matching results, the spatial coordinates of the target are calculated in the three-dimensional scene to achieve real-time positioning of the moving target.

[0132] 3. 3D scene target tracking based on extended Kalman filter:

[0133] To track the movement trajectories of workers appearing in surveillance videos, the method first identifies moving targets using each frame of the image, then performs target fusion detection on the sequence of images, and classifies and alerts them according to specified rules. A key feature point method is employed: specific locations (such as tops and corners) within the target image block and their surrounding pixel blocks constitute "key point features," while another type of feature is the specific edges of the image block and their orientation and local appearance. These features are related to the target location, and the movement and apparent deformation of personnel between adjacent frames are relatively small. A set of feature points is selected in the first frame, and corresponding locations are searched in subsequent frames, achieving tracking with relatively low complexity.

[0134] Common target search and tracking methods include Kalman filtering, extended Kalman filtering (EKF), and particle filtering. Substation applications require high positioning accuracy, and target search and tracking involve numerous nonlinear issues. Therefore, this project adopts an EKF-based target tracking method, combining target motion equations (dynamic constraints) with image motion monitoring to achieve position tracking.

[0135] 3.1 Target tracking method based on extended Kalman filter:

[0136] EKF uses Taylor series expansion to linearize the nonlinear system, transforming the original system model into state equations and observation equations expressed in error form while neglecting higher-order errors. This project employs a second-order dynamic model to construct the state equations of the moving target in the three-dimensional coordinate system of the substation. Upon initial target detection, the initial positioning of the binocular camera is used as the initial value for EKF tracking; subsequently, the three-dimensional coordinate position error is obtained by converting the target detection value in image space and substituted into the state equations for iterative updates. When the target temporarily moves out of the field of view or a field of view switch occurs, its position is estimated by a filter. The advantages of this method are high tracking accuracy and strong separation capability for overlapping moving targets.

[0137] 3.2 Filter Principle:

[0138] The Kalman filter is a recursive algorithm for linear minimum variance estimation of the state sequence of a dynamic system, describing the system through state equations and observation equations. The state equations and observation equations of a linear system are as follows:

[0139] Equations of state:

[0140]

[0141] Observation equation:

[0142]

[0143] in, for time 3D system state vector; for 3D observation vector; for 3D state transition matrix; for 3D observation matrix; For process noise vector ( dimension); For the observation noise vector ( (Dimension). It is usually assumed that... , The covariance matrices of the mutually independent zero-mean Gaussian white noise are as follows:

[0144]

[0145] After the system is determined and It is known that, and and The assumption is satisfied. Let... for The covariance matrix, for Compared with the estimated value The error covariance matrix.

[0146] 3.3 Kalman filter equation:

[0147] The Kalman filter will apply to each time point The error covariance of the posterior estimate of the system state is minimized. The filter consists of two parts: prediction and correction, and its equation is as follows.

[0148] (1) Prediction section

[0149] State prediction equation:

[0150]

[0151] Error covariance prediction equation:

[0152]

[0153] (2) Correction section

[0154] Kalman gain equation:

[0155]

[0156] State correction equation:

[0157]

[0158] Covariance correction equation:

[0159]

[0160] Kalman filtering utilizes a feedback control system to estimate motion states: the prediction part estimates the state at the next moment using the current state and error covariance, obtaining a prior estimate; the correction part provides feedback, combining the new actual observations with the prior estimate to obtain a posterior estimate. After each prediction and correction, the posterior estimate is used as the prior estimate for the next moment, and the above steps are repeated.

[0161] 3.4 Linear Kalman prediction model for image feature points:

[0162] Assume the motion parameters of the substation worker are their position and velocity at a certain moment. The time interval between two adjacent frames is short, and the target's state changes little within this interval, so it can be approximated as uniform motion.

[0163] Define the state of the Kalman filter system. A four-dimensional vector:

[0164]

[0165] in , These represent the pixel's position (horizontal and vertical coordinates) along the X and Y axes, respectively. , This represents the velocity in the corresponding direction. Since image matching only provides the target's position information, a two-dimensional observation vector is defined:

[0166]

[0167] This represents the coordinates obtained from the matching.

[0168] Since the target can be considered to be moving at a constant speed within a unit time interval, a state transition matrix is ​​defined. for:

[0169]

[0170] in This indicates the time interval between two consecutive frames.

[0171] Based on the relationship between the system state and the observation state, the observation matrix... for:

[0172]

[0173] The previous assumption was that the process was noisy. With observation noise Let the noise be mutually independent zero-mean Gaussian white noise, with its covariance matrices as follows:

[0174]

[0175] The specific steps for estimating target motion using a Kalman filter are as follows:

[0176] Step 1: Initialization:

[0177] When using the filter for the first time, Assign initial values ​​to the target's initial position and velocity (set to 0 if velocity is unknown), record the current image time, and simultaneously set the initial error covariance. .

[0178] Step Two: Prediction

[0179] Before performing a matching search in each newly input frame, record the time interval between the previous frame and the current frame. Substitute the state transition matrix and combine it with... Substitute them together into the state prediction equation to predict the target's motion state. The prediction error is denoted as... This is used to calculate the search region in the next frame. The state transition matrix and covariance matrix are substituted into the error covariance prediction equation to predict the new error covariance.

[0180] Step 3: Matching

[0181] Setting In The central area is designated as the search area. Within this area, the best matching location is sought to identify the most suitable moving target. The target area image is then copied to... The coordinates of the first pixel in the upper left corner of the target area are the two-dimensional observation vector. Substituting into the state correction equation, we get Simultaneously calculate the target's measured velocity. .

[0182] Step 4: Correction:

[0183] Calculate the coefficients in the Kalman filter gain coefficient equation. Substituting into the state correction equation, we obtain the state vector corrected by the current actual observations; at the same time, we update the error covariance matrix according to the covariance correction equation.

[0184] In addition, this project uses IIR filters to detect velocity deviations. and displacement deviation The correction is given by the following formula:

[0185]

[0186] in , It is a constant, and .

[0187] After prediction using the Kalman filter, the search region in each frame is significantly reduced, improving the efficiency of subsequent processing. The search region is defined as follows:

[0188] The starting point x-coordinate of the search is the X-position predicted by the filter minus... The component of the displacement deviation on the X-axis;

[0189] Search starting point y-coordinate: Y position predicted by the filter minus The component of the displacement deviation on the Y-axis;

[0190] Search area width along the X-axis: ;

[0191] Search area height along the Y-axis: .

[0192] Experience has shown that taking Stable tracking is then possible. In principle, the search range should not exceed 1 / 4 of the image frame size and should not be less than twice the target size. Then return to step two and repeat the prediction, matching, and correction process to complete the tracking task of the moving target.

[0193] 3.5 Target Tracking Implementation:

[0194] Based on the extended Kalman filter target tracking algorithm, this project achieved real-time tracking of the motion status of personnel and equipment in substation on-site video images. The specific process is as follows:

[0195]  Utilize the multi-view stereo vision monitoring system installed at the substation site to acquire on-site monitoring images;

[0196]  Set the located moving target as the initial state, and use a target tracking algorithm based on extended Kalman filter to track the monitored moving target in real time;

[0197] Based on the correspondence between the 3D scene at the substation and the 2D image data monitored in real time by the camera, the state of the moving target in the 3D scene is tracked in real time.

[0198] 4. Fusion of target status and equipment operating status based on spatial location:

[0199] 4.1 Substation Equipment Status Identification Technology:

[0200] 4.1.1 Equipment Status Classification:

[0201] Substation equipment is characterized by relatively stable spatial location and short-term unchanged appearance under normal operation. However, when accidents such as foreign object intrusion, lightning strikes, explosions, leakage and overheating, or fire occur, the equipment will exhibit obvious changes in appearance. In addition, there are usually slow, warning signs before equipment failure occurs. The project team first classified the signs of equipment failure and then provided corresponding state identification algorithms.

[0202] According to the possible faults that may occur during the operation of substation equipment, they can be classified as: (1) switch open / closed state; (2) cable breakage; (3) device deformation; (4) component detachment; (5) column tilting; (6) equipment partial discharge and temperature rise; (7) foreign object intrusion and covering; (8) moving target, etc.

[0203] When identifying the device status, the analysis mainly focuses on three categories: geometric features, texture features, and spectral features, and corresponding key algorithms have been designed accordingly.

[0204] Geometric Feature Class: For issues such as switch opening and closing, wire breakage, component deformation, component detachment, and column tilting, technologies such as straight line detection, contour matching, and change detection are used to implement algorithms for monitoring the closed state of crank arm type switches, the closed state of displacement type switches, and column tilting.

[0205] Texture Feature Class: For issues such as foreign object hanging and covering, and moving target intrusion, texture feature analysis, grayscale difference and discontinuity analysis are used to implement algorithms for moving target detection and foreign object covering detection.

[0206] Spectral Feature Class: For contact point discharge, local equipment heating, fire, etc., ultraviolet discharge monitoring, infrared temperature measurement and other methods are used to realize temperature monitoring, emergency state monitoring such as fire and multispectral feature analysis algorithms based on infrared features.

[0207] 4.1.2. Impact and Treatment of Environmental Factors at the Substation Site:

[0208] On-site monitoring is constrained by various factors. In geometric and texture feature identification, the impact of complex backgrounds with intersecting devices, low lighting conditions, and rain or snow is a key consideration. In spectral feature identification, the distance between the target device and the camera, ambient humidity, and the material characteristics of the device are crucial factors. The processing methods are described below from the perspectives of device segmentation, low-light conditions, rain / snow weather, and complex backgrounds.

[0209] 4.1.2.1 Equipment partitioning at the substation site:

[0210] To improve monitoring accuracy, the device to be monitored needs to be separated from irrelevant devices or the background. An image segmentation-based algorithm for segmenting and identifying devices against complex backgrounds was designed, employing a region-based watershed segmentation method.

[0211] Watershed segmentation method (steps actually used in this project):

[0212]  Calculate the segmentation function (the darker areas in the image are the segmentation targets);

[0213]  Calculate foreground markers (blob pixels connected within the object);

[0214]  Calculate the background markers (pixels that do not belong to any object);

[0215]  Modify the segmentation function so that it has a minimum value only at the foreground and background marker positions;

[0216] Perform watershed transformation calculations on the modified segmentation function.

[0217] To eliminate oversegmentation, the gradient image is thresholded. The gradient image is calculated using the Sobel operator, and the thresholding eliminates oversegmentation caused by small changes in grayscale.

[0218] Substation equipment image segmentation software: includes four functional parts:

[0219] Image region interactive selection: Provides four selection methods: rectangle, circle / ellipse, line, and polygon.

[0220] Precise image segmentation: The watershed algorithm is used to segment the selected region, which can successfully divide the image into multiple parts with precise and clear boundaries.

[0221] Image device basic information configuration: Number the segmented regions and enter their names, corresponding to the device information in the database.

[0222] Image device mask storage: The region is divided into regions in the form of a mask. The image number is a 32-bit unsigned integer. The high 22 bits are used for the device number and the low 10 bits are used for the device sub-region number.

[0223] 4.1.2.2 Equipment status identification under low light conditions:

[0224] For low-light scenes, three methods are used to enhance recognition capabilities: histogram equalization, adjusting the camera aperture, and adjusting the camera exposure time. The histogram equalization algorithm is the primary method used.

[0225] For grayscale images, let Represents grayscale The number of times it appears, and the total number of pixels is The number of gray levels is Then grayscale The probability of occurrence is:

[0226]

[0227] The cumulative probability function is:

[0228]

[0229] The transformation function is Map grayscale to The range is then transformed to the actual grayscale range. For color images, after conversion to HSI space, the luminance component is equalized, and then converted back to RGB. Experiments show that after histogram equalization, the device outline can be clearly distinguished.

[0230] 4.1.2.3 Equipment status identification in rainy and snowy weather:

[0231] Rain and snow can introduce noise into images, affecting feature recognition. A Gaussian filtering algorithm is used to remove speckled interference. Gaussian filtering is a low-pass filter that applies a weighted average to the entire image. Its Gaussian function is:

[0232]

[0233] In practical use, adopt or The template and weight distribution are shown in the original table. For Template, calculation formula is:

[0234]

[0235] The Gaussian filter weights in this project can be adjusted according to the image content. Combined with histogram equalization, the clarity of the device outline under rain and snow conditions is significantly improved, and interference from raindrops or snowflakes is essentially eliminated.

[0236] 4.1.2.4 State Identification in Complex Backgrounds:

[0237] The background of the equipment contains interference such as pebbles, weeds, and uneven ground. Gaussian filtering can also remove background interference. Experiments show that the outline of the disconnect switch is greatly affected by the grass before filtering, while the interference is effectively suppressed after filtering.

[0238] 4.1.3, Visual Recognition Algorithm Library for Typical Equipment States:

[0239] According to the requirements of the visual recognition technology research project, the image processing and recognition algorithms must meet the characteristics of portability (compatible with both Windows and Linux), modularity, simplicity, and efficiency. The algorithms are implemented using C / C++ and OpenCV.

[0240] 4.1.3.1 Algorithm for identifying the closed state of crank arm type disconnect switches:

[0241] The basic principle of the algorithm is as follows: Morphological processing is performed on the monitoring image of the disconnect switch to filter out noise and surrounding interference and obtain the disconnect switch mask; the edge is extracted by the Canny edge detection operator and significant straight lines are extracted by Hough transform, and the direction of the straight lines represents the direction of the disconnect switch; the average angle between the two arms of the straight lines is calculated based on the cosine theorem to determine the closed state.

[0242] Key sub-technology:

[0243]  Canny edge detection: includes three steps—Gaussian smoothing for noise reduction, calculating brightness gradient (using a 4-directional mask), and hysteresis thresholding to track edges (high threshold and low threshold).

[0244]  Hough transform line detection: mapping points in the rectangular coordinate system to the polar coordinate parameter space Points on the same straight line intersect at a single point in the parameter space, and the line parameters are obtained by statistically analyzing the peak values.

[0245]  Cosine theorem angle measurement: For the extracted line segment, calculate the included angle using the relationship between the side lengths of a triangle:

[0246]

[0247] in Let be the lengths of the three sides of the triangle.

[0248] Main algorithm flow:

[0249] Input: Switch image .

[0250] Image preprocessing (histogram equalization, Gaussian filtering);

[0251]  Maximum entropy threshold segmentation is used to obtain the switch mask, and opening operation is used to remove noise and cable interference;

[0252] Canny edge detection;

[0253]  Subtracting the self-erosion result from the mask yields the edge of the knife gate;

[0254] The Hough transform statistically identifies the most significant line, and its direction is used as the significant direction of the switch.

[0255] If the straight line is in the closed area of ​​the switch, calculate the angle between the two arms; if it is in the open area, calculate the angle with the horizontal direction; if no switch is found in either area, return "Not in closed or open position".

[0256] Output: Knife switch angle and status description.

[0257] State determination: Compare the measured included angle with the standard closed / open included angle in the database:

[0258] If the angle deviation from the standard closed state is less than the threshold, it is judged as closed in place; otherwise, it is not closed in place.

[0259] If the angle deviation from the standard open state is less than the threshold, it is judged as fully open; otherwise, it is not fully open.

[0260] If the algorithm returns "not in position", it is determined that the switch is not in the closed or open position.

[0261] 4.1.3.2 Algorithm for Identifying the Closure State of Displacement-Type Switches:

[0262] The basic principle of the algorithm is as follows: Based on the existence of stationary and moving ends in displacement-type disconnectors, the wavelet floating threshold method is used to extract the boundary contours of the two ends, calculate the actual distance between the two ends, and determine the displacement distance.

[0263] Key sub-technology:

[0264] (1) Wavelet transform edge detection:

[0265] Let a two-dimensional smoothing function be defined. To satisfy the normalization and attenuation conditions, two-directional wavelets are defined:

[0266]

[0267] scale ( The lower wavelet is:

[0268]

[0269] image The wavelet transform components are , The gradient magnitude is:

[0270]

[0271] The edge is located at a local maximum of the modulus. The main process of the wavelet transform algorithm (including the construction of low-pass and high-pass filters, upsampling, and multi-level decomposition) is implemented according to the steps in the original text.

[0272] (2) Floating threshold method: The boundary contour is tracked by gradually decreasing the threshold; when a breakpoint occurs, the threshold is reduced and the search is restarted until the boundary is complete. The algorithm records the length of the breakpoint; when it exceeds a set value... When the threshold is reached, return to the initial position of the breakpoint and reduce the threshold to re-binarize. Main algorithm flow:

[0273] Input: Switch image Pixel resolution factor .

[0274] Image preprocessing (histogram equalization, Gaussian filtering);

[0275] K-wavelet transform yields the contour image;

[0276] The floating threshold method is used to extract the boundary coordinates of the stationary and moving ends respectively;

[0277]  Calculate the distance between boundaries, multiply by the resolution factor to obtain the actual displacement distance.

[0278] Output: Displacement distance of the disconnector.

[0279] Status determination: Compare the calculated displacement distance with the displacement reference threshold in the database when the switch is fully closed: if it is less than the threshold, no alarm is triggered; otherwise, an alarm is triggered stating "Displacement switch not fully closed".

[0280] 4.1.3.3 Substation Foreign Object Intrusion Detection Algorithm:

[0281] The basic principle of the algorithm is based on target region contour matching (difference of the first component of the Hu moment) or texture feature differences for detection. The Canny algorithm is used to characterize the contour, and the first component of the Hu moment is calculated. By comparing the image to be detected with a reference image (without cover) Similarity is used to determine whether there is a foreign object covering the surface.

[0282] Hu moment calculation:

[0283] Normalized central moments Central Moment ,in , ordinary moments .

[0284] Algorithm flow:

[0285] Input: Surveillance image .

[0286]  Load the image to be detected and the reference image;

[0287]  Preprocessing (histogram equalization, Gaussian filtering), Canny edge detection;

[0288]  Calculate the Hu moments of the contour within the preset target area;

[0289] μ Calculate two images Differences and similarities (the rate of change of the object to be detected relative to the reference).

[0290] Output: Similarity measurement results.

[0291] Status determination: Based on the similarity and comparison with the alarm threshold in the database, return "No foreign object in device" or "Foreign object present".

[0292] 4.1.3.4 Substation Moving Target Monitoring Algorithm:

[0293] A static reference template is trained using Codebook and connected component analysis. The current image is compared with the template to obtain a mask. The difference between the current image and the neighboring frames and the training frames is analyzed using background subtraction to obtain masks respectively. The three masks are fused to obtain the final moving target detection result.

[0294] (1) Codebook algorithm:

[0295] A codebook (CB) is created for each pixel, and each CB contains multiple codewords (CWs). CW = {High, Low, max, min, t_last, stale}. During training, if the current pixel value matches a CW (Low ≤ I(x,y) ≤ High), then update max / min and their upper and lower bounds; otherwise, create a new CW. During detection, if a CW exists... and If it is, then it is the background; otherwise, it is the foreground.

[0296] (2) Connected component analysis: Mark the connected components in the binary image, delineate the gate for each target, establish a multi-target location list, and separate or merge targets through erosion and dilation.

[0297] (3) Symmetric difference method (background subtraction method):

[0298] Input three consecutive frames :

[0299] φ Local contrast enhancement;

[0300]  Calculate the difference , ;

[0301]  Smoothing filtering and threshold denoising ;

[0302] μ Logical AND by pixel ;

[0303] The target mask is obtained through morphological post-processing. The output corresponds to... The goal.

[0304] Status determination: Determine whether there is a moving target within the monitoring range based on whether the rectangle is empty, and return "Motion detected" or "Motion not detected".

[0305] 4.1.3.5 Algorithm for monitoring emergency operating conditions of equipment:

[0306] In response to emergencies such as overheating, smoke, and fire, a combination of motion detection and infrared thermography is used. The temperature in the fire area rises rapidly, while the flames and smoke are in motion.

[0307] Algorithm flow:

[0308] Input: Surveillance image (infrared or visible light).

[0309]  Load image;

[0310]  Generate a sequence of training images and a sequence of multiple images preceding the current frame;

[0311]  Moving target detection based on Codebook+ connected component analysis;

[0312] μ Based on background subtraction, differences are detected (with the training image and with neighboring multiple frames), resulting in two masks;

[0313]  Combine the three masks;

[0314] The image is an infrared image; the highest temperature captured in the monitored image is compared with the fire alarm threshold. Comparison: If the temperature exceeds the threshold and movement is detected, it is considered a fire and smoke, and the system returns to the fire location; otherwise, it is considered normal.

[0315] Output: Fire and smoke detection and location.

[0316] Status determination: Based on the detected movement and temperature exceeding the threshold, determine whether a fire has occurred.

[0317] 4.2 Safety monitoring at the substation working site:

[0318] By combining data such as the spatial relationship between the target equipment and the operator, the time the operator spends in the target area, and the current operating status of the equipment, it is determined whether the operation meets the standard requirements, thereby avoiding omissions in steps or violations of regulations during the operation.

[0319] The system can acquire positioning data of multiple moving targets based on sequence images, and combine this with motion information such as angular acceleration and linear acceleration obtained by MEMS to perform real-time positioning and monitoring of the status of workers and equipment, in order to prevent safety hazards and dangerous phenomena caused by health problems, operational errors, etc.

[0320] 4.3 Integration of task target status detection and equipment status:

[0321] To detect multiple moving targets in substation monitoring videos, a Gaussian mixture model based on image morphology processing is employed. Since the background often remains relatively constant compared to moving targets over long periods, background subtraction can easily detect moving objects. However, the background changes slowly with lighting and environmental factors, necessitating continuous updates to the background image, requiring the establishment of a background model and an update algorithm. The Gaussian mixture model, based on the statistical characteristics of background samples, performs Gaussian mixture modeling on each pixel. It is insensitive to subtle background changes and can objectively and accurately represent the background information of the image.

[0322] 4.4 Multi-site collaborative violation identification and resource scheduling strategy:

[0323] 4.4.1 Real-time target identification and filtering:

[0324] The system first filters out the sites that most need real-time monitoring based on the configuration data of each monitoring target (work site) and the unidentified personnel data.

[0325] Monitoring target: Sites currently undergoing power maintenance, testing, or construction.

[0326] Behavior recording terminal configuration data: including the number of wearable recorders at each site.

[0327] Unidentified staff data: This refers to situations where, within the most recent preset time period, the staff member has not been identified or processed for any violations among all the monitoring targets they are in.

[0328] The screening process is as follows:

[0329] If there are no "unmonitored personnel" (i.e., all of them have been identified before) on site, they will not be listed as real-time identification targets.

[0330] If all staff members on site are not monitored, they will be directly listed as real-time identification targets (high risk).

[0331] If the number of behavior recording terminals configured is less than a preset threshold (e.g., 5), the recognition reliability is insufficient and it will not be listed as a real-time recognition target.

[0332] Otherwise, calculate the monitoring matching factor:

[0333]

[0334] When the factor is greater than the preset matching factor threshold, the monitored target is listed as a real-time identification target.

[0335] This screening method ensures that limited computing resources are prioritized for high-risk sites with many regulatory blind spots and sufficient equipment.

[0336] 4.4.2 Remaining battery power observation strategy for potential targets:

[0337] For potential targets that are not selected as real-time identification targets, the system dynamically determines the observation frequency of the remaining battery power of the behavior recording terminal based on its risk characteristics, so as to prevent monitoring failure due to abnormal drop in device battery power.

[0338] The observation period is divided into two intervals:

[0339] First remaining power range (e.g., power below 60%): Applicable to high-risk potential targets, allowing for early intervention and observation.

[0340] Second remaining power range (e.g., power below 20%): suitable for lower-risk targets, delaying observation to conserve resources.

[0341] The specific interval to be used is determined by the following decision-making process:

[0342] If the total number of potential targets exceeds the preset threshold (e.g., 5), the first interval (high alert) will be used directly.

[0343] Otherwise, if the number of potential targets with unidentified personnel exceeds the preset target number threshold (e.g., 3), the first interval will be used.

[0344] Otherwise, calculate the global unidentified personnel ratio (total number of unidentified personnel / total number of staff among potential targets). If this ratio is lower than the preset matching factor threshold (e.g., 0.2), the second interval can be used; otherwise, proceed to the next step.

[0345] Calculate the overall matching factor:

[0346]

[0347] If the overall matching factor is greater than the preset value of the matching factor (e.g., 0.1), the first interval is used; otherwise, the second interval is used.

[0348] Through the aforementioned tiered decision-making, the system achieves differentiated and refined management of on-site power consumption at potential risks.

[0349] 4.4.3 Strategy for Switching and Handling Violations:

[0350] When no violations are detected in a real-time identified target for a period of time, but a potential identified target shows abnormal power or high-risk signals, the system should be able to intelligently switch the identified target.

[0351] Anomaly identification target: Through remaining battery power observation and analysis, determine the location of terminals recording abnormal changes such as sudden battery drops. The switching rule is:

[0352] Emergency Priority: If the remaining battery power of all terminals in a potential target is lower than the target's remaining battery power range (e.g., <10%), then immediately add it to the real-time target identification list and complete monitoring before the power outage.

[0353] High-risk level response: If all terminals in a certain anomaly identification target are abnormal and the proportion of unidentified personnel is higher than a preset value (e.g., 70%), then pause a real-time identification target that is "not fruitful in a short period of time" (e.g., no violation was found within 30 minutes), switch to identify the high-risk target for a fixed period of time (e.g., 10 minutes), and then switch back.

[0354] Related Risk Response: If the proportion of unidentified personnel does not reach the special threshold, calculate the average number of overlapping abnormal targets for the identified abnormal target (i.e., the number of other abnormal targets in the same low power range). If the average is greater than the preset threshold (e.g., 2), suspend the "short-term no-result" target; otherwise, suspend the "long-term no-result" target (e.g., no violation found within 60 minutes) and perform temporary identification processing.

[0355] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications made based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for identifying and controlling personnel behavior in substations based on artificial intelligence, comprising the following steps: S1. Multispectral fusion detection steps with region constraints: acquire visible light and infrared image sequences, perform watershed segmentation on the infrared image to generate three types of region masks; use the region masks as spatial priors to perform adaptive weighted fusion of visible light and infrared images, with the infrared weight at the boundary increasing with the gradient magnitude. Codebook background modeling is performed with the background region mask as a spatial constraint, and the codeword state of the target region and scene region is frozen; the codebook foreground probability map and the motion edge map of the symmetric difference method are fused to generate a comprehensive moving target mask; S2, Binocular localization and tracking steps with prediction feedback: The feature search range is limited by using the comprehensive moving target mask as the region of interest constraint, and the target's three-dimensional coordinates are calculated after cross-camera matching; the extended Kalman filter is initialized with the three-dimensional coordinates to predict the position and velocity at the next moment; the predicted position deviation is fed back to the feature search step, and the region of interest is dynamically adjusted with the deviation direction as the search offset direction and the magnitude as the window expansion amount; Maintain tracking in predicted state when crossing the field of view or when occluded; S3. Spatiotemporal correlation identification steps for status code mapping: Perform geometric, texture, and spectral feature identification on the device to generate device status codes; Establish a safety situation map by aligning the target's 3D trajectory with the equipment status code in time and space; dynamically adjust the safety zone boundary and permitted time limit based on the equipment status code; detect collisions between personnel trajectory and dynamic safety zone; compare the dwell time with the permitted time limit; match the operation sequence with the status code change sequence; and trigger graded alarms when abnormalities occur. S4. Risk-coupled multi-site dispatching steps: Construct a risk feature vector that includes the proportion of unidentified personnel, terminal configuration redundancy, power attenuation rate, and alarm frequency. The monitoring matching factor is generated by the coupled calculation of the proportion of unidentified personnel and the redundancy of terminal configuration. The contribution of configuration redundancy is nonlinearly amplified as the proportion of unidentified personnel increases. When the matching factor exceeds the threshold, the video stream bitrate is increased and priority resources are allocated; when the power attenuation rate exceeds the slope threshold, key frame pre-extraction is triggered; when the alarm frequency is lower than the preset value and the potential on-site matching factor exceeds the switching threshold, the migration extended Kalman filter maintains the continuity of the tracking state.

2. The method for identifying and controlling personnel behavior in substations based on artificial intelligence according to claim 1, characterized in that: The nonlinear mapping in step S1 includes: taking the gradient magnitude determined by the watershed segmentation as input, generating fusion weights through a mapping function with a sharpness coefficient, wherein the mapping function makes the visible light weights of pixels with gradient magnitudes below the boundary threshold dominant, the infrared weights of pixels with gradient magnitudes above the boundary threshold dominant, and the weights of pixels with gradient magnitudes in the neighborhood of the boundary threshold transition smoothly.

3. The method for identifying and controlling personnel behavior in substations based on artificial intelligence according to claim 1, characterized in that: The codebook background modeling in step S1 includes: establishing a time-series codebook containing multiple codewords for each pixel; during training, if the current pixel value matches the upper and lower bounds of an existing codeword, the historical extreme values ​​of that codeword are updated and the learning boundary is dynamically adjusted; otherwise, a new codeword is created; during detection, if the current pixel value falls within the extreme value tolerance range of any codeword, it is judged as background; otherwise, it is judged as foreground. The codeword update is performed only at pixel positions covered by the background region mask, and the target region mask and scene region mask freeze the codeword state at pixel positions, thus limiting the learning range of the background model to spatial constraints.

4. The method for identifying and controlling personnel behavior in substations based on artificial intelligence according to claim 1, characterized in that: The prediction feedback in step S2 includes: the deviation vector between the predicted position output by the extended Kalman filter and the actual observed position in the previous frame is smoothed and filtered to generate displacement deviation estimates and velocity deviation estimates; a dynamic region of interest is generated with the magnitude of the displacement deviation estimate as the radius expansion amount and the predicted position as the center; the direction of the velocity deviation estimate is used as the search window offset direction; when the magnitude of the displacement deviation estimate exceeds a preset upper limit, the search window is expanded to a predetermined proportion of the entire image range; when the magnitude of the displacement deviation estimate is lower than a preset lower limit, the search window is shrunk to a predetermined multiple of the target template size.

5. The substation personnel behavior recognition and control method based on artificial intelligence according to claim 1, characterized in that: S3 The dynamic safety zone in the steps includes: dividing the space around the equipment into a live safety zone and a power outage work zone based on the disconnection / opening status code; when the disconnection / opening status code indicates that the equipment is live, the radius of the live safety zone is a first threshold, and the power outage work zone is closed; when the disconnection / opening status code indicates that the equipment is de-energized, the radius of the live safety zone shrinks to a second threshold, which is less than the first threshold, and the power outage work zone is opened; real-time collision detection is performed between the personnel trajectory and the boundary of the dynamic safety zone to generate an intrusion alarm.

6. The method for identifying and controlling personnel behavior in substations based on artificial intelligence according to claim 1, characterized in that: The spatiotemporal alignment in step S3 includes: establishing a unified timestamp index and interpolating and aligning the sampling time of the target three-dimensional trajectory sequence with the generation time of the device status code; projecting the three-dimensional coordinates of the target three-dimensional trajectory sequence onto the device coordinate system and calculating the real-time spatial distance between personnel and equipment; determining that an intrusion into the dynamic safety zone occurs when the real-time spatial distance is less than the safety radius threshold associated with the switch opening / closing status code; and determining that overstaying occurs when the duration of personnel's continuous stay within the safety radius threshold exceeds the permitted time limit associated with the displacement status code.

7. The substation personnel behavior recognition and control method based on artificial intelligence according to claim 1, characterized in that: The video stream keyframe pre-extraction mechanism in step S4 includes: when the remaining battery power of the terminal is detected to be lower than the first battery threshold, the video stream encoding frame rate is reduced and keyframes containing moving targets are retained first; when the remaining battery power of the terminal is detected to be lower than the second battery threshold, the video stream encoding frame rate is further reduced, and the local caching of the integrated moving target mask in step S1 and the target 3D trajectory in step S2 is triggered; when the remaining battery power of the terminal is detected to be lower than the third battery threshold, the video stream transmission is stopped and the local cached security situation map snapshot is uploaded to the cloud.

8. The method for identifying and controlling personnel behavior in substations based on artificial intelligence according to claim 1, characterized in that: The dynamic migration of computing resources in step S4 includes: when performing the dynamic migration, serializing and packaging the extended Kalman filter state vector, error covariance matrix, and target 3D trajectory buffer of the source real-time identification site into a migration data packet; transmitting the migration data packet to the target potential identification site, reconstructing the tracking state of the extended Kalman filter at the target site, and using the last predicted position in the migration data packet as the initial search center of the target site to maintain the cross-site tracking continuity of the same target; The coupling calculation in step S4 includes: the monitoring matching factor is a nonlinear mapping of the weighted product of the proportion of unidentified personnel and the redundancy of terminal configuration after adjustment by alarm frequency, and the alarm frequency is used as an adjustment coefficient to amplify the monitoring matching factor of the high alarm frequency site and attenuate the monitoring matching factor of the low alarm frequency site.

9. An AI-based substation personnel behavior recognition and control system, comprising a multispectral image acquisition module, a regional constraint fusion detection module, a closed-loop feedback positioning and tracking module, a spatiotemporal correlation fusion identification module, and a multi-site collaborative scheduling module, characterized in that: A multispectral image acquisition module includes a parallel binocular visible light camera and an infrared camera, used to simultaneously acquire visible light image sequences and infrared image sequences. The cameras are jointly calibrated to have a unified extrinsic parameter matrix. The region-constrained fusion detection module, used to execute step S1 of claim 1, includes: a watershed segmentation submodule, generating masks for the target region, scene region, and background region; an adaptive weighted fusion submodule, using the region masks as spatial prior weights to fuse visible light and infrared images, wherein the fusion weights at the target boundary are dynamically adjusted by gradient magnitude through nonlinear mapping; a codebook background modeling submodule, using the background region mask as a spatial constraint to perform background modeling, freezing codeword states in the target region and scene region; and a symmetric difference fusion submodule, generating a comprehensive moving target mask. A closed-loop feedback positioning and tracking module is used to execute step S2 of claim 1, comprising: a region of interest constraint submodule, which limits the feature search range with the comprehensive moving target mask; a binocular matching positioning submodule, which calculates the three-dimensional spatial coordinates of the target; an extended Kalman filter tracking submodule, which outputs the predicted position and velocity; and a prediction feedback interface, which feeds back the predicted position deviation to the region of interest constraint submodule, and dynamically adjusts the search window by using the deviation direction as the search window offset direction and the modulus as the radius expansion amount. The spatiotemporal correlation fusion identification module is used to execute the S3 step of claim 1, including: a geometric feature identification submodule, a texture feature identification submodule, a spectral feature identification submodule, and a security situation map construction submodule. The security situation map construction submodule spatiotemporally aligns the target's three-dimensional trajectory with the device status code and dynamically adjusts the security area boundary and permission time limit based on the device status code. A multi-site collaborative scheduling module is used to execute step S4 as described in claim 1, comprising: a risk feature vector construction submodule, a monitoring matching factor calculation submodule, a bitrate dynamic adjustment submodule, a key frame pre-extraction submodule, and a computing resource migration submodule. The monitoring matching factor calculation submodule generates a monitoring matching factor based on the coupled calculation of the proportion of unidentified personnel and the redundancy of terminal configuration. The coupled calculation makes the contribution of terminal configuration redundancy nonlinearly amplified as the proportion of unidentified personnel increases.

10. A substation personnel behavior recognition and control system based on artificial intelligence according to claim 9, characterized in that: it also... include: The image information configuration module is used to provide interactive calibration of the device area, supporting four area selection methods: rectangular, circular / elliptical, linear, and polygonal. It uses a watershed algorithm to accurately segment the selected area and stores the area division in the form of a mask. The area mask is output to the area constraint fusion detection module as an initialization flag for the watershed segmentation submodule. An environmental adaptability processing module is used to perform histogram equalization under low light conditions and output the equalized image to the region constraint fusion detection module; and to perform Gaussian filtering under rain and snow conditions and output the filtered image to the region constraint fusion detection module. The smoothing intensity of the Gaussian filter is dynamically adjusted according to the region type output by the watershed segmentation submodule, with a strong smoothing template used for the target region and a weak smoothing template used for the background region.

Images