A power grid monitoring bionic robot driven accident automatic analysis and voice reporting method

Abstract

The present application belongs to the technical field of power detection, and particularly relates to a power grid monitoring bionic robot driven accident automatic analysis and voice reporting method; the method comprises the following steps: step 1: the power grid monitoring bionic robot terminal inspects at a target site, collects video data, audio data and environmental state data; a perception data block set is generated; step 2: a solidification dictionary atom library is loaded, and a visual observation block, an acoustic observation block and an environmental observation block and corresponding residual error masks are respectively established; step 3: within a fixed time window of 100 milliseconds, unexplained conflicts are formed by regions that have not been marked; step 4: based on a candidate explanation set, an accident determination result, a severity level and an influence range are generated according to a preset rule; the present application has the beneficial effects of high detection accuracy, traceable explanation results, complete reporting content and direct guidance for emergency disposal.

Description

Technical Field

[0001] This invention belongs to the field of power detection technology, specifically relating to a method for automatic accident analysis and voice notification driven by a bionic robot for power grid monitoring. Background Technology

[0002] Rapid detection, accurate analysis, and effective reporting of accidents during power grid operation have always been crucial aspects of ensuring safety in the power industry. With the continuous expansion of the power grid, the number of transmission lines, substations, and distribution stations has increased rapidly, and the operating environment has become increasingly complex. Traditional power grid monitoring mainly relies on manual inspections and video surveillance from centralized monitoring centers. While this approach met certain needs in the early stages, it has significant shortcomings. On the one hand, manual inspections are time-consuming and infrequent, making it difficult to capture early signs of accidents in a timely manner. On the other hand, although video surveillance enables 24 / 7 data collection, the sheer volume of monitored data makes it highly susceptible to oversights and delays if entirely dependent on manual monitoring, thus reducing the timeliness of accident response.

[0003] In recent years, with the development of artificial intelligence, sensor networks, and robotics, researchers have gradually attempted to use autonomous inspection robots to replace manual labor in complex power grid environments. These robots are often equipped with cameras, microphones, temperature sensors, and gas detection sensors, enabling them to acquire images, sounds, and environmental information from the scene. Simultaneously, deep learning-based image recognition and speech analysis methods have been used to process this data, achieving automatic detection of spark discharges, smoke diffusion, abnormal noise, and sudden temperature changes in some scenarios. However, these existing methods often rely on large amounts of training data and complex neural network models, making their detection results highly sensitive to the distribution of training samples. When encountering novel accident manifestations that have not been learned from, they are prone to misjudgment or missed detection. Furthermore, deep learning models typically operate as black boxes, making it difficult to provide traceable evidence for accident explanations, leading to a lack of trust among operators when receiving the results. Summary of the Invention

[0004] The main objective of this invention is to provide a method for automatic accident analysis and voice notification driven by a bionic robot for power grid monitoring. The method includes: Step 1: The bionic robot terminal for power grid monitoring inspects the target site, collecting video data, audio data, and environmental status data. After standardizing the three types of data, the time location, spatial location, and source type are labeled according to unified fields to generate a set of perception data blocks. The time location is a millisecond-level timestamp; the spatial location is a two-dimensional or three-dimensional coordinate within the station. If the location cannot be determined, the device number or the collection point number is recorded; the source type is marked as one of video / audio / environment.

[0005] Step 2: Load the solidified dictionary atomic library, establish visual observation blocks, acoustic observation blocks and environmental observation blocks and corresponding residual masks respectively, and execute the sparse coding reconstruction process to obtain the interpreted region through template matching and mask updating.

[0006] Step 3: Within a fixed time window of 100 milliseconds, unmarked regions are considered unexplained conflicts, and a candidate explanation set is generated based on the simultaneous occurrence relationship of selected dictionary atoms. The adversarial game engine is started, and the action of the incident explanation party is to cover unexplained conflicts, while the action of the normal explanation party is to mark perturbations that are allowed according to the compliance rule checklist. In each round, the dominance strategy is eliminated first, and then both parties simultaneously select actions and score: if the number of unexplained conflicts decreases, the incident explanation party gets 1 point, and if the number of unexplained conflicts marked as compliant increases, the normal explanation party gets 1 point. The game ends when both parties take the same action for 2 consecutive rounds or the above two quantities remain unchanged for 2 consecutive rounds.

[0007] Step 4: Based on the candidate explanation set, generate the accident judgment result, severity level and impact scope according to the preset rules, form structured accident elements and output them to the notification terminal; the notification terminal synthesizes broadcast text based on the structured accident elements and broadcasts it.

[0008] Furthermore, in step 1, the standardization process for video data, audio data, and environmental state data includes: dividing the video data into equal-sized blocks with sides of 32 pixels per frame to obtain a block sequence; converting each block into grayscale values, with grayscale values ​​ranging from 0 to 255, where grayscale values ​​greater than or equal to 200 are considered bright pixels and those less than 200 are considered dark pixels; dividing the audio data into blocks with a frame length of 20 milliseconds and a step size of 10 milliseconds; recording each sample point with an amplitude of zero or a positive value as bit 1 and a negative value as bit 0 to obtain a frame-by-frame bit sequence; and recording the environmental state in chronological order as a discrete event sequence, with event names selected from the following set: temperature transition, access control opening, gas detection trigger, and wind speed transition; each record includes an event name, start time, and end time.

[0009] Furthermore, in step 2, the solidified dictionary atom library includes: visual dictionary atoms, acoustic dictionary atoms, and environmental dictionary atoms; the visual dictionary atom is a binary template with a size of 32×32 pixels; the acoustic dictionary atom is a bit template with a duration of 20 milliseconds; and the environmental dictionary atom is a discrete event template.

[0010] Further, in step 2, within any continuous time window, the tile sequence, frame-by-frame bit sequence, and discrete event sequence are extracted and denoted as visual observation block, acoustic observation block, and environmental observation block, respectively. A residual mask of the same dimension is established for each of the three types of observation blocks, initially in an uninterpreted state. For the visual observation block, a sparse coding reconstruction process is executed, including: for each tile, each visual dictionary atom is translated and aligned within the tile at four rotation angles of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and the pixel consistency count is calculated, i.e., the number of bright pixels of the tile where the template foreground falls. If the pixel consistency count of a certain visual dictionary atom is improved by no less than 10 pixels compared to the currently interpreted part, then the visual dictionary atom and the aligned position are selected, and the tile pixels corresponding to the template foreground are marked as interpreted on the visual residual mask. This process is repeated for the same tile until the above improvement cannot be achieved or the interpreted area reaches 90% of the tile pixels.

[0011] Further, in step 2, for the acoustic observation block, a sparse coding reconstruction process is performed, including: for each audio frame, aligning each acoustic dictionary atom point by point within the frame according to the starting sample, and calculating the bit consistency count; if the bit consistency count of an acoustic dictionary atom is improved by no less than 50 sample points compared to the currently interpreted portion, then the acoustic dictionary atom and the aligned position are selected, and the corresponding sample segment is marked as interpreted on the acoustic residual mask; repeating this process for the same frame until the above improvement cannot be achieved or the interpreted segment covers 90% of the frame length; for the environmental observation block, a sparse coding reconstruction process is performed, including: for each event record in the environmental observation block, if its event name is the same as that of an environmental dictionary atom and its time interval does not overlap with the interpreted segment, then marking the segment as interpreted on the environmental residual mask; repeating this process for the same sequence until the interpreted segment covers 90% of the time span of the sequence or there are no uninterpreted records with the same name.

[0012] Furthermore, in step 2, among the three types of residual masks, regions that are still in an unexplained state are formed into an unexplained conflict set: visual uses 4-neighbor connectivity to generate pixel clusters, acoustic uses continuous unexplained sample segments, and environmental uses unexplained event records; each unexplained conflict is accompanied by source type, temporal location, and spatial location; with a time window of 100 milliseconds as the boundary, the selected visual dictionary atoms, acoustic dictionary atoms, and environmental dictionary atoms within the same window are combined into candidate explanation entries; each candidate explanation entry is linked to at least one unexplained conflict.

[0013] Furthermore, in step 3, the execution process of the adversarial game engine includes: setting up two participants, an incident interpreter and a normal interpreter; the incident interpreter's action is to select several items from the candidate interpretation set to maximize the number of linked unexplained conflicts; the normal interpreter's action is to select several items from the compliance rule checklist to maximize the number of unexplained conflicts marked as compliant; at the beginning of each round, dominance strategy elimination is performed on the action sets of both parties: if an action does not increase the score of one party and does not decrease the score of the other party in all comparable cases, then the action is deleted; after the dominance strategy elimination is completed, both parties simultaneously select and execute actions; if the total number of unexplained conflicts at the end of the round is less than that at the beginning of the round, the incident interpreter gets 1 point; if the number of unexplained conflicts marked as compliant at the end of the round is more than that at the beginning of the round, the normal interpreter gets 1 point; the round terminates when the same pair of actions is repeated synchronously for two consecutive rounds, or the number of unexplained conflicts and the number of unexplained conflicts marked as compliant remain unchanged for two consecutive rounds.

[0014] Furthermore, in step 4, if the unexplained conflict set is empty, output a normal operation judgment; if the unexplained conflict set is not empty, output an accident type according to preset rules.

[0015] Furthermore, in step 4, the process of generating the severity level includes: if the number of pixels of any visually unexplained conflict is greater than or equal to 20, or the length of any acoustically unexplained conflict covers 50% of the frame length, or any environmentally unexplained conflict spans two or more consecutive records, it is judged as high-level; if none of the above conditions are met and the set of unexplained conflicts is not empty, it is judged as medium-level; the process of generating the scope of influence includes: if the unexplained conflict involves two or more adjacent tiles, or involves two or more consecutive audio frames, or involves two or more consecutive environmental records, the scope is multi-point; otherwise, it is single-point.

[0016] This invention discloses a method for automatic accident analysis and voice notification driven by a bionic robot for power grid monitoring, which has the following beneficial effects: it can automate the entire process from data acquisition to accident notification in the complex operating environment of the power grid, resulting in significant benefits. First, through on-site inspections by a bionic robot terminal for power grid monitoring, the system can simultaneously acquire three types of data: video, audio, and environmental status. Standardization processing is performed with millisecond-level temporal and spatial coordinate accuracy to ensure the integrity and consistency of information, improving the reliability of accident detection from the source. Second, this invention utilizes a fixed dictionary atomic library and a sparse coding reconstruction process to combine abnormal patterns in video observation blocks, acoustic observation blocks, and environmental observation blocks with residual masks, thereby gradually eliminating noise interference during template matching and accurately identifying unexplained conflicts. This approach avoids the shortcomings of relying on large-scale training samples and maintains stability and traceability even with novel accident manifestations not previously seen. Furthermore, this invention introduces an adversarial game engine, allowing accident interpreters and normal interpreters to interact within a candidate interpretation set. Under the action of dominance strategy elimination and scoring mechanisms, a balanced accident interpretation result is ultimately output. This design not only ensures the rationality of the results but also provides transparency and verifiability to the accident determination process. Finally, the structured accident elements generated based on the candidate interpretation set encompass the accident determination results, severity level, and scope of impact. The notification terminal can directly synthesize and broadcast notification texts for dispatching, operation and maintenance, and communication, making the notification content more targeted and practical. The implementation of this invention can significantly shorten the time from accident occurrence to emergency response, improve the efficiency and accuracy of accident handling, reduce the rate of misjudgment and missed judgment, and enhance the overall safety level of power grid operation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a method for automatic accident analysis and voice notification driven by a bionic robot for power grid monitoring, provided in an embodiment of the present invention. Detailed Implementation

[0019] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0020] refer to Figure 1 A method for automatic accident analysis and voice notification driven by a bionic robot for power grid monitoring, the method comprising: Step 1: The bionic robot terminal for power grid monitoring inspects the target site, collecting video data, audio data, and environmental status data. After standardizing the three types of data, the time location, spatial location, and source type are labeled according to unified fields to generate a set of perception data blocks. The time location is a millisecond-level timestamp; the spatial location is a two-dimensional or three-dimensional coordinate within the station. If the location cannot be determined, the device number or the collection point number is recorded; the source type is marked as one of video / audio / environment.

[0021] In the specific implementation process, after the power grid monitoring bionic robot terminal enters the target site, it initiates the synchronous acquisition of video data, audio data, and environmental status data. To ensure time and location consistency, the power grid monitoring bionic robot terminal first synchronizes with the time source within the site, ensuring that its local clock is consistent with the time source within milliseconds. If there is no time source within the site, it synchronizes with the time signal from the dispatch control center. After time synchronization, video data standardization, audio data standardization, and environmental status data standardization are performed sequentially, and each data record is labeled with a unified field indicating the time location, spatial location, and source type, ultimately generating a set of sensing data blocks.

[0022] The bionic robot terminal for power grid monitoring captures images at a fixed frame rate. To reduce blurring caused by motion, the attitude information from the inertial measurement unit is used to geometrically align adjacent frames, followed by light deblurring. This improves detail contrast without increasing computational burden, making subsequent grayscale determination more reliable. If lens distortion exists, radial and tangential corrections are performed first; if the scene contains obvious vertical or horizontal lines, these benchmarks are used for one-time calibration, and the calibration results are reused throughout the inspection, thus ensuring a stable scale for spatial position estimation. Each frame is divided into equally divided blocks with sides of 32 pixels. The divided blocks are uniformly converted to a grayscale range of 0 to 255 and linearly stretched to ensure the grayscale distribution of the frame covers the entire range. The benefit of linear stretching is that it offsets the overall darkness or brightness caused by illumination fluctuations, thus ensuring that the fixed threshold remains effective under different lighting conditions. Within each block, a grayscale value of 200 is used as a fixed threshold to classify pixels into bright and dark categories. The reason for choosing a threshold of 200 is that electric arcs, sparks, and highly reflective metal surfaces will produce obvious bright areas under normal exposure. Setting the threshold at a high level can suppress misjudgments caused by ordinary reflections and shadow changes. At the same time, combined with the aforementioned linear stretching, it can maintain consistency in judgment without relying on environmental adaptive parameters. Millisecond-level timestamps are recorded for each frame and each tile, and the source type is marked as video. This allows for precise alignment of individual tiles with audio frames and environmental events within the same time window. If the power grid monitoring bionic robot terminal has in-station 2D or 3D positioning capabilities, the camera pose is recorded for each frame and projected onto the in-station coordinate system to obtain the spatial position of the tile. If it also has depth acquisition capabilities, the main peak closer to the camera side is selected as the target depth range in the depth histogram, and the pixels within this range are used for position estimation. The main peak closer to the camera side is chosen because glass reflections and strong light from afar in power grid scenarios easily form secondary peaks at a distance. Using the main peak closer to the camera side can avoid mistaking reflections or background highlights for foreground objects, thereby improving the reliability of spatial position. If the location cannot be determined, record the device number or the data collection point number, and then perform offline mapping by matching the number with the site layout map.

[0023] The bionic robot terminal for power grid monitoring collects sound waves at a fixed sampling rate and performs band-limiting processing at the input to suppress stray noise exceeding the bandwidth. This reduces the masking effect of external broadband noise on short-duration impact sounds. Audio data is segmented with a frame length of 20 milliseconds and a step size of 10 milliseconds. This frame length and step size are chosen because 20 milliseconds can fully cover the main energy of an electrical discharge, while the 10-millisecond step size provides half overlap, improving temporal resolution without significantly increasing computational load and reducing the probability of short events being cut off across frames. For each sample point, a frame-by-frame bit sequence is generated based on the sign of the amplitude: non-negative values ​​are bits 1, and negative values ​​are bits 0. The benefit of using symbolic binarization is that differences in sound source distance and microphone sensitivity will change the amplitude, but will not change the sequence structure of positive and negative signs, thus maintaining comparability between different scenes without gain calibration. A millisecond-level timestamp is recorded for each frame, and the source type is marked as audio for alignment with tiles and environmental events within the same time window. If the bionic robot terminal for power grid monitoring has in-station positioning capabilities, the center position of the microphone array will be recorded as the in-station coordinates; if positioning is not possible, the device number or the acquisition point number will be recorded. If the array contains multiple pickup points, the array geometry can be recorded for subsequent positioning purposes, but only a uniform field content needs to be written in step 1.

[0024] Environmental conditions are recorded as discrete event sequences in chronological order. Event names are selected from the following set: temperature transition, access control opening, gas detection trigger, and wind speed transition. Each record includes the event name, start time, and end time. A temperature transition is determined by a temperature rise or fall reaching a set level within one second, reflecting a sudden change in thermal state. Such abrupt changes are often associated with overheating faults or abnormal cooling. Access control opening is triggered by a door magnetic switch changing from closed to open, clearly indicating the environmental changes that personnel entry may bring. Gas detection trigger is generated by a gas sensor alarm changing to an triggered state, directly reflecting a hazardous gas or gas leak. Wind speed transition is generated by a wind speed rise reaching a set level within one second, used to explain non-accident sources such as wire swaying and increased noise. Using discrete events avoids the problem of inconsistent scaling of continuous quantities across different devices, ensuring consistent availability of events across different sites. A millisecond-level timestamp is recorded for each event as the start and end times, and the source type is marked as environment. Spatial location is recorded as the sensor's two-dimensional or three-dimensional coordinates within the station; if location is not possible, the device number or data collection point number is recorded.

[0025] When the bionic robot terminal for power grid monitoring can self-locate within the station's coordinate system, it records the 2D or 3D coordinates of the camera center within the station for each frame and writes them into the tile record. If depth information is available, the main peak near the camera is selected as the target depth range on the depth histogram, and pixels within this range that meet the grayscale criteria for bright pixels are then used to estimate the tile's spatial position. Selecting the main peak near the camera eliminates misleading information caused by strong light and reflections from afar, making the estimation closer to the actual equipment surface. When station coordinates are unavailable, the device number or acquisition point number is recorded for video data, audio data, and environmental status data, respectively. The number has a unique correspondence in the station layout diagram, facilitating subsequent offline location recording using the number. Even when location is not possible, a unified field is still written to ensure data flow consistency and subsequent traceability. The source type is only allowed to have three fixed values: video, audio, or environment. Fixed values ​​ensure that subsequent steps can select the correct processing path based on the source type, avoiding pipeline interruptions caused by unrecognizable categories.

[0026] When there are significant changes in lighting within the station, after grayscale conversion, the entire frame is truncated using a histogram and then linearly stretched, before the brightness is determined using a threshold of 200. This enhances adaptability to extreme exposure scenes without changing the threshold. When the audio environment contains continuous strong noise, amplitude limiting is applied before framing, uniformly cropping samples exceeding the set amplitude to that amplitude before symbol binarization. Amplitude limiting can suppress the disruption of symbol sequence continuity caused by occasional saturation without changing the binarization rules. When only two-dimensional coordinates are available, the spatial position of the image patch is recorded as the station's two-dimensional coordinates within the image's plane, and the plane identifier of this coordinate system is marked in the record. If height information is subsequently added, it can be calculated by tracing back the image through the original content pointer without modifying the already written record.

[0027] Step 2: Load the solidified dictionary atomic library, establish visual observation blocks, acoustic observation blocks and environmental observation blocks and corresponding residual masks respectively, and execute the sparse coding reconstruction process to obtain the interpreted region through template matching and mask updating.

[0028] Specifically, the fixed dictionary atom library includes visual dictionary atoms, acoustic dictionary atoms, and environmental dictionary atoms. Visual dictionary atoms are 32×32 pixel binary templates, containing the following 5 types of templates, with at least one template in each type: Spark stripe template: composed of connected paths with a width of 1 pixel, a path length between 12 and 28 pixels, and no more than 2 turns; Smoke plume template: composed of multiple semi-transparent, similarly shaped patches, which, after binarization, appear as 3 to 7 irregular connected regions, with the largest connected region area between 60 and 120 pixels; Insulator damage boundary template: composed of zigzag high-contrast boundaries, with 3 to 6 zigzag segments; Arc bright spot template: composed of solid circular spots with a radius of 3 to 5 pixels, with 1 to 3 circular spots; Conductor swaying contour template: composed of approximately horizontal or approximately vertical slender connected regions, with an aspect ratio of not less than 8. The acoustic dictionary atoms are bit templates with a duration of 20 milliseconds, containing the following four types of templates, with one template for each type: Discharge pulse template: composed of 6 consecutive segments of bit 1, each segment with lengths of 16, 8, 16, 8, 16, 8, separated by bits 0, with corresponding interval lengths of 8, 8, 8, 8, 8; Metal impact template: composed of one consecutive segment of bit 1 with a length of 64, followed by three segments of bit 1 with lengths of 32, 16, and 8 respectively, each separated by a bit 0 of length 16; Mechanical loosening template: composed of 8 segments of bit 1, each segment separated by a bit 0 of length 8; Wind-induced vibration template: composed of alternating bits 1 and bits 0, with a total length of 160, and the maximum length of consecutive identical bits not exceeding 4. The environmental dictionary atoms are discrete event templates, containing the following four types of templates: Temperature transition template; (k) Access control opening template; (l) Gas detection trigger template; (m) Wind speed transition template. Each template corresponds one-to-one with the same event.

[0029] From the set of perceptual data blocks, select any continuous time window and extract the corresponding tile sequence, frame-by-frame bit sequence, and discrete event sequence, which are used as visual observation blocks, acoustic observation blocks, and environmental observation blocks, respectively. Establish a residual mask of the same dimension for each of the three types of observation blocks. The residual mask is initialized in an uninterpreted state. The reason for using a residual mask is to record the area covered by the template without changing the original data, facilitating multiple overlay matching on the same data and avoiding duplicate counting.

[0030] For each patch in the visual observation block, each visual dictionary atom is sequentially translated and aligned pixel-by-pixel within the patch at four rotation angles: 0°, 90°, 180°, and 270°. At each alignment position, a pixel consistency count is calculated, defined as the number of bright pixels in the patch where the template foreground of that visual dictionary atom falls. Four fixed angles are used instead of consecutive angles to cover the most common directional changes in power grid components, while keeping the computational load within a stable range, ensuring matching can still be completed even with limited computing power on-site. If the pixel consistency count at a certain alignment position is at least 10 pixels higher than the currently explained portion, then that visual dictionary atom and that alignment position are selected. The patch pixels corresponding to the template foreground are marked as explained on the visual residual mask. Using 10 pixels as the minimum improvement effectively eliminates accidental consistency caused by random noise and fine textures, while prioritizing the explanation of true structural bright spots, stripes, and boundaries. The above process is repeated for the same patch until an improvement of at least 10 pixels cannot be achieved, or the explained area reaches 90% of the patch pixels. Setting a 90% upper limit prevents template overlays from completely covering all tiles, thus preserving uninterpreted areas with identification value for subsequent steps. When two or more alignment positions have the same pixel count, the alignment position with the larger uninterpreted area is prioritized. If the coverage areas are also the same, the templates are selected according to a fixed priority order: arc spot template, spark stripe template, insulator damage boundary template, conductor sway profile template, and smoke plume template. This order is derived from the priority of safety risks in the power grid field, interpreting features with more direct impact on personal and equipment safety first, which is beneficial for quickly forming stable interpreted areas in scenarios with limited computing power.

[0031] For each audio frame in the acoustic observation block, each acoustic dictionary atom is aligned point-by-point within that frame according to the starting sample, and a bit consistency count is calculated. The bit consistency count is defined as the number of sample points where the acoustic dictionary atom at that alignment position matches the bit sequence of the audio frame. Using bit consistency count instead of amplitude error eliminates amplitude inconsistencies caused by differences in pickup gain and sound source distance, ensuring comparability of the same acoustic pattern across different sites. If the bit consistency count at a given alignment position is at least 50 sample points higher than the currently explained portion, that acoustic dictionary atom and alignment position are selected, and the corresponding sample segment is marked as explained on the acoustic residual mask. The lower limit of 50 sample points ensures sufficient continuity for the selected structure to distinguish it from short bursts of ambient noise. This process is repeated for the same frame until an increase of at least 50 sample points is no longer achievable, or the explained segment covers 90% of the frame length. The upper limit of 90% preserves some unexplained segments, facilitating the identification of unexplained conflicts caused by anomalous impulses or long-term noise accumulations in subsequent steps. When two or more alignment positions have the same bit count and overlapping coverage, the alignment position with the earlier starting sample position is preferred; if the starting positions are the same, the alignment position with a longer coverage area is selected. This rule keeps the matching results left-aligned on the time axis, facilitating pairing with visual observation blocks within a fixed time window.

[0032] For each event record in the environmental observation block, if the event name is the same as an environmental dictionary atom and its time interval does not overlap with the interpreted segment, then that time interval is marked as interpreted on the environmental residual mask. There is a one-to-one correspondence between environmental dictionary atoms and event records; therefore, using name matching directly conveys state changes and avoids incorporating differences in continuous quantity thresholds into subsequent judgments. The rule prohibiting overlapping time intervals prevents multiple templates from repeatedly interpreting the same event.

[0033] In the matching process of the visual observation block, horizontal and vertical mirror versions are added to the four rotation angles for matching. Mirror matching can cover the appearance differences of symmetrical structures while maintaining the consistency of 32×32 pixels and binary templates. In the matching process of the acoustic observation block, a 5-millisecond protection zone is reserved at the beginning and end of each audio frame. The protection zone does not participate in the first matching. The protection zone can reduce the impact of frame segmentation boundaries on bit consistency counts and reduce the count drop caused by missegmentation of cross-frame events. In the matching process of the environmental observation block, event records with the same name and adjacent time are merged. When the interval between two records does not exceed 200 milliseconds, they are considered as continuous segments of the same event and merged into a single record before matching with the same name. Merging can reduce duplicate interpretations caused by short-term jitter and improve the temporal continuity of interpreted segments. In the decision conflict handling of the visual observation block, the priority order is adjusted to be consistent with the site safety management requirements. If the risk priority of a site for spark stripes is higher than that for arc spot, the spark stripe template can be placed first in the priority order. This adjustment does not change the way matching and mask updating are performed; it only changes the deterministic choice when there are equal counts.

[0034] Step 3: Within a fixed time window of 100 milliseconds, unmarked regions are considered unexplained conflicts, and a candidate explanation set is generated based on the simultaneous occurrence relationship of selected dictionary atoms. The adversarial game engine is started, and the action of the incident explanation party is to cover unexplained conflicts, while the action of the normal explanation party is to mark perturbations that are allowed according to the compliance rule checklist. In each round, the dominance strategy is eliminated first, and then both parties simultaneously select actions and score: if the number of unexplained conflicts decreases, the incident explanation party gets 1 point, and if the number of unexplained conflicts marked as compliant increases, the normal explanation party gets 1 point. The game ends when both parties take the same action for 2 consecutive rounds or the above two quantities remain unchanged for 2 consecutive rounds.

[0035] In the specific implementation, the temporal position of the sensing data block set is divided into non-overlapping time windows starting from the earliest millisecond-level timestamp, with a step size of 100 milliseconds. Each time window uses an interval method of closing the left endpoint and opening the right endpoint to ensure that all records fall within only one time window. Choosing 100 milliseconds as the fixed time window length can simultaneously cover the co-occurrence of strong pulses of electrical discharge and changes in image brightness, while avoiding mixing adjacent independent events in the same time window, reducing ambiguity of candidate interpretation entries. To eliminate the slight offset between video and audio, the time offset registered in step 1 is used for one-time alignment, making the boundaries of the three types of sources consistent in each time window. Within each time window, the uninterpreted portions of the visual residual mask, acoustic residual mask, and environmental residual mask are read, and an uninterpreted conflict set is constructed according to the source type.

[0036] The composition of visual unexplained conflicts: On each tile within the time window, 4-neighborhood connectivity aggregation is performed on unexplained pixels to obtain pixel clusters; each pixel cluster is considered a separate visual unexplained conflict, accompanied by source type, temporal location, spatial location, and a set of pixel indices for the cluster. 4-neighborhood connectivity is used because adjacent pixels share the same boundary, which stably represents continuous spatial unexplained regions and avoids spurious merging caused by diagonal contact only. The composition of acoustic unexplained conflicts: On each frame of audio within the time window, unexplained samples are merged into consecutive unexplained sample segments in chronological order; if the last and first samples of two adjacent frames are temporally consecutive and both unexplained, the two segments are merged into a single acoustic unexplained conflict, accompanied by source type, temporal location, spatial location, and sample index range. Consecutive sample segments are used to maintain the temporal continuity of acoustic energy, facilitating subsequent coverage of the entire segment with a single candidate explanation entry. The composition of environmental unexplained conflicts: Unexplained event records intersecting the window interval within the time window are treated as environmental unexplained conflicts, accompanied by source type, temporal location, spatial location, and event name. Environmental events are already discrete records with start and end boundaries. Treating them directly as unexplained conflicts avoids boundary inconsistencies caused by further splitting. All unexplained conflicts are addressable objects at the time of generation, and subsequent operations overwrite or mark them with object identifiers to avoid redundant processing.

[0037] Within the same time window, the alignment positions of the visual dictionary atoms, acoustic dictionary atoms, and environmental dictionary atoms selected in step 2 are read. Candidate interpretation entries are generated based on their simultaneous occurrence relationship, and links are established with unexplained conflicts. Formation of candidate interpretation entries: At least one visual dictionary atom, at least one acoustic dictionary atom, or at least one environmental dictionary atom appearing within the same time window are aggregated into candidate interpretation entries according to their temporal overlap relationship. Each candidate interpretation entry records the list of dictionary atoms constituting the entry, as well as their temporal and spatial positions. The rationale for using simultaneous occurrence relationships is that multiple sources of power grid accidents often appear together within a very short time; binding them into a single candidate interpretation entry can improve the efficiency of covering unexplained conflicts. Linking candidate interpretation entries with unexplained conflicts: The visual linking rule is: if the foreground coverage area of ​​a visual dictionary atom in a candidate interpretation entry overlaps with a pixel cluster of a visually unexplained conflict by at least one pixel or has a 4-neighbor contact, a link is established. The acoustic linking rule is: if the alignment position of an acoustic dictionary atom in a candidate interpretation entry intersects with a sample segment of an acoustically unexplained conflict in time, a link is established. The environment linking rule is as follows: if the time interval of an environment dictionary atom in a candidate interpretation entry intersects with the time interval of an uninterpreted conflict in a certain environment, a link is established. Each candidate interpretation entry must link to at least one uninterpreted conflict; if this condition is not met, the entry is discarded. By employing the criteria of "overlap or contact," "temporal intersection," and "temporal convergence," the linking relationship can be determined by the objective relationship between geometry and temporal sequence without relying on weights, ensuring consistency across different implementations.

[0038] Within each time window, the adversarial game engine operates in rounds, aiming to cover or mark unexplained conflicts for compliance within a finite number of steps, and output stable results with explicit scoring and termination rules. The action set of the incident interpreter consists of all candidate explanation entries from the candidate explanation set. Each action selects several candidate explanation entries to cover unexplained conflicts. To facilitate deterministic implementation, a greedy selection method is used in each round: first, the candidate explanation entry that covers the most unexplained conflicts is selected; if there are ties, the candidate explanation entry containing the visual link with a larger coverage area, the acoustic link with a longer coverage length, or the environmental link with a longer coverage time span is selected; if unexplained conflicts still exist after this entry is executed, the second candidate explanation entry that adds the most new coverage is selected. A maximum of two candidate explanation entries are selected per round to facilitate stable convergence under limited on-site computing power. The action set of the normal interpreter consists of rule entries from the compliance rule checklist. The compliance rule checklist must include at least the following three rules: R1 Wind-induced vibration compliance mode, where a wind speed transition template and a conductor sway profile template exist within the same time window, while an arc flash template and a discharge pulse template are absent; R2 Maintenance door opening compliance mode, where an access control opening template exists within the same time window, while a temperature transition template and a gas detection trigger template are absent; R3 Non-electromechanical impact compliance mode, where a metal impact template exists within the same time window, while an arc flash template and a discharge pulse template are absent. In each round, the normal interpreter selects one or two rules that can mark the most unexplained conflicts as compliant, and adds a compliance mark to the unexplained conflicts that meet the corresponding conditions. Selecting one or two rules ensures coverage while controlling the complexity of the judgment, facilitating consistent results across all sites.

[0039] Before the start of each round, dominance policy elimination is performed on the action sets of both parties. On the incident interpreter's side, if a candidate interpreter's set of unexplained conflicts is completely contained within another candidate interpreter's set, the contained candidate interpreter's set is deleted. On the normal interpreter's side, if a rule entry can be marked as compliant within a set of unexplained conflicts completely contained within another rule entry, the contained rule entry is deleted. Dominance policy elimination reduces the search space without losing any actions that might lead to better coverage or better compliance marking. After dominance policy elimination, both parties simultaneously select and execute their respective actions. The execution order adopts a parallel update model: Step 3.1 Record the number of unexplained conflicts at the start of the round and the number of compliant markings at the start of the round on the current unexplained conflict set. Step 3.2 Apply the candidate interpreter's selected action to the unexplained conflict set; all linked unexplained conflicts are marked as covered and removed from the unexplained conflict set. Step 3.3 Apply the rule entry selected by the normal interpreter to the remaining unexplained conflicts; all unexplained conflicts that meet the corresponding rule conditions are marked as compliant, but are not removed from the unexplained conflict set. The reason for not removing them is to distinguish between the two types of results, "eliminated through explanation" and "explained through compliance mode," during scoring, so that subsequent judgments can clearly distinguish between incidents and permitted disturbances. Step 3.4 Record the number of unexplained conflicts at the end of the round and the number of compliant markers at the end of the round.

[0040] If the number of unexplained conflicts at the end of a round is less than at the start of the round, the party explaining the incident receives 1 point; if the number of compliant flags at the end of a round is more than at the start of the round, the party providing the correct explanation receives 1 point. Scoring is based solely on integer scores, without weighting. The actions chosen by both parties in this round, their scores, and state changes are recorded in the round log for this time window for review and tracking. If both parties choose the exact same actions for two consecutive rounds, or if the number of unexplained conflicts and compliant flags remains unchanged for two consecutive rounds, the adversarial game in this time window terminates. After termination, three sets are output: the covered set, the compliant flag set, and the remaining unexplained conflict set. These three sets, along with the round log, are used in subsequent steps. The advantage of using the above termination conditions is that it allows for rapid detection of stable states, avoiding continued calculations without any gains.

[0041] The time window uses a sliding mechanism with a window length of 100 milliseconds and a sliding step size of 50 milliseconds. This sliding mechanism enhances the ability to capture events across window boundaries. During subsequent output, results from overlapping windows are deduplicated, prioritizing the window result that first reaches the termination condition. Candidate interpretation entries are deduplicated during generation. When two candidate interpretation entries contain the same dictionary atomic list and are linked to the same group of unexplained conflicts, the candidate interpretation entry with the earlier time position is retained. Deduplication prevents multiple selections of the same fact in adversarial games. The compliance rule checklist allows for order adjustments across different sites. If a site prioritizes the maintenance-open compliance mode over the wind-induced vibration compliance mode, the maintenance-open compliance mode is prioritized when normal interpretation options are selected concurrently. Order adjustments do not change the rule content, only the deterministic selection during concurrent selection, ensuring consistency with site security management priorities. Small pixel clusters of visually unexplained conflicts at tile boundaries are first filtered out using a pixel count threshold of 5 pixels. This threshold effectively removes noise caused by edge alignment errors without affecting meaningful unexplained areas. The original index mapping is preserved before and after filtering for auditing purposes.

[0042] Step 4: Based on the candidate explanation set, generate the accident judgment result, severity level and impact scope according to the preset rules, form structured accident elements and output them to the notification terminal; the notification terminal synthesizes broadcast text based on the structured accident elements and broadcasts it.

[0043] Specifically, after step 3, each time window has a covered set, a compliant marked set, and a remaining unexplained conflict set, along with a round log and a candidate explanation set. To facilitate the generation of a unified conclusion, all time windows are first organized chronologically to form a timeline with timestamps increasing in milliseconds. Adjacent time windows of the same type are merged into consecutive segments during subsequent mapping to avoid splitting the same incident into multiple fragmented conclusions. Within each time window, a type determination is made based on the candidate explanation set and the remaining unexplained conflict set; subsequently, consecutive time windows are merged to obtain the final incident determination result.

[0044] The rules for determining the type of a single time window are as follows: Discharge Accident: If both the arc flash template and the discharge pulse template exist in the candidate explanation set, and the remaining unexplained conflict set for this time window is not empty, it is determined to be a discharge accident. Thermal Runaway Accident: If both the smoke plume template and the temperature transition template exist in the candidate explanation set, and the remaining unexplained conflict set for this time window is not empty, it is determined to be a thermal runaway accident. Mechanical Loosening Accident: If both the insulator damage boundary template and the mechanical loosening template exist in the candidate explanation set, and the remaining unexplained conflict set for this time window is not empty, it is determined to be a mechanical loosening accident. Normal Operation: If the remaining unexplained conflict set for this time window is empty, it is determined to be normal operation. If only the compliant marked set is not empty, the corresponding compliance explanation is recorded in the notes. Using the condition of "templates exist simultaneously and the remaining unexplained conflict set is not empty" allows us to assume "there are indeed abnormal signs that have not yet been fully explained," thereby reducing the probability of misjudging permissible disturbances as accidents.

[0045] Merging and final selection across time windows: Judgments of the same type within adjacent time windows are merged into consecutive segments, recording the start and end times. If multiple accident types appear consecutively within the timeline, the accident type with the most time windows is selected as the final accident judgment result; if the number is the same, the accident type with the earliest start time is selected. Using the "most time windows" selection method allows for prioritizing persistent anomalies, facilitating the dispatcher's focus on handling accidents with a wider impact.

[0046] The severity level is generated based on discrete rules of the remaining unexplained conflict set, with the maximum value at the continuous segment level being the final output. Within a single time window, a high severity level is determined if any of the following conditions are met: the number of pixels in any visually unexplained conflict is greater than or equal to 20; the length of any acoustically unexplained conflict covers 50% of the frame length; or any environmentally unexplained conflict spans two or more consecutive records. If none of the above conditions are met and the remaining unexplained conflict set for the time window is not empty, a medium severity level is determined. If the remaining unexplained conflict set for the time window is empty, no level is generated. For continuous segments, if any time window is determined to be high severity, the severity level of that segment is high severity; otherwise, if a time window is determined to be medium severity, the severity level of that segment is medium severity. This grading method allows for the differentiation between anomalies with longer durations but lower intensity and short-term strong anomalies, ensuring that the response level matches the actual risk.

[0047] The impact range is generated based on discrete rules of spatial location and temporal continuity, and summarized at the continuous segment level. Within a single time window, the impact range is multi-point if any of the following occurs: Remaining unexplained conflicts involve two or more adjacent tiles; Remaining unexplained conflicts involve two or more consecutive audio frames; Remaining unexplained conflicts involve two or more consecutive environmental records. If the above conditions are not met and remaining unexplained conflicts exist, the impact range is single-point. For continuous segments, if any time window within it is determined to be multi-point, the impact range of that segment is multi-point; otherwise, it is single-point. Using adjacent tiles, consecutive audio frames, and consecutive environmental records as criteria allows for the direct representation of spatial diffusion and temporal duration of impact in the range, avoiding implementation differences caused by introducing proportional weights. After obtaining the final accident determination result, severity level, and impact range, structured accident elements are generated. Structured accident elements must contain at least the following fields: Accident determination result; Severity level; Impact range; Start and end time positions, in milliseconds. The primary spatial location is recorded as 2D or 3D coordinates within the station; if location is not possible, the device number or data collection point number is recorded. The triggering basis includes a list of template names used for judgment and their time and location. Compliance descriptions, if a set of compliant tags exists, list the matching rule entry numbers and their corresponding time and location. Audit information includes the number of time windows, the number of merged segments, and the round log pointer. The reason for retaining "triggering basis" and "audit information" in the structured incident elements is to enable the dispatching side to quickly verify the source, directly locate the corresponding image block, audio frame, and event record, and reduce secondary investigation time.

[0048] After receiving structured incident elements, the notification terminal generates and broadcasts a text according to a fixed procedure. **Dispatch Text:** The dispatch text begins with the incident assessment result, followed by the severity level and impact range, then the start and end times / locations, and finally lists the main spatial location and the first three template names from the triggering criteria. This order allows dispatchers to first obtain the conclusion and level, then the scope and location information, aligning with the reading habits for rapid decision-making. **Operation and Maintenance Text:** The operation and maintenance text begins with the main spatial location, followed by the incident assessment result and severity level, then suggestions for on-site response and safety reminders. Safety reminders are derived from the template names in the triggering criteria, such as prompting for power outage and voltage testing procedures when a discharge pulse template is present. This arrangement allows on-site personnel to obtain work location information and personal protective equipment points in the shortest possible time. **Communication Text:** The communication text begins with the incident assessment result and impact range, followed by the estimated processing time interval and information synchronization frequency, and then includes the start and end times / locations. This text is used for external communication and reporting, emphasizing scope and time information to facilitate business impact assessment.

[0049] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result according to substantially the same method falls within the scope of the present invention. Therefore, the scope of the present invention is defined only by the appended claims.

Claims

1. A method for automatic accident analysis and voice notification driven by a bionic robot for power grid monitoring, characterized in that, The method includes: Step 1: The bionic robot terminal for power grid monitoring inspects the target site, collecting video data, audio data, and environmental status data. After standardizing the three types of data, the time location, spatial location, and source type are labeled according to unified fields to generate a set of perception data blocks. Specifically, when standardizing audio data, the audio data is divided into audio frames with a frame length of 20 milliseconds and a step size of 10 milliseconds. The time location is a millisecond-level timestamp. The spatial location is a two-dimensional or three-dimensional coordinate within the station. If the location cannot be determined, the device number or the collection point number is recorded. The source type is marked as one of video / audio / environment. Step 2: Load the solidified dictionary atomic library, establish visual observation blocks, acoustic observation blocks and environmental observation blocks and corresponding residual masks respectively, and execute the sparse coding reconstruction process to obtain the interpreted region through template matching and mask updating; Step 3: Within a fixed time window of 100 milliseconds, unlabeled regions are classified as unexplained conflicts, and candidate interpretation sets are generated based on the simultaneous occurrence relationships of selected dictionary atoms. Specifically, among the three types of residual masks, regions still in an unexplained state are classified as unexplained conflict sets: visual regions use 4-neighborhood connectivity to generate pixel clusters, acoustic regions use continuous unexplained sample segments, and environmental regions use unexplained event records. When constructing acoustic unexplained conflicts, unexplained samples are merged into continuous unexplained sample segments in chronological order on each audio frame within the time window. Each unexplained conflict is accompanied by its source type, temporal location, and spatial location. Using the 100-millisecond time window as a boundary, selected visual dictionary atoms, acoustic dictionary atoms, and environmental dictionary atoms within the same window are grouped into candidate interpretation entries. Each candidate interpretation entry is linked to at least one unexplained conflict. The adversarial game engine is activated, with the action of the incident interpreter covering unexplained conflicts and the action of the normal interpreter marking perturbations as permissible according to a compliance rule checklist. The execution process of the adversarial game engine includes: setting up two participants, the incident interpreter and the normal interpreter; the incident interpreter's action is to select several items from the candidate interpretation set to maximize the number of linked unexplained conflicts; the normal interpreter's action is to select several items from the compliance rule checklist to maximize the number of unexplained conflicts marked as compliant; at the beginning of each round, dominance strategy elimination is performed on both parties' action sets: if an action does not increase one's own score or decrease the opponent's score in all comparable cases, then the action is deleted; after dominance strategy elimination, both parties simultaneously select and execute actions; if the total number of unexplained conflicts at the end of the round is less than at the beginning of the round, the incident interpreter scores 1 point; if the number of unexplained conflicts marked as compliant at the end of the round is greater than at the beginning of the round, the normal interpreter scores 1 point; the round terminates when the same pair of actions is repeated synchronously for two consecutive rounds, or when the number of unexplained conflicts and the number of unexplained conflicts marked as compliant remain unchanged for two consecutive rounds. Step 4: Based on the candidate explanation set, generate the accident judgment result, severity level and impact scope according to the preset rules, form structured accident elements and output them to the notification terminal; the notification terminal synthesizes broadcast text based on the structured accident elements and broadcasts it.

2. The automatic accident analysis and voice notification method driven by a bionic robot for power grid monitoring as described in claim 1, wherein step 1, the standardization process for video data, audio data, and environmental status data includes: The video data is divided into equal-sized blocks with sides of 32 pixels per frame, resulting in a block sequence. Each block is converted to grayscale values, ranging from 0 to 255. Grayscale values ​​greater than or equal to 200 are considered bright pixels, and those less than 200 are considered dark pixels. The audio data is divided into blocks with a frame length of 20 milliseconds and a step size of 10 milliseconds. Each sample point is recorded as a bit 1 if the amplitude is zero or positive, and as a bit 0 if negative, resulting in a frame-by-frame bit sequence. The environmental state is recorded as a discrete event sequence in chronological order, with event names selected from the following set: temperature transition, access control opening, gas detection trigger, and wind speed transition. Each record includes the event name, start time, and end time.

3. The automatic accident analysis and voice notification method driven by a bionic robot for power grid monitoring as described in claim 2, wherein in step 2, the fixed dictionary atomic library includes: Visual dictionary atoms, acoustic dictionary atoms, and environmental dictionary atoms; Visual The dictionary atoms are binary templates with a size of 32×32 pixels; The acoustic dictionary atoms are bit templates with a duration of 20 milliseconds; the environment dictionary atoms are discrete event templates.

4. The automatic accident analysis and voice notification method driven by a bionic robot for power grid monitoring as described in claim 3, in step 2, a tile sequence, a frame-by-frame bit sequence, and a discrete event sequence are extracted within any continuous time window, denoted as a visual observation block, an acoustic observation block, and an environmental observation block, respectively; a residual mask of the same dimension is established for each of the three types of observation blocks, initially in an uninterpreted state; for the visual observation block, a sparse coding reconstruction process is executed, including: For each patch, each visual dictionary atom is translated and aligned within the patch at four rotation angles: 0 degrees, 90 degrees, 180 degrees, and 270 degrees. The pixel consistency count is calculated, which is the number of bright pixels in the patch where the template foreground falls. If the pixel count of a certain visual dictionary atom is improved by no less than 10 pixels compared to the currently explained portion, then select the visual dictionary atom and the alignment position, and mark the tile pixels corresponding to the template foreground as explained on the visual residual mask; repeat this process for the same tile until the above improvement cannot be achieved or the explained area reaches 90% of the tile pixels.

5. The automatic accident analysis and voice notification method driven by a bionic robot for power grid monitoring as described in claim 4, in step 2, a sparse coding reconstruction process is performed for the acoustic observation block, including: For each audio frame, align each acoustic dictionary atom point by point within the frame according to the starting sample and calculate the bit consistency count; If the bit consistency count of a certain acoustic dictionary atom is improved by at least 50 sample points compared to the currently explained portion, then select the acoustic dictionary atom and the alignment position, and mark the corresponding sample segment as explained on the acoustic residual mask; repeat this process for the same frame until the above improvement can no longer be achieved or the explained segment covers 90% of the frame length; For the environmental observation block, a sparse coding reconstruction process is performed, including: for each event record in the environmental observation block, if its event name is the same as an atom in a certain environmental dictionary and its time interval does not overlap with the explained segment, then the segment is marked as explained on the environmental residual mask; repeat for the same sequence until the explained segment covers 90% of the time span of the sequence or there are no unexplained records with the same name.

6. In the automatic accident analysis and voice notification method driven by a bionic robot for power grid monitoring as described in claim 1, in step 4, if the unexplained conflict set is empty, output a normal operation judgment; if the unexplained conflict set is not empty, output the accident type according to preset rules.

7. The automatic accident analysis and voice notification method driven by a bionic robot for power grid monitoring as described in claim 6, wherein step 4, the process of generating a severity level, includes: If the number of pixels in any visually unexplained conflict is greater than or equal to 20, or the length of an acoustically unexplained conflict on any audio frame covers 50% of the length of that audio frame, or any environmentally unexplained conflict spans two or more consecutive records, it is classified as high-level; if none of the above conditions are met and the set of unexplained conflicts is not empty, it is classified as medium-level; the process of generating the scope of influence includes: if the unexplained conflict involves two or more adjacent tiles, or involves two or more consecutive audio frames, or involves two or more consecutive environmental records, the scope is multi-point; otherwise, it is single-point.

Images