Electric power field operation safety monitoring method and system based on AI driving

Abstract

The invention discloses an electric power field operation safety monitoring method and system based on AI driving. The method comprises the following steps: synchronously acquiring voice, operation video and equipment state identification information, and de-noising to generate a multi-modal sequence; using a multi-modal cross reconstruction model to reconstruct other modals for any modal, and identifying suspicious fragments according to reconstruction errors; multi-layer consistency matching is executed on the suspicious fragments, and the consistency matching degree and mismatch attribution are calculated through time soft alignment, semantic embedding unification and semantic graph reasoning; dispersing the voice, action and equipment state elements into an event sequence, and matching the event sequence with a preset process causal graph to position an abnormal event; and finally, abnormal nodes and induced nodes are analyzed in combination with a Bayesian algorithm, and whole-process safety monitoring is realized. According to the method, password inconsistency, action out-of-order, object mismatching and equipment response abnormity in electric power operation can be intelligently identified and traced, and the real-time performance and accuracy of operation safety are improved.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring technology, specifically to an AI-driven method and system for monitoring the safety of power field operations. Background Technology

[0002] Power field operations are among the highest-risk aspects of power system operation and maintenance, and their safety directly impacts the stability of the power supply system and the safety of personnel. Current safety monitoring methods primarily rely on manual supervision, video surveillance, and operation ticket verification, lacking the ability to intelligently integrate and automatically analyze multi-source data in real time. Because field operations typically involve multimodal signals including voice commands, actions, and equipment status, traditional monitoring systems struggle to achieve semantic-level behavioral understanding and logical matching, resulting in the inability to promptly identify structural risks such as "instruction errors," "out-of-order actions," and "abnormal equipment responses."

[0003] Furthermore, inconsistent sampling frequencies, time-series drift, and severe noise interference among different modal data render multimodal fusion algorithms insufficiently robust in complex power environments. In recent years, the development of artificial intelligence and graph neural networks has provided new ideas for power safety monitoring; however, existing methods are often limited to single-modal identification or static rule matching, failing to perform cross-modal correlation analysis and process-level anomaly tracing. This paper addresses the characteristics of asynchronous multimodal data, complex process logic, and concealed anomaly types in power field operations. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: an AI-driven method for safety monitoring of power field operations, comprising:

[0006] Simultaneously collect voice, operation video and equipment status recognition information at the work site, and complete the alignment and denoising of sampling time to generate multimodal sequences;

[0007] For any modality sequence, the remaining modalities are reconstructed using a multimodal cross-reconstruction model. The reconstruction error is used for coarse screening, and suspicious segments and their corresponding modal labels are output.

[0008] Perform multi-layer consistency matching on the suspicious segments: by combining temporal soft alignment, semantic embedding unification and semantic graph reasoning of instruction-action-object-step, obtain the consistency matching degree and mismatch attribution of each modality at each alignment time;

[0009] The corresponding voice, action and device status elements are discretized into event sequences. The consistency matching degree and mismatch attribution corresponding to each time are used as event features. The event sequences are matched with a preset process cause-effect graph to locate abnormal events.

[0010] By locating the aforementioned abnormal events, safety monitoring of power field operations can be achieved.

[0011] As a preferred embodiment of the AI-driven power field operation safety monitoring method described in this invention, the multimodal cross-reconstruction model is a multimodal generation network sharing a potential coding space, consisting of a coding layer, a decoding layer, and an antifactual perturbation generation unit;

[0012] The coding layer extracts features from each frame of the speech and operation video and the device status recognition information, and maps them to a unified latent representation space to obtain modality-independent semantic vectors.

[0013] The decoding layer reconstructs each modality based on the shared latent semantic vector, enabling data mutual prediction from any modality to the other modalities;

[0014] The multimodal cross-reconstruction model introduces a counterfactual reconstruction mechanism during training and online inference: during reconstruction, the input features are randomly perturbed and replaced to form a dual reconstruction result of factual samples and counterfactual samples; by performing sensitivity analysis on the reconstruction error difference between factual samples and counterfactual samples, it is determined whether the anomaly is caused by object mismatch, out-of-order action, and noise, thereby improving the model's ability to distinguish structural anomalies; the time window where the error difference is stable is taken as a suspicious segment, and modal labels for the suspicious segments are generated.

[0015] As a preferred embodiment of the AI-driven power field operation safety monitoring method described in this invention, the multi-layer consistency matching includes: calculating the cross-modal alignment cost through the time soft alignment; obtaining the input features of the semantic graph by semantically embedding and unifying the alignment results of the time soft alignment with speech segments, action frames and equipment status; and performing inference and updating based on the graph neural network structure on the semantic graph to output the process adaptability result and conflict information for each alignment moment.

[0016] The total consistency matching degree is obtained by summing the cross-modal alignment cost and process adaptability results. Through iteration, the maximum value of the total consistency matching degree is retained in each iteration until the stopping condition is reached. The matching results of each suspicious segment in each other modality are obtained, and the conflict information corresponding to each time step is output as the mismatch attribution.

[0017] Meanwhile, the process adaptability result at each time step—the cross-modal alignment cost—is used as the consistency matching degree at each alignment time step.

[0018] As a preferred embodiment of the AI-driven power field operation safety monitoring method described in this invention, the time soft alignment is a process of sliding alignment of each suspicious segment with other modal time windows, taking into account the different delays between the data of each modality when an anomaly occurs: a graph-constrained dynamic time planning algorithm is used to extract segments through sliding time windows and determine the alignment path between voice segments, action frame segments and equipment status.

[0019] The graph-constrained dynamic time planning algorithm includes generating a process cause-effect graph based on each step in the operation ticket, forming a valid transition matrix with each operation step as a node and the executable order and object dependencies as directed edges; in the cumulative cost calculation of dynamic time planning, only node pairs marked as reachable by the valid matrix are allowed to participate in the alignment path search, and time paths that do not conform to the process order or have no edge connections are given a penalty cost.

[0020] During the alignment process, a local rhythm estimation module is introduced into the sliding time window, allowing the time window to advance or lag, and applying time distance penalty weights to the matching points that are advanced or lagging; when the offset direction is the same for n consecutive time steps and the magnitude exceeds the tolerance, the center of the time window is reset, and the time distance penalty weights are cleared to zero.

[0021] The time-distance penalty weight is an additional value of its own weight 1, so that the weight of the cost is greater than 1;

[0022] By weighted summing the penalty cost and penalty weight, the cross-modal alignment cost of each suspicious segment in each other modality is obtained;

[0023] After summing the cross-modal alignment costs of all suspicious segments, the cross-modal alignment cost calculated by the temporal soft alignment is obtained.

[0024] As a preferred embodiment of the AI-driven power field operation safety monitoring method described in this invention, the semantic embedding unification includes vectorizing the segments of the three modes for each group of cross-modal segments with time soft alignment, using them as a three-dimensional feature as input, and performing process adaptability analysis through the semantic graph.

[0025] The semantic graph is based on three modalities and contains three types of nodes. A graph structure is constructed based on each type of node, and edge relationships are built through logical relationships. Furthermore, edge relationships between nodes of different modalities are built based on the process causal graph. When the semantic graph obtains the three-dimensional features, the semantic graph matches multiple nodes in each modality according to the features of each modality's corresponding dimension, obtaining nodes in the three dimensions respectively. These nodes are then propagated through edge relationships to obtain the process adaptability result and the conflict information.

[0026] As a preferred embodiment of the AI-driven power field operation safety monitoring method described in this invention, the process adaptability result includes an adaptability score for each alignment moment, which is used to characterize whether the voice command, action behavior and equipment status conform to the step logic specified in the operation ticket at that moment.

[0027] The status label corresponding to the three-dimensional feature is either normal or conflict; if the label is normal, the conflict information is empty; if the label is conflict, the conflict information is generated.

[0028] The conflict information includes the relationships corresponding to all conflict edges during the propagation process.

[0029] As a preferred embodiment of the AI-driven power field operation safety monitoring method described in this invention, the preset process cause-effect graph is generated by the operation ticket, with nodes representing specific steps; edges represent the legal sequence and object dependencies, and the probability of other nodes being affected when a node is abnormal is analyzed using a Bayesian algorithm.

[0030] Matching the event sequence with a preset process causal graph includes: for each suspicious segment with conflict, matching the corresponding abnormal node in the preset process causal graph; using the consistency matching degree and all mismatch attributions in the segment as features, inputting them into the Bayesian algorithm to obtain the triggering node of each suspicious segment with conflict;

[0031] After summarizing the abnormal nodes and the triggering nodes, the corresponding events are mapped to obtain the abnormal events.

[0032] A power field operation safety monitoring system based on AI using the method described in this invention is characterized by: a data acquisition unit that simultaneously acquires voice, operation video and equipment status recognition information at the operation site, and completes the alignment and denoising of the sampling time to generate a multimodal sequence;

[0033] The processing unit reconstructs the remaining modes for any modality sequence using a multimodal cross-reconstruction model, performs coarse screening based on the reconstruction error, and outputs suspicious segments and their corresponding modality labels.

[0034] The analysis unit performs multi-layer consistency matching on the suspicious segments: by combining temporal soft alignment, semantic embedding unification and semantic graph reasoning of instruction-action-object-step, it obtains the consistency matching degree and mismatch attribution of each modality at each alignment time.

[0035] The positioning unit discretizes the corresponding voice, action, and equipment status elements into event sequences, and uses the consistency matching degree and mismatch attribution corresponding to each time as event features. It then matches the event sequences with a preset process cause-effect graph to locate abnormal events. Through the location of abnormal events, it achieves safety monitoring of power field operations.

[0036] A computer device includes: a memory and a processor; the memory stores a computer program, wherein: when the processor executes the computer program, it implements the steps of the method described in any one of the present invention.

[0037] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the present invention.

[0038] The beneficial effects of this invention are as follows: This invention achieves unified modeling and logical matching of voice, video actions, and equipment status through an AI-driven multimodal fusion monitoring mechanism. By introducing a multimodal cross-reconstruction model, it can automatically identify potential abnormal segments at the reconstruction error level; the time soft alignment algorithm solves the time drift problem between multimodal data, enabling cross-modal events to be analyzed in a unified time domain; the unified semantic embedding and semantic graph reasoning structure enable the system to have the semantic understanding ability of the "instruction-action-object-step" relationship; the process matching method based on Bayesian causal analysis further realizes the source tracing judgment of abnormal nodes and inducing nodes. This method can not only detect potential violations in the early stage of operation, but also generate a traceable evidence chain, providing intelligent basis for accident early warning and responsibility determination, and significantly improving the intelligent monitoring level and safety assurance capability of power field operations. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 The first embodiment of the present invention provides an overall flowchart of an AI-driven power field operation safety monitoring method. Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0042] Reference Figure 1 As an embodiment of the present invention, an AI-driven method for safety monitoring of power field operations is provided, comprising:

[0043] S1: Simultaneously collect voice, operation video and equipment status recognition information at the work site, and complete the alignment and denoising of sampling time to generate a multimodal sequence.

[0044] Multi-source data acquisition terminals are deployed at the power site, including wearable voice acquisition devices, high-definition video acquisition terminals, and equipment status acquisition modules. Voice acquisition devices are used to collect operators' commands and voice communication in real time. Video acquisition terminals are deployed at key viewpoints in the work area to capture operators' postures, movements, and tool usage. Equipment status acquisition modules acquire the opening and closing status, voltage, current, and indicator signals of equipment such as circuit breakers, disconnectors, and grounding switches through communication interfaces or sensors. All acquisition terminals are connected to the central processing unit via wireless or wired networks. Timestamps are synchronized and calibrated by a unified clock source (such as a GPS or NTP server) to ensure that data from different modalities are comparable on the same timeline. The acquired raw signals undergo multi-stage denoising and preprocessing, including: performing spectral subtraction and voice endpoint detection on the voice signal to remove environmental noise and silent segments; applying Gaussian blur and illumination equalization to the video frames to improve motion edge recognition; and performing outlier removal and filtering smoothing on the equipment status data to remove communication jitter and sampling errors. Subsequently, the three modalities are uniformly time-aligned and interpolated to form a multimodal time series with the same sampling step size. Each time point contains a speech feature vector, an action frame feature vector, and a device state vector, thereby generating a multimodal sequence that can be used as input for AI models.

[0045] By synchronously acquiring and aligning data with time, we ensure that voice commands, operational actions, and device responses form a one-to-one correspondence under the same time reference. Through denoising and feature processing, we improve the distinguishability of signals and the quality of model input, providing a stable and clean data foundation for subsequent multimodal cross-reconstruction and consistency matching.

[0046] S2: For any modality sequence, the remaining modalities are reconstructed using a multimodal cross-reconstruction model. The reconstruction error is used for coarse screening, and suspicious segments and their corresponding modal labels are output.

[0047] The described multimodal cross-reconstruction model is a multimodal generation network sharing a latent coding space, consisting of an encoding layer, a decoding layer, and a counterfactual perturbation generation unit. The encoding layer extracts features from each frame of the speech and operation video, along with device status recognition information, and maps them to a unified latent representation space to obtain modality-independent semantic vectors. The decoding layer reconstructs each modality based on the shared latent semantic vectors, enabling mutual prediction of data from any modality to the others. By constructing a multimodal cross-reconstruction model with a shared latent coding space, information from different modalities can mutually predict and constrain each other at the semantic level. When voice commands, operation actions, and device status remain logically consistent, each modality can accurately reconstruct itself from the others; however, when there are out-of-order operations, object mismatches, or command errors, the reconstruction error increases significantly, thus automatically identifying abnormal segments. This design enables the system to possess cross-modal semantic understanding and structural anomaly perception capabilities, providing a high-confidence input foundation for subsequent consistency matching and anomaly localization.

[0048] In implementation, the model can employ a multimodal variational autoencoder or a Transformer-based shared latent space network architecture. The encoding layer extracts speech semantic vectors, video action features, and device state embeddings and maps them to a unified semantic space. The decoding layer reconstructs the remaining modalities based on this latent vector, achieving multi-directional data mutual prediction. By introducing counterfactual perturbation samples into the latent space and comparing the reconstruction errors, anomaly sensitivity detection can be achieved. The model can be deployed on GPU servers or edge computing terminals, trained using existing job data, and possesses real-time operation and online inference capabilities. The technical implementation path is clear, and the engineering feasibility is high.

[0049] The multimodal cross-reconstruction model introduces a counterfactual reconstruction mechanism during training and online inference: during reconstruction, the input features are randomly perturbed and replaced to form a dual reconstruction result of factual samples and counterfactual samples; by performing sensitivity analysis on the reconstruction error difference between factual samples and counterfactual samples, it is determined whether the anomaly is caused by object mismatch, out-of-order action, and noise, thereby improving the model's ability to distinguish structural anomalies; the time window where the error difference is stable is taken as a suspicious segment, and modal labels for the suspicious segments are generated.

[0050] To effectively distinguish between three different types of anomalies in power field operations—"object mismatch," "action out of order," and "noise interference"—an anomaly identification mechanism based on counterfactual reconstruction and sensitivity analysis is proposed. This mechanism first generates corresponding counterfactual samples for different anomaly types to simulate the possible behavior of each modality of data under specific error scenarios. For object mismatch anomalies, the system replaces the object identifiers in voice, video, or equipment status with the spatially closest object among similar equipment or the object actually touched by the operator, reconstructing semantic associations without changing the action type and step number, thus simulating a misoperation scenario. For action out of order anomalies, the system adjusts the execution order of operation steps, partially swapping or time-shifting adjacent or logically related operation steps in the process to detect semantic shifts caused by incorrect execution order. For noise anomalies, typical background noise from the field is superimposed on the voice signal, brightness jitter, blurring, or partial occlusion is introduced into the video frame, and time jitter or a small amount of dropped reports is applied to the equipment status data to verify whether the anomaly is caused by signal degradation. The generation of all counterfactual samples follows physical and procedural feasibility constraints to ensure that they do not violate operational logic or system security boundaries.

[0051] After obtaining factual samples and multiple types of counterfactual samples, the system performs sensitivity analysis to compare the degree of difference and stability of reconstruction errors. First, within a fixed time window, the system calculates the difference between the factual reconstruction error and each type of counterfactual reconstruction error, forming a multimodal error fingerprint. Then, through sliding window aggregation and exponentially weighted smoothing, the direction and amplitude changes of the error difference within multiple consecutive sub-windows are evaluated. If the error difference remains stable and exceeds a threshold, a structural anomaly is identified for that period. The threshold is determined based on the statistical characteristics of historical normal samples and is dynamically adjusted using quantile intervals or robust standard scores to ensure the algorithm's adaptability to different field environments. Next, the system identifies the anomaly type based on the trend of the error difference: if the reconstruction error significantly decreases after object replacement, and the improvement is concentrated between actions and equipment states, it indicates that the current anomaly is mainly caused by object mismatch; if the error significantly decreases after adjusting the operation sequence, it indicates that the anomaly originates from out-of-order actions; if structural disturbances do not improve the results, and noise disturbances cause a significant increase in error variance, it is judged as signal quality degradation rather than a process error.

[0052] S3: Perform multi-layer consistency matching on the suspicious fragments: obtain the consistency matching degree and mismatch attribution of each modality at each alignment time by combining temporal soft alignment, semantic embedding unification and semantic graph reasoning of instruction-action-object-step.

[0053] The multi-layer consistency matching includes: calculating the cross-modal alignment cost through the temporal soft alignment; obtaining the input features of the semantic graph by semantically embedding the alignment result of the temporal soft alignment into speech segments, action frames and device states; and performing inference updates based on a graph neural network structure on the semantic graph to output the process adaptability result and conflict information for each alignment moment.

[0054] The total consistency matching degree is obtained by summing the cross-modal alignment cost and the process adaptability result. Through iteration, the maximum total consistency matching degree is retained in each iteration until the stopping condition is reached. The matching result of each suspicious segment in each other modality is obtained, and the conflict information corresponding to each time step is output as the mismatch attribution. At the same time, the process adaptability result at each time step minus the cross-modal alignment cost is used as the consistency matching degree at each alignment time step.

[0055] Traditional job monitoring methods can only perform static comparisons of a single modality, lacking semantic-level fusion and temporal dynamic modeling, making it difficult to detect hidden structural problems such as misordering, accidental touches, and misjudgments. In contrast, this invention combines time-aligned temporal constraints with semantically embedded content understanding through a "multi-layer consistency matching" mechanism, and then achieves process-level logical verification through semantic graph reasoning, thus elevating the monitoring system from "appearance recognition" to "causal understanding".

[0056] In this process, temporal soft alignment undertakes the dynamic synchronization of multimodal segments in the temporal dimension, ensuring that voice, action, and device signals logically belong to the same operation stage; semantic embedding unification maps information from different modalities to a shared semantic space through feature fusion, enabling the system to understand "who is doing what and which device is being operated on"; subsequently, semantic graph reasoning uses the structured feature propagation mechanism of graph neural networks to compare and propagate the causal relationships of these semantic nodes with the process defined in the operation ticket, outputting the process adaptability result at each moment (i.e., whether the behavior conforms to the operation logic) and conflict information (i.e., which logical edge or object constraint is violated).

[0057] By fusing and summing the cross-modal alignment cost (reflecting temporal differences between modalities) with the process adaptability result (reflecting semantic consistency), this step can calculate the overall consistency matching degree at each time point and dynamically update it during the iteration process to find the globally optimal matching result. Ultimately, the system can not only determine "an anomaly exists here" but also explain "the semantic source and process location of the anomaly".

[0058] The aforementioned soft time alignment considers the different delays between the data of each modality when an anomaly occurs, and performs sliding alignment of each suspicious segment with other modal time windows: a graph-constrained dynamic time planning algorithm is used to extract segments through sliding time windows, and the alignment path is determined between the calculated speech segments, action frame segments, and device status.

[0059] The graph-constrained dynamic time planning algorithm includes generating a process causal graph based on each step in the operation ticket, forming a valid transition matrix with each operation step as a node and the executable order and object dependencies as directed edges; in the cumulative cost calculation of dynamic time planning, only node pairs marked as reachable by the valid matrix are allowed to participate in the alignment path search, and time paths that do not conform to the process order or have no edge connections are assigned a penalty cost.

[0060] During the alignment process, a local rhythm estimation module is introduced into the sliding time window, allowing the time window to advance or lag, and applying time distance penalty weights to the matching points that are advanced or lagging; when the offset direction is the same for n consecutive time steps and the magnitude exceeds the tolerance, the center of the time window is reset, and the time distance penalty weights are cleared to zero.

[0061] The time-distance penalty weight is an additional value to its own weight of 1, making the weight of the cost greater than 1.

[0062] By weighted summing of penalty cost and penalty weight, the cross-modal alignment cost of each suspicious segment in each other modality is obtained.

[0063] After summing the cross-modal alignment costs of all suspicious segments, the cross-modal alignment cost calculated by the temporal soft alignment is obtained.

[0064] It should be noted that due to uncertainties in on-site operations, such as instruction lag, premature actions, and equipment response delays, traditional methods using fixed time windows or simple synchronization strategies are prone to mismatches and false alarms. This invention introduces a soft time alignment mechanism, enabling the system to establish flexible temporal correlations between multimodal signals. This achieves stable cross-modal matching under different operational rhythms and response delays, ensuring the authenticity and continuity of the timing basis for anomaly identification. This mechanism not only considers time offset issues but also introduces process logic constraints, ensuring that the alignment results conform to the operation sequence and object dependencies, fundamentally guaranteeing the legitimacy and interpretability of the alignment.

[0065] In terms of implementation, the graph-constrained dynamic time planning algorithm proposed in this invention binds alignment calculations to the logical order specified in the operation ticket by incorporating process causal graph constraints into the time path search. Traditional dynamic time planning algorithms are based solely on minimizing time distance, while this method requires that paths can only transfer between logically reachable nodes, meaning the operation order must conform to the actual work process. If there is no process association between a voice command and an action segment or device state, the algorithm automatically assigns a high penalty cost, excluding it from the optimal path search. Simultaneously, the algorithm introduces a local rhythm estimation module, dynamically calculating the allowable advance or lag range based on historical action duration and response delay. When multiple consecutive time steps are detected to have the same offset direction and amplitude exceeding the tolerance, the algorithm automatically resets the time window center to adapt to changes in operator rhythm, thereby ensuring that the alignment result remains stable and reasonable under real-world working conditions. This algorithm can be implemented based on an improved dynamic time planning framework, and the calculation of the alignment cost matrix can be performed in real time on a GPU or edge computing module. The process constraint layer and rhythm estimation layer can be operated in parallel to ensure that the alignment result is output within millisecond-level time steps. The system can adapt to different field environments during operation, such as slow-paced substation operations or intensive live-line maintenance work, thus balancing accuracy and real-time performance. The feasibility of this design has been verified in deep learning tasks such as multimodal monitoring and semantic matching, demonstrating good deployability.

[0066] Furthermore, the process of generating a causal graph based on the operation ticket is a crucial step in implementing logical constraints in this algorithm. The system first parses the operation ticket text or electronic instructions, extracting the number, action type, operation object, preconditions, and expected result of each step, defining each step as a node in the graph. Then, directed edges are generated based on the dependencies between steps: when the result of one operation satisfies the preconditions of another operation, a directed edge is established from the former to the latter; if multiple steps act on the same device and the order must be fixed, a mandatory edge is established; for operations that can be executed in parallel, parallel branches are formed. The system further generates a valid transition matrix using node numbers, allowing only logically reachable node pairs to participate in the time path search. This matrix becomes the core constraint template of the algorithm, ensuring that the dynamic programming path search only occurs within the allowed scope of the process.

[0067] During system operation, the process cause-effect graph is dynamically updated based on real-time status. When an operation is detected to be completed or a device status change, the system automatically updates the node status, adjusts the executable path, and prevents the algorithm from matching expired or invalid processes. In this way, time alignment not only reflects the temporal consistency between signals but also provides a deep understanding of the job logic and operational rhythm. Ultimately, this design achieves cross-modal adaptive time synchronization, enabling the AI ​​system to accurately determine the logical consistency between voice, actions, and device feedback, providing a stable and reliable timing foundation for subsequent anomaly identification and process reasoning.

[0068] The unified semantic embedding involves vectorizing the segments of the three modalities for each set of cross-modal segments with temporal soft alignment, using them as a three-dimensional input feature, and then analyzing the process adaptability through the semantic graph. The key to unified semantic embedding is to vectorize speech, action, and device state separately, and then map them to a unified latent space through a feature encoder, giving them a comparable and reasonable embedding structure. Speech segments extract keywords and instruction relationships through a semantic vector model; action frames obtain high-dimensional representations of operational actions through human pose estimation and behavior recognition networks; and device states are transformed into semantic signals of operational responses through a state change encoder. The system pairs these three sets of features temporally to form an input unit, namely a three-dimensional feature set (speech vector, action vector, and device vector), and uses it as a node input to the semantic graph. In this semantic graph, nodes of different modalities are connected by logical relationship edges. During the propagation and update process, the graph neural network calculates the semantic adaptability of the three-dimensional feature set with the standard process based on the causal logic and historical matching rules defined in the operation ticket, thereby outputting the logical consistency result at that moment. Semantic embedding can be implemented based on existing multimodal representation learning frameworks, such as BERT-like speech encoders, 3D convolutional action recognition networks, and time-series state encoding networks. Through feature normalization and spatial alignment, features from different modalities achieve semantic consistency. The semantic graph can be constructed based on the causal graph structure generated from the operation ticket, embedding each three-dimensional feature as a node. Graph neural networks (such as GCN, GAT, or GraphSAGE) are used to propagate semantic relationships between nodes, enabling cross-modal logical reasoning. This process can run in batch mode on edge computing terminals or in the cloud, allowing for both real-time updates and adaptation to different operating environments. The overall solution enables the AI ​​monitoring system to understand the operation process at the semantic level, thereby determining whether the match between the current voice, action, and device status is reasonable, providing a higher-level semantic foundation for anomaly detection and mismatch attribution.

[0069] The semantic graph is based on three modalities and contains three types of nodes. A graph structure is constructed based on each type of node, and edge relationships are built through logical relationships. Furthermore, edge relationships between nodes of different modalities are established based on the process causal graph. When the semantic graph acquires the three-dimensional features, it matches multiple nodes in each modality according to the features of each modality's corresponding dimension, obtaining nodes in each of the three dimensions. These nodes are then propagated through edge relationships to obtain the process adaptability result and the conflict information. Each type of node generates initial features using its respective encoder: the speech side uses a text / acoustic encoder to output semantic vectors, the action side uses a temporal visual encoder to output behavior vectors, and the device side uses a state sequence encoder to output state vectors. All nodes carry a time index, object identifier, and confidence level to facilitate gating and alignment during subsequent propagation.

[0070] Within a single modality, a separate graph structure is constructed and edges are defined for each type of node: firstly, temporal edges connect adjacent or near-neighbor segments within the same stream, ensuring continuous semantic propagation over time; secondly, semantically similar edges connect synonymous / near-synonymous or template-similar nodes to enhance robustness under noisy conditions; and thirdly, object-binding edges connect highly related segments within the same object or interval, forming object-dimensional "clusters." These internal edges provide channels for "self-verification" and "self-correction" for each type of node; for example, action continuity helps smooth jittery frames, and speech synonyms resist differences in accent and sentence breaks.

[0071] Between modalities, cross-modal edges are constructed based on the causal graph generated from the operation ticket: the voice→action edge reflects the relationship of "command triggering action", the action→device edge reflects the relationship of "action causing device response", and the voice→device edge reflects the "consistency confirmation between command and device response". These edges not only carry the allowed sequence of events and object dependencies, but also label the edges with legality tags (allowed, prohibited, conditionally allowed), as well as temporal compatibility from temporal soft alignment and semantic similarity from the embedding space, which are used as attention weights and gating factors in subsequent propagation. In this way, when the network transmits information, it will prioritize propagation along "legal and highly matched" edges, naturally suppressing non-compliant paths.

[0072] Furthermore, "the semantic graph matches multiple nodes in each modality based on the features of each modality's corresponding dimension." To address the requirement of "matching multiple nodes in each modality," the semantic graph performs Top-K matching on each time-aligned unit during the node selection phase: for the voice dimension, it selects the K voice nodes most relevant to the current command; for the action dimension, it selects the K action nodes closest to the current posture / trajectory; and for the device dimension, it selects the K device nodes most consistent with the current object / state. This results in a set of nodes across three dimensions at each time step. To avoid mispropagation due to mismatches, the Top-K threshold and the K value are adaptively set according to the scenario (e.g., a stricter threshold is used for live-line work), and placeholder nodes are automatically inserted and their edge weights reduced for low-confidence nodes to ensure that missing modalities do not skew the graph.

[0073] The graph neural network employs a heterogeneous attention mechanism for propagation and updates: an independent attention head is assigned to each "node type—edge type—node type" ternary relation, and message weights are calculated comprehensively based on "legitimacy label, semantic similarity, temporal compatibility, and node confidence." Illegal or weakly legitimate paths are automatically weighted lower. Propagation proceeds in one to several rounds in an "internal first, cross-modal last" order to first stabilize each modality itself before fusing cross-modal evidence. Regularization is performed after each round to prevent a single strong signal from monopolizing propagation. During propagation, statistics on "suppressed illegal edges" and "amplified legitimate edges" are continuously recorded to provide a basis for subsequent conflict interpretation.

[0074] It's also worth mentioning that the process adaptability results are summarized and output at two levels: at the node level, each candidate node's process adaptability score is given (whether it matches the current step, object, and rhythm); at the time level, the adaptability of nodes across the three dimensions at that time is summarized into a process adaptability score for that time. Conflict information is generated at both edges and nodes: edge conflict labels are out of order, object mismatch, password mismatch, state non-response, illegal transfer, etc.; node conflict labels are missing, low confidence, or inconsistent with adjacent steps. The network also outputs evidence chain fragments that triggered the conflict (time index of relevant nodes and edges, object ID, suppressed / amplified propagation path), facilitating traceability.

[0075] In terms of training and deployment, the network first performs self-supervised and comparative learning using historical "normal process" data to learn the distribution of "high weight for legitimate edges and low weight for illegitimate edges." Then, it undergoes supervised fine-tuning using labeled anomalous samples or synthetic counterfactual samples to enhance its sensitivity in identifying out-of-order, mismatched, and unresponsive behavior. Online inference employs a sliding window incremental update: new nodes are added, old nodes are eliminated, edge weights are quickly recalculated, and the propagation rounds and thresholds adapt to different scenarios and loads to meet real-time requirements. Through this construction, the semantic graph network organically couples temporal alignment, semantic understanding, and process constraints within the same propagation framework. This reduces false alarms caused by single-modal noise and provides clear, auditable conflict localization and causal explanations when structural violations occur.

[0076] The process adaptability results include an adaptability score for each alignment moment, used to characterize whether the voice command, action behavior, and device status at that moment conform to the step logic specified in the operation ticket. It also includes a status label corresponding to the three-dimensional features: normal or conflict; if the label is normal, the conflict information is empty; if the label is conflict, the conflict information is generated. The conflict information includes the relationships corresponding to all conflict edges during the propagation process.

[0077] Traditional AI identification methods often only provide anomaly probabilities or overall mismatch conclusions, lacking causal explanations for the "source of the anomaly" and failing to support the tracing of safety risks and the determination of responsibility at the power field. This method introduces "process adaptability scores" and "conflict edge recording mechanisms" into graph neural network inference, enabling the system to possess causal-level logical diagnostic capabilities, explaining "why this place is determined to be abnormal" and "how the anomaly propagates in the semantic process."

[0078] The process adaptability results are calculated by aggregating the semantic consistency of voice nodes, action nodes, and device nodes at each moment, and then calculating a comprehensive score for that moment to measure whether it conforms to the logical path defined in the operation ticket. A high score indicates that the three-modal information at that moment is consistent with the standard process; a significant drop in score indicates a potential conflict. Based on this, the system automatically labels the three-dimensional features with a status: "normal" or "conflict". When the label is "conflict", the model not only outputs an anomaly judgment but also generates corresponding conflict information to explain the type of anomaly and its propagation relationship.

[0079] The term "relationships corresponding to all conflicting edges during propagation" refers to the process in semantic graph reasoning where the model tracks logically inconsistent events on all edges (i.e., information interaction paths between nodes) and records these edges as conflicting edges. Conflicting edges reflect the types of connections that violate process logic during semantic propagation, primarily including the following categories: First, out-of-order edges, indicating that an action or device state executes before its legal predecessor node in time; for example, an action occurs before a voice command, or a device response occurs before a permitted step. Second, mismatched edges, indicating inconsistencies between voice, action, and device state at the object level; for example, a voice command targets device A, while the action and device feedback correspond to device B. Third, missing edges, indicating that required logical dependencies in the flowchart are absent; for example, the required "voltage check → grounding" link is not triggered. Fourth, illegal edges, indicating that the system discovers during propagation that a node has established a relationship with another node that is not permitted by the causal graph of the process; for example, reverse propagation of "closing the switch → voltage check".

[0080] These conflicting edges are explicitly recorded during semantic graph propagation, and their corresponding relational information includes the start and end nodes of the edge, the relation type, the time index, and the triggering modal combination (voice-action, action-device, or voice-device). After reasoning is completed, the system generates conflict information based on the distribution of conflicting edges, so that anomalies not only stop at the conclusion of "there is a problem," but also indicate "which logical link the problem occurred in." This mechanism enables the AI ​​system to have structured explanatory capabilities, providing a traceable chain of evidence for safety auditing and risk warning, thereby significantly improving the intelligence and credibility of power operation safety monitoring.

[0081] It's also important to know that during semantic graph reasoning, each time-aligned unit generates a local graph structure containing voice nodes, action nodes, device nodes, and the logical relationship edges between them. During message propagation, the system simultaneously monitors the performance of two types of edges: non-conflicting edges (legitimate edges), which conform to the logical order, object binding, and execution dependencies of the operation ticket flow causal graph; and conflicting edges (violation edges), which trigger out-of-order, mismatched, or illegal association conditions during propagation. The core idea of ​​the comprehensive score is to normalize the evaluation of the opposing dynamics of these two types of edges. That is, in the same propagation round, the strengthening of legitimate relationships and the weakening of conflicting relationships counterbalance each other, ultimately forming a semantic-level "net consistency measure."

[0082] Each legitimate edge corresponds to a semantically "correct transmission path," representing a reasonable causal chain between voice commands, operational actions, and device responses. During propagation, the system assigns positive weights to the information transmission of these edges and amplifies them based on semantic similarity, temporal compatibility, and the probability of legitimate transitions. Specifically, when voice nodes and action nodes have a high degree of matching in the semantic space, and the response delay from action to device conforms to historical distribution, the weights of these legitimate edges are enhanced, meaning that "the operational logic at that moment is stable and self-consistent." This cumulative enhancement constitutes a positive contribution to the overall score. When an edge in the propagation path violates the causal graph (e.g., "closing the switch → power testing" or "operating device A → feedback to device B"), the system automatically marks it as a conflict edge. For the message propagation of these edges, the network executes a negative feedback mechanism: first, it weakens the propagation weight of the edge so that its information no longer affects downstream nodes; second, it applies penalty weights to adjacent nodes, reducing their local semantic consistency score; and third, it records the relationship type, start and end nodes, and triggering mode of the edge in the conflict information cache. In other words, conflicting edges are not only identified but also generate a "negative energy effect" during propagation, thus lowering the overall consistency of that time window. At each time alignment moment, the system calculates the positive propagation amount (from non-conflicting edges) and negative inhibition amount (from conflicting edges) of all edges in the semantic graph. These two quantities together constitute an "opposing balance system": if non-conflicting edges dominate, it indicates that the current semantic relationship is smooth and logically coherent, and the overall score is close to a high value. If there are many conflicting edges with high weights, it indicates that there are multiple logical breaks or object deviations in semantic propagation, and the overall score is significantly lowered. If the two are similar, the system will output a neutral score based on the stability and temporal continuity of the process context, indicating that the moment is in a suspicious but not completely non-compliant state.

[0083] When calculating the overall score, the system also incorporates time smoothing and modal confidence weighting mechanisms to prevent short-term fluctuations caused by single-frame anomalies. For example, if a legitimate edge continues to dominate within three consecutive time windows, short-term local conflicts will not lead to an overall anomaly; conversely, if a conflicting edge remains stable during multiple rounds of propagation, the system will mark it as a structural anomaly.

[0084] S4: Discretize the corresponding voice, action and device status elements into event sequences, and use the consistency matching degree and mismatch attribution corresponding to each time as event features, so that the event sequences are matched with the preset process cause-effect graph to locate abnormal events.

[0085] Event discretization can be achieved using a rule engine or sequence segmentation network, automatically segmenting time series based on speech pauses, action boundaries, and equipment state change points. The design of discretizing speech, action, and equipment state elements into event sequences aims to transform continuous multimodal monitoring data from "signal-level" to "semantic-level temporal logical units," enabling subsequent process matching and anomaly inference to be performed at the causal level. Voice commands, action execution, and equipment responses in field operations are often continuous dynamic processes; single frames or short segments of data cannot directly reflect the operational logic. Through discretization, speech segments, action frames, and equipment state change points at each moment are integrated into an event unit, namely, "the semantic action occurring at time t." Each event not only contains the operational behavior at that moment but also the state information left over from previous steps, thus forming a complete causal sequence in the time dimension. In this way, the system no longer processes the raw temporal signal but rather the "semantic event chain," providing a clear and structured input for subsequent process causal matching.

[0086] Another important reason for event serialization is to facilitate the integration of semantic graph outputs. The "consistency matching degree" and "mismatch attribution" obtained from semantic graph reasoning reflect the semantic rationality and conflict sources at each point in time. By assigning these features to event units, each event not only represents "what happened," but also carries information on "whether it is reasonable" and "why it is unreasonable." In this way, the event sequence becomes a temporal chain with logical confidence and anomaly explanation capabilities, which can be directly and structurally compared with a pre-defined process causal graph.

[0087] The process cause-effect graph can be automatically generated by structured parsing of operation tickets. The preset process cause-effect graph is generated from the operation ticket, with nodes representing specific steps and edges representing legal sequences and object dependencies. A Bayesian algorithm is used to analyze the probability of other nodes failing when a node fails. The preset process cause-effect graph is generated from the operation ticket to transform human-expert-defined operating procedures into a machine-understandable logical model. Each step in the operation ticket is defined as a node in the graph, and directed edges between nodes represent legal sequences or object dependencies. For example, "opening the circuit breaker" must precede "testing for voltage," and "testing for voltage" must be completed before "grounding." This graph structure not only reflects the execution order of operations but also embodies the logical constraints of electrical safety. To enable the system to have adaptive anomaly reasoning capabilities, this invention further embeds a Bayesian analysis mechanism into the graph: by learning from historical operation data and anomaly samples, the conditional probability of other nodes failing when a certain node fails is calculated, thus forming a process cause-effect model with uncertainty propagation capabilities. This means that when a step is detected as abnormal, the system can infer its possible triggering source based on probability relationships, and determine whether it is caused by the previous step not being executed, parallel action conflict, or device status feedback delay.

[0088] Matching the event sequence with a preset process causal graph includes: for each suspicious segment with a conflict, matching the corresponding abnormal node in the preset process causal graph; using the consistency matching degree and all mismatch attributions in the segment as features, inputting them into the Bayesian algorithm to obtain the triggering node for each suspicious segment with a conflict. After summarizing the abnormal nodes and the triggering nodes, mapping them to the corresponding events, obtaining the abnormal events. During the matching process between the event sequence and the process causal graph, the system searches for the node in the graph that is semantically closest to each suspicious segment with a conflict. The matching is based not only on the operation type and object identifier of the event, but also on its consistency matching degree and mismatch attribution features. The matched node is marked as an "abnormal node," indicating that the current behavior violates the normal logic of that step. Simultaneously, the system inputs the semantic features of the segment into the Bayesian model and calculates its "triggering node" through conditional probability inference, i.e., the potential upstream step that caused the current abnormality. Triggering nodes are usually steps that were not executed, executed incorrectly, or whose execution order was disrupted. Their probabilistic inference allows anomaly localization to go beyond surface errors and reveal the root cause of the problem. Finally, the system aggregates the abnormal nodes and the triggering nodes, and maps them into an abnormal event chain according to their time sequence and logical relationship, which serves as the final output of security monitoring.

[0089] S5: By locating the abnormal events, safety monitoring of power field operations can be achieved.

[0090] The system first identifies and locates abnormal events at the semantic layer. Once the event sequence matches a pre-defined cause-effect graph, each abnormal event is labeled as an "abnormal node—triggering node" pair, indicating the location of the abnormality and its possible root cause. At this point, the system categorizes and processes abnormal events according to their type (e.g., out-of-order operation, object mismatch, missing response, signal degradation, etc.) and confidence level. High-confidence structural anomalies (e.g., "grounding without voltage verification") are immediately identified as high-risk safety events and trigger voice or visual alarms; while low-confidence minor deviations (e.g., action delays or voice overlap) enter an observation state, where the system continues to track whether they escalate into actual violations.

[0091] Subsequently, the system spatially maps the anomaly location results with the on-site digital twin model. Each anomaly event's corresponding voice, action, and equipment status segment carries a timestamp and spatial tag, allowing the system to pinpoint the specific location and object of the anomaly within the digital twin environment. For example, when identifying an object mismatch anomaly where "an operation on cabinet A is accompanied by a command targeting cabinet B," the system highlights the relative positions of cabinets A and B in the 3D twin model, while also annotating the operator's identity and action trajectory, enabling dispatchers and monitoring personnel to intuitively understand the risk scenario.

[0092] During the dynamic monitoring phase, the system executes different response strategies based on the type of abnormal event. For abnormalities involving the risk of misoperation, the system can immediately alert the operator and provide corrective prompts through smart terminals (such as helmet displays, voice headsets, and operating terminal screens). For abnormalities that may affect equipment safety, the system can coordinate with the remote monitoring center or on-site intelligent switch system to execute automated safety actions, such as suspending operations, locking circuits, or cutting off command inputs to prevent further escalation of the accident. Simultaneously, the system continuously updates the risk situation map, combining abnormal events with work plans, personnel locations, and equipment health status to form a multi-dimensional safety situation analysis.

[0093] Furthermore, based on anomaly location, the system establishes a traceable event recording and analysis mechanism. Each anomaly corresponds to a complete chain of evidence, including the original voice command, video clips of actions, equipment status change records, and semantic graph reasoning results. Through event playback and visualization analysis, the system can reproduce the entire process of an anomaly, providing data support for safety audits, accident debriefing, and workflow optimization. Moreover, the system can continuously optimize the AI ​​model using labeled anomaly samples, enabling it to have higher sensitivity and accuracy in subsequent operations, thereby achieving self-learning and closed-loop improvement.

[0094] On the other hand, this embodiment also provides an AI-driven power field operation safety monitoring system, which includes: a data acquisition unit that simultaneously acquires voice, operation video and equipment status recognition information at the work site, and completes the alignment and denoising of sampling time to generate a multimodal sequence.

[0095] The processing unit reconstructs the remaining modes for any modality sequence using a multimodal cross-reconstruction model, performs coarse screening based on the reconstruction error, and outputs suspicious segments and their corresponding modality labels.

[0096] The analysis unit performs multi-layer consistency matching on the suspicious segments: by combining temporal soft alignment, semantic embedding unification and semantic graph reasoning of instruction-action-object-step, it obtains the consistency matching degree and mismatch attribution of each modality at each alignment time.

[0097] The positioning unit discretizes the corresponding voice, action, and equipment status elements into event sequences, and uses the consistency matching degree and mismatch attribution corresponding to each time as event features. It then matches the event sequences with a preset process cause-effect graph to locate abnormal events. Through the location of abnormal events, it achieves safety monitoring of power field operations.

[0098] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0100] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0101] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0102] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for safety monitoring of power field operations based on AI, characterized in that, include: Simultaneously collect voice, operation video and equipment status recognition information at the work site, and complete the alignment and denoising of sampling time to generate multimodal sequences; For any modality sequence, the remaining modalities are reconstructed using a multimodal cross-reconstruction model. The reconstruction error is used for coarse screening, and suspicious segments and their corresponding modal labels are output. Perform multi-layer consistency matching on the suspicious segments: by combining temporal soft alignment, semantic embedding unification and semantic graph reasoning of instruction-action-object-step, obtain the consistency matching degree and mismatch attribution of each modality at each alignment time; The corresponding voice, action and device status elements are discretized into event sequences. The consistency matching degree and mismatch attribution corresponding to each time are used as event features. The event sequences are matched with a preset process cause-effect graph to locate abnormal events. By locating the aforementioned abnormal events, safety monitoring of power field operations can be achieved.

2. The AI-driven power field operation safety monitoring method as described in claim 1, characterized in that: The multimodal cross-reconstruction model is a multimodal generation network sharing a latent coding space, consisting of a coding layer, a decoding layer, and an antifactual perturbation generation unit; The coding layer extracts features from each frame of the speech and operation video and the device status recognition information, and maps them to a unified latent representation space to obtain modality-independent semantic vectors. The decoding layer reconstructs each modality based on the shared latent semantic vector, enabling data mutual prediction from any modality to the other modalities; The multimodal cross-reconstruction model introduces a counterfactual reconstruction mechanism during training and online inference: during reconstruction, the input features are randomly perturbed and replaced to form a dual reconstruction result of factual samples and counterfactual samples; by performing sensitivity analysis on the reconstruction error difference between factual samples and counterfactual samples, it is determined whether the anomaly is caused by object mismatch, out-of-order action, and noise, thereby improving the model's ability to distinguish structural anomalies; the time window where the error difference is stable is taken as a suspicious segment, and modal labels for the suspicious segments are generated.

3. The AI-driven power field operation safety monitoring method as described in claim 2, characterized in that: The multi-layer consistency matching includes calculating the cross-modal alignment cost through the temporal soft alignment; the alignment result of the temporal soft alignment is used to obtain the input features of the semantic graph by semantically embedding and unifying the speech segments, action frames and device states. The semantic graph is updated based on a graph neural network structure, and the process adaptability result and conflict information at each alignment time are output. The total consistency matching degree is obtained by summing the cross-modal alignment cost and process adaptability results. Through iteration, the maximum total consistency matching degree is retained in each iteration until the stopping condition is reached. The matching results of each suspicious segment in each other modality are obtained, and the conflict information corresponding to each time step is output as the mismatch attribution. Meanwhile, the process adaptability result at each time step—the cross-modal alignment cost—is used as the consistency matching degree at each alignment time step.

4. The AI-driven power field operation safety monitoring method as described in claim 3, characterized in that: The aforementioned soft time alignment takes into account the different delays between the data of each modality when an anomaly occurs, and performs sliding alignment of each suspicious segment with other modal time windows: a graph-constrained dynamic time planning algorithm is used to extract segments through sliding time windows, and the alignment path is determined between calculating speech segments, action frame segments and device status; The graph-constrained dynamic time planning algorithm includes generating a process cause-effect graph based on each step in the operation ticket, forming a valid transition matrix with each operation step as a node and the executable order and object dependencies as directed edges. In the cumulative cost calculation of dynamic time planning, only node pairs marked as reachable by the legal matrix are allowed to participate in the alignment path search, and time paths that do not conform to the process order or have no edge connections are given a penalty cost. During the alignment process, a local rhythm estimation module is introduced into the sliding time window, allowing the time window to be advanced or delayed, and applying time distance penalty weights to the advanced and delayed matching points. When the offset direction is the same for n consecutive time steps and the magnitude exceeds the tolerance, the time window center is reset to make the time interval penalty weight zero. The time-distance penalty weight is an additional value of its own weight 1, so that the weight of the cost is greater than 1; By weighted summing the penalty cost and penalty weight, the cross-modal alignment cost of each suspicious segment in each other modality is obtained; After summing the cross-modal alignment costs of all suspicious segments, the cross-modal alignment cost calculated by the temporal soft alignment is obtained.

5. The AI-driven power field operation safety monitoring method as described in claim 4, characterized in that: The unified semantic embedding includes vectorizing the segments of the three modalities for each set of cross-modal segments in the temporal soft alignment, using them as a three-dimensional feature as input, and performing process adaptability analysis through the semantic graph. The semantic graph is based on three modalities and contains three types of nodes. A graph structure is constructed based on each type of node, and edge relationships are built through logical relationships. Furthermore, edge relationships between nodes of different modalities are built based on the process causal graph. When the semantic graph obtains the three-dimensional features, the semantic graph matches multiple nodes in each modality according to the features of each modality's corresponding dimension, obtaining nodes in the three dimensions respectively. These nodes are then propagated through edge relationships to obtain the process adaptability result and the conflict information.

6. The AI-driven power field operation safety monitoring method as described in claim 5, characterized in that: The process adaptability results include an adaptability score for each alignment moment, which is used to characterize whether the voice command, action behavior, and device status at that moment conform to the step logic specified in the operation ticket. And the status labels corresponding to the three-dimensional features: normal or conflict; If the label is normal, then the conflict information is empty; If the tag is conflicting, then the conflict information is generated; The conflict information includes the relationships corresponding to all conflict edges during the propagation process.

7. The AI-driven power field operation safety monitoring method as described in claim 6, characterized in that: The preset process cause-effect graph is generated from the operation ticket, with nodes representing specific steps and edges representing the legal sequence and object dependencies. The probability of other nodes being affected when a node is abnormal is analyzed using a Bayesian algorithm. Matching the event sequence with a preset process causal graph includes: for each suspicious segment with conflict, matching the corresponding abnormal node in the preset process causal graph; using the consistency matching degree and all mismatch attributions in the segment as features, inputting them into the Bayesian algorithm to obtain the triggering node of each suspicious segment with conflict; After summarizing the abnormal nodes and the triggering nodes, the corresponding events are mapped to obtain the abnormal events.

8. A power field operation safety monitoring system based on AI, employing the method described in any one of claims 1-7, characterized in that: The acquisition unit simultaneously acquires voice, operation video, and equipment status recognition information at the work site, and completes the alignment and denoising of sampling time to generate multimodal sequences; The processing unit reconstructs the remaining modes for any modality sequence using a multimodal cross-reconstruction model, performs coarse screening based on the reconstruction error, and outputs suspicious segments and their corresponding modality labels. The analysis unit performs multi-layer consistency matching on the suspicious segments: by combining temporal soft alignment, semantic embedding unification and semantic graph reasoning of instruction-action-object-step, it obtains the consistency matching degree and mismatch attribution of each modality at each alignment time. The positioning unit discretizes the corresponding voice, action, and equipment status elements into event sequences, and uses the consistency matching degree and mismatch attribution corresponding to each time as event features. It then matches the event sequences with a preset process cause-effect graph to locate abnormal events. Through the location of abnormal events, it achieves safety monitoring of power field operations.

9. A computer device, comprising: A memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images