A power supply risk early warning method, device and system based on multi-modal data fusion

Abstract

The application provides a power supply risk early warning method, device and system based on multi-modal data fusion, which comprises the following steps: collecting a visual frame sequence, an audio signal and environmental sensor data around a power supply device; inputting the visual frame sequence into a visual feature extraction network of a lightweight MobileViT architecture to obtain a visual feature vector; inputting the audio signal into an acoustic feature extraction network based on a convolutional neural network after short-time Fourier transform to obtain an acoustic feature vector; inputting the environmental sensor data into an environmental feature extraction network composed of a full connection layer to obtain an environmental feature vector; calculating the weight of each feature respectively, and generating a fusion feature vector through a weighted sum method; inputting a continuous preset number of fusion feature vectors into a bidirectional time sequence network to obtain a time sequence context representation; obtaining a risk probability score through a Sigmoid function; and generating an alarm signal when the exponential weighted average value continuously exceeds a preset threshold and the duration exceeds a preset time length.

Description

Technical Field

[0001] This application relates to the field of power grid monitoring and early warning technology, and in particular to a power supply risk early warning method, alarm device, and power supply risk early warning system based on multimodal data fusion. Background Technology

[0002] In existing power supply monitoring systems, most rely on single visual or sensor data for anomaly detection, lacking collaborative analysis of video, acoustic, and environmental signals. Single-modal analysis often leads to false alarms and missed alarms due to changes in lighting, noise interference, or localized faults. Furthermore, various alarm thresholds, time windows, and level settings are scattered across different systems, making unified management and dynamic adjustment difficult. Simultaneously, current early warning systems are largely based on empirical rules, failing to quantitatively assess and dynamically simulate potential risks under complex operating conditions, thus failing to meet the demands of variable power grid operations and high-time emergency response.

[0003] With the development of IoT and edge computing technologies, power grid monitoring terminals have acquired multimodal data acquisition capabilities, but lack a deep fusion and unified modeling mechanism for multi-source information such as vision, sound, infrared, and status variables. Traditional systems have failed to fully utilize the advantages of large-scale model inference, remaining at the stage of static alarms and manual threshold adjustments, lacking proactive early warning capabilities based on real-time simulation analysis. To improve the accuracy of early warnings and operational efficiency, it is urgent to build an intelligent risk early warning framework that can cover multimodal data and support dynamic gating and time-series aggregation. Summary of the Invention

[0004] In view of this, it is necessary to provide a power supply risk early warning method, device and system based on multimodal data fusion, which can at least overcome one of the above defects.

[0005] In a first aspect, embodiments of this application provide a power supply risk early warning method based on multimodal data fusion, applied to monitoring power supply equipment, the method comprising: Visual frame sequences, audio signals, and environmental sensor data around the power supply equipment are synchronously acquired using a precise time protocol. The visual frame sequence is input into the visual feature extraction network of the lightweight MobileViT architecture to obtain visual feature vectors; After performing a short-time Fourier transform on the audio signal, it is input into an acoustic feature extraction network based on a convolutional neural network to obtain acoustic feature vectors. The environmental sensor data is input into an environmental feature extraction network composed of fully connected layers to obtain environmental feature vectors. The weights of the visual feature vector, the acoustic feature vector, and the environmental feature vector are calculated using a gating attention mechanism, and a fused feature vector is generated by weighted summation. A predetermined number of fused feature vectors are input into a bidirectional temporal network to obtain a temporal context representation, which includes historical information and future prediction information. The temporal context representation is fully connected and mapped, and the risk probability score is obtained through the Sigmoid function; The risk probability score is calculated as an exponentially weighted average, and an alarm signal is generated when the exponentially weighted average continuously exceeds a preset threshold for a duration exceeding a preset time.

[0006] In one embodiment, inputting the visual frame sequence into a visual feature extraction network of a lightweight MobileViT architecture to obtain visual feature vectors includes: Each frame in the visual frame sequence is normalized and Gaussian denoising is performed to obtain a preprocessed frame set. The normalization formula is as follows:

[0007] in, These are the normalized pixel values. For pixel values, For the average pixel value of the training set, The standard deviation of the training set pixels; The formula for Gaussian denoising is:

[0008] in, These are the denoised video frames. For the size of a Gaussian kernel, Gaussian operation; The preprocessed frame set is input into the visual feature extraction network of the lightweight MobileViT architecture to obtain visual features after global average pooling. The calculation formula for the visual features is as follows:

[0009] in, Let k be the visual feature vector of the k-th time window. A lightweight visual feature extractor; For global average pooling, is the dimension of the visual feature vector.

[0010] In one embodiment, the step of performing a short-time Fourier transform on the audio signal and then inputting it into an acoustic feature extraction network based on a convolutional neural network to obtain an acoustic feature vector includes: Calculate the short-time Fourier transform for a continuous audio segment to obtain a time-spectrum graph. The formula for the short-time Fourier transform is:

[0011] in, This represents the time-spectrum graph of the k-th time window. For frequency index, For time frame indexing, For the sample index within the window, The short-time Fourier transform window length. For time The original audio sample value at that location, For frame shift, ; The time-frequency spectrum is input into a convolutional neural network (CNN) consisting of multiple convolutional and pooling layers. The CNN extracts the time-frequency features of the audio through local receptive fields and weight sharing mechanisms, generating a fixed-dimensional acoustic feature representation. The output formula of the CNN is:

[0012] in, For the first The acoustic feature vector of the window, for The feature map after convolution and pooling operations, To flatten the feature map into a one-dimensional vector, This is the weight matrix of the fully connected layer. For the bias vector of the fully connected layer, is the dimension of the acoustic feature vector.

[0013] In one embodiment, the step of inputting the environmental sensor data into an environmental feature extraction network composed of fully connected layers to obtain an environmental feature vector includes: The environmental parameter series is processed by interpolation and moving average for each channel. The formula for calculating the moving average is as follows:

[0014] in, This represents the smoothed value of the j-th environmental parameter channel at time point t. Represents the original measurement value. For smoothing coefficients; Calculate the average value of the smoothed environment parameters within the window, input the average value into a fully connected transform layer, and generate an environment feature vector through a nonlinear activation function. The expression of the nonlinear activation function is as follows:

[0015] in, Let be the environmental feature vector for the k-th time window. It is the ReLU activation function. This is the weight matrix, with dimensions r×d (d is the number of environmental parameter channels). Let r be the bias vector, and r be the dimension of the environmental feature vector.

[0016] In one embodiment, the step of calculating the weights of the visual feature vector, the acoustic feature vector, and the environmental feature vector using a gated attention mechanism, and generating a fused feature vector through a weighted summation, includes: The visual feature vector, the acoustic feature vector, and the environmental feature vector are concatenated into a fusion vector. The gating weights for each modality are calculated using a gating attention mechanism, and the formula is as follows:

[0017] in, Let be the gated weight vector for mode m in the k-th time window, representing the reliability weight of this mode under the current environmental conditions. The weight matrix is ​​a learnable matrix with dimension . , This represents the total dimension of the concatenated vectors. , For bias vectors, This represents the Sigmoid activation function. Represents the fusion vector. Represent each mode; A fused feature vector is generated by weighted summation, and the formula for calculating the weighted summation is as follows:

[0018] in, To fuse feature vectors, the dimension is , This indicates element-wise multiplication. Let m be the eigenvector of mode m. express 3D real space.

[0019] In one embodiment, the step of inputting a consecutive preset number of fused feature vectors into a bidirectional temporal network to obtain a temporal context representation includes: The forward hidden state and backward hidden state are calculated based on the bidirectional temporal network. The formula for calculating the forward hidden state is as follows:

[0020] in, This represents the forward hidden state at time t. The computation function representing the feedforward LSTM network; The fused feature vector at time t, This is the forward hidden state at time t-1; The formula for calculating the backward hidden state is:

[0021] in, This represents the backward hidden state at time t. This represents the computation function of the feedforward LSTM network. This represents the backward hidden state at time t+1; The forward hidden state and the backward hidden state are merged into the temporal context representation.

[0022] In one embodiment, the method further includes: Acquire multiple alarm signals; Obtain the alarm time, alarm type, and alarm area of ​​multiple alarm signals; If the alarm type and alarm area are the same, and the difference between the alarm times is within a preset range, then the multiple alarm signals will be merged into one alarm signal.

[0023] In one embodiment, the method further includes: A risk evolution trend is generated based on the exponentially weighted average of the risk probability scores; An alarm signal is generated when the risk evolution trend shows a continuous upward trend for more than a preset time.

[0024] Secondly, embodiments of this application provide an alarm device, comprising: Processor; and A memory having computer-readable instructions stored thereon for controlling the processor to execute the power supply risk warning method based on multimodal data fusion as described in the first aspect.

[0025] Thirdly, embodiments of this application provide a power supply risk early warning system based on multimodal data fusion, applied to the power supply risk early warning method based on multimodal data fusion as described in the first aspect, the system comprising: The data acquisition module is used to synchronously collect visual frame sequences, audio signals, and environmental sensor data around the power supply equipment via a precision time protocol. The feature extraction module is used to input the visual frame sequence into a lightweight MobileViT architecture visual feature extraction network to obtain visual feature vectors; it is also used to perform a short-time Fourier transform on the audio signal and input it into an acoustic feature extraction network based on a convolutional neural network to obtain acoustic feature vectors; it is also used to input the environmental sensor data into an environmental feature extraction network composed of fully connected layers to obtain environmental feature vectors. The feature fusion module is used to calculate the weights of the visual feature vector, the acoustic feature vector, and the environmental feature vector respectively through a gated attention mechanism, and generate a fused feature vector by weighted summation. A bidirectional temporal network is used to accept a predetermined number of fused feature vectors to obtain a temporal context representation, which includes historical information and future prediction information. The risk probability score acquisition module is used to perform a fully connected mapping on the temporal context representation and obtain the risk probability score through the Sigmoid function. An alarm module is used to calculate the exponentially weighted average of the risk probability score based on the risk probability score, and to generate an alarm signal when the exponentially weighted average continuously exceeds a preset threshold and the duration exceeds a preset time.

[0026] This application provides a power supply risk early warning method, device, and system based on multimodal data fusion. Through multimodal data fusion, visual, acoustic, and environmental sensor information are analyzed collaboratively, significantly improving the accuracy of risk identification and reducing false alarms and missed alarms caused by single sensor failure or interference. Real-time time-series aggregation and gating mechanisms effectively smooth short-term fluctuations, ensuring greater stability and reliability of alarms. Lightweight edge-side inference enables a closed loop of "local early warning + cloud calibration," meeting the requirements for low-latency, high-concurrency real-time early warning while dynamically and adaptively adjusting thresholds and model parameters, improving the overall system's robustness and maintainability. An integrated intelligent decision engine utilizes reinforcement learning and graph neural networks to automatically generate scheduling and emergency response plans, assisting maintenance personnel in quickly locating faults and optimizing resource allocation, significantly shortening fault response and handling time, and achieving lean and intelligent power supply services. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a power supply risk early warning method based on multimodal data fusion provided in an embodiment of this application.

[0028] Figure 2 A schematic diagram of a power supply risk early warning system module based on multimodal data fusion, provided for another embodiment of this application.

[0029] Figure 3 An alarm device provided in one embodiment of this application.

[0030] Explanation of main component symbols Power supply risk early warning system based on multimodal data fusion 10 Data Acquisition Module 11 Feature extraction module 12 Feature fusion module 13 Bidirectional temporal network 14 Risk Probability Score Acquisition Module 15 Alarm module 16 Alarm device 20 Processor 21 Memory 22 Method steps S100-S800 Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0032] It should be noted that, in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0033] It should be noted that in the embodiments of this application, the terms "first," "second," etc., are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. Features specified as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0034] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In existing power supply monitoring systems, most rely on single visual or sensor data for anomaly detection, lacking collaborative analysis of video, acoustic, and environmental signals. Single-modal analysis often leads to false alarms and missed alarms due to changes in lighting, noise interference, or localized faults. Furthermore, various alarm thresholds, time windows, and level settings are scattered across different systems, making unified management and dynamic adjustment difficult. Simultaneously, current early warning systems are largely based on empirical rules, failing to quantitatively assess and dynamically simulate potential risks under complex operating conditions, thus failing to meet the demands of variable power grid operations and high-time emergency response.

[0036] With the development of IoT and edge computing technologies, power grid monitoring terminals have acquired multimodal data acquisition capabilities, but lack a deep fusion and unified modeling mechanism for multi-source information such as vision, sound, infrared, and status variables. Traditional systems have failed to fully utilize the advantages of large-scale model inference, remaining at the stage of static alarms and manual threshold adjustments, lacking proactive early warning capabilities based on real-time simulation analysis. To improve the accuracy of early warnings and operational efficiency, it is urgent to build an intelligent risk early warning framework that can cover multimodal data and support dynamic gating and time-series aggregation.

[0037] This application provides a power supply risk early warning method, device, and system based on multimodal data fusion. Through multimodal data fusion, visual, acoustic, and environmental sensor information are analyzed collaboratively, significantly improving the accuracy of risk identification and reducing false alarms and missed alarms caused by single sensor failure or interference. Real-time time-series aggregation and gating mechanisms effectively smooth short-term fluctuations, ensuring greater stability and reliability of alarms. Secondly, lightweight edge-side inference enables a closed loop of "local early warning + cloud calibration," meeting the requirements for low-latency, high-concurrency real-time early warning while dynamically and adaptively adjusting thresholds and model parameters, improving the overall system's robustness and maintainability. Thirdly, the system integrates an intelligent decision engine, utilizing reinforcement learning and graph neural networks to automatically generate scheduling and emergency response plans, assisting maintenance personnel in quickly locating faults and optimizing resource allocation, significantly shortening fault response and handling time, and achieving lean and intelligent power supply services. Finally, the solution is highly modular and scalable, supports hierarchical visual alarm management and batch configuration, and can be smoothly integrated into third-party GIS, SCADA and other systems. It has good promotional value and benchmark demonstration effect, and provides an innovative solution for the industry's "smart power supply" management.

[0038] Figure 1 This is a schematic flowchart of a power supply risk early warning method based on multimodal data fusion provided in an embodiment of this application. Figure 1The power supply risk early warning method based on multimodal data fusion shown includes at least the following steps: S100: Synchronously collect visual frame sequences, audio signals, and environmental sensor data around the power supply equipment using a precise time protocol; S200: Input the visual frame sequences into a lightweight MobileViT architecture visual feature extraction network to obtain visual feature vectors; S300: Perform a short-time Fourier transform on the audio signals and input them into a convolutional neural network-based acoustic feature extraction network to obtain acoustic feature vectors; S400: Input environmental sensor data into an environmental feature extraction network composed of fully connected layers to obtain environmental feature vectors; S500: Parallel... The gating attention mechanism calculates the weights of the visual feature vector, acoustic feature vector, and environmental feature vector respectively, and generates a fused feature vector through a weighted summation method; S600: Input a preset number of fused feature vectors into a bidirectional temporal network to obtain a temporal context representation, which includes historical information and future prediction information; S700: Perform a fully connected mapping on the temporal context representation and obtain a risk probability score through a sigmoid function; S800: Calculate the exponentially weighted average of the risk probability scores based on the risk probability scores, and generate an alarm signal when the exponentially weighted average continuously exceeds a preset threshold and the duration exceeds a preset duration.

[0039] S100: Synchronously acquire visual frame sequences, audio signals, and environmental sensor data around the power supply equipment via a precision time protocol.

[0040] Specifically, PTP clients are deployed at each acquisition node (including cameras, microphones, and environmental sensors) and a bidirectional time synchronization relationship is established with the PTP master clock source located at the edge gateway or local server. Upon startup, the PTP client continuously adjusts its local hardware clock through exchange steps such as delayed request-delayed response and synchronization message-tracking message, ensuring that the nanosecond-level clock deviation of each node eventually converges to within ±1ms. During acquisition, visual frames are continuously output at 30fps, with each frame timestamped with a PTP timestamp; audio sampling is batched at 16kHz with 16-bit quantization depth, with each 1024 sampling points corresponding to a timestamp; environmental sensors (including temperature, humidity, wind speed, air pressure, and PMU phasors) sample at 50Hz or higher frequencies, and the timestamp is synchronized immediately after each reading. All data sources package the timestamped raw data into a unified buffer. The edge inference engine extracts subsets of visual, acoustic, and environmental data from the buffer according to fixed time windows (e.g., 1 second or 8 frames), aligns them by timestamp, and sends them to the subsequent processing pipeline. This process is protected by both hardware timestamps and software buffer queues, ensuring that even in the event of network congestion or node restarts, the synchronization state can be quickly restored without affecting subsequent multimodal fusion.

[0041] Understandably, this step, through high-precision PTP time synchronization and multi-level buffering design, achieves a strict correspondence between visual, acoustic, and environmental data in the time dimension. This ensures that multi-source information at the same moment can be simultaneously incorporated into the analysis, avoiding the timing misalignment and information loss problems caused by traditional asynchronous sampling. In power supply risk early warning scenarios, even a deviation of only a few milliseconds between the audio signal and the image frame can lead to incorrect event correlation or missed alarms. Through this step, the system can accurately capture the co-occurrence of abnormal power grid equipment (such as discharge sounds and arc flashes), sudden environmental changes (such as a sudden increase in wind speed), and potential abnormal targets in the visual image, providing a solid time baseline for subsequent multimodal fusion, thereby significantly improving the timeliness and reliability of risk perception.

[0042] S200: Input the visual frame sequence into the visual feature extraction network of the lightweight MobileViT architecture to obtain visual feature vectors.

[0043] Specifically, each preprocessed image frame is first linearly normalized according to the pixel mean and standard deviation statistically derived from the training set, mapping pixel values ​​to the [0,1] interval. Then, a 3×3 Gaussian convolution kernel is applied for edge-preserving denoising, smoothing out small-particle noise while preserving edge details as much as possible. The processed image is divided into 16×16 non-overlapping patches and mapped to fixed-dimensional PatchEmbed features through a lightweight convolutional layer (containing a combination of Depthwise and Pointwise convolutions). Next, each PatchEmbed feature block undergoes a self-attention transformation within its local window. This transformation first calculates the Query, Key, and Value matrices within each local window, then performs weighted aggregation to capture the long-range dependencies between object boundaries and texture information within the same window. After several layers of Transformer Blocks that cyclically embed local attention mechanisms and cross-window information transfer, the network outputs multi-scale feature maps at different depths. Finally, global average pooling (GAP) and batch normalization are applied to all scale features to obtain a p-dimensional fixed-length visual feature vector for a single frame. The entire network has been pruned and quantized to optimize model size and parameter count, so that inference can be completed with only a small number of floating-point operations, and the processing time per frame on edge computing devices can be controlled within 20ms.

[0044] Understandably, this visual feature extraction network balances the efficient extraction of local textures by convolutional operations with the effective modeling of global semantics by Transformer. Through hierarchical multi-scale self-attention and lightweight design, it can identify high-frequency anomalies such as surface cracks, foreign object obstruction, or flash arcs in power grid equipment, while maintaining high anti-interference performance in complex backgrounds. Furthermore, the model's pruning and quantization measures ensure high frame rate processing capabilities even on edge nodes with limited computing resources, enabling near real-time visualization and monitoring of power grid operation status, and providing reliable and rich visual representations for subsequent multimodal fusion.

[0045] S300: After performing a short-time Fourier transform on the audio signal, it is input into an acoustic feature extraction network based on a convolutional neural network to obtain an acoustic feature vector.

[0046] Specifically, the system extracts a window of continuous audio signal in 64ms segments and generates static time-domain frames using a 50% frame-shift overlap. A 1024-point Fast Fourier Transform (FFT) is performed on each frame to obtain a complex spectrum. The amplitude spectrum is then calculated and its logarithm is taken to generate a time-frequency two-dimensional matrix of size F×N. Subsequently, a 40-Melbourne filter bank is applied to the amplitude spectrum matrix, and a logarithmic Mel spectrum is obtained through matrix multiplication and logarithmic operations to highlight the sensitivity of the human ear and device structure to different frequency bands. This Mel spectrum is input to a lightweight CNN network consisting of three layers of convolutional layers, batch normalization, ReLU, and max pooling, stacked alternately. Each convolutional kernel has a 3×3 kernel size, and the number of channels is halved with each layer. The pooling stride is 2 to continuously aggregate frequency and time domain features. The Flatten operation at the end of the network flattens the convolutional output, which is then mapped to a q-dimensional acoustic feature vector through a 256-dimensional fully connected layer, and finally normalized using LayerNorm. During the training phase, pruning and 8-bit quantization are performed on the weights of convolutional kernels and fully connected layers to keep the network inference latency to around 10ms while ensuring feature representation capabilities.

[0047] Understandably, this acoustic feature extraction network, by fully mining frequency and time domain information, can accurately identify high-frequency harmonic noise, switch tripping sounds, and mechanical vibrations generated by power grid equipment during operation. Simultaneously, Mel filtering and logarithmic transformation approximate the characteristics of human hearing, giving the network a stronger ability to perceive sudden abnormal sounds. Combined with pruning and quantization optimization, the network can respond efficiently during edge inference, providing accurate acoustic cues to the multimodal fusion module, supplementing blind spots in visual and environmental data, and thus constructing a more comprehensive risk profile.

[0048] S400: Input the environmental sensor data into an environmental feature extraction network composed of fully connected layers to obtain an environmental feature vector.

[0049] Specifically, raw time-series data from channels such as temperature, humidity, wind speed, air pressure, and synchronization phasor (PMU) are first read at a uniform sampling frequency (e.g., 50Hz), and occasional missing values ​​for each channel are imputed using a linear interpolation algorithm. To suppress noise interference from transient changes, a 5-point moving average filter is applied to smooth the data for each channel. The processed sequence is then used to calculate the mean and variance for each channel within each sliding window (length N), ultimately outputting an environmental feature vector. Before model deployment, this is done through... Norm pruning and quantization-aware training sparsify and quantize the parameters of the fully connected network to 8 bits, enabling the subnetwork to complete inference within milliseconds.

[0050] Understandably, this environmental feature extraction network, through statistical feature compression and nonlinear mapping of multi-channel sensor data, can not only reflect the impact of environmental conditions (such as high temperature and humidity, strong winds, or abnormal voltage and current) on equipment operation, but also uncover the deep correlation between environmental changes and potential equipment failures. For example, a sudden increase in wind speed may cause insulator vibration and trigger discharge. The features extracted by this network can complement visual and acoustic signals, forming a multi-dimensional perception of risk events and achieving more comprehensive and accurate power supply risk warnings.

[0051] S500: The weights of the visual feature vector, the acoustic feature vector, and the environmental feature vector are calculated respectively through a gating attention mechanism, and a fused feature vector is generated by weighted summation.

[0052] Specifically, the multimodal feature vectors are first concatenated into a merged vector. Then, attention scores are generated for each modality through a corresponding linear transformation and activation function. This process, through an end-to-end learnable gating matrix, enables the network to dynamically evaluate the reliability of each modality based on the real-time input scenario and accordingly amplify or suppress the corresponding feature dimensions, achieving optimal weighted fusion of multi-source information.

[0053] Understandably, the gating attention mechanism plays a dynamic balancing role in multimodal fusion. When the signal-to-noise ratio of a certain modality is low or missing, its corresponding attention weight automatically decreases, thereby reducing the impact of that modality's noise on the final fused features. Conversely, when a certain modality exhibits highly reliable risk precursors (such as visual sparks, acoustic arcing, or a sudden increase in wind speed), the network automatically increases its weight, enhancing its responsiveness to that modality. Furthermore, through fine-grained channel-level weighting using element-wise multiplication, the network can distinguish the importance of signals in different dimensions within the same modality, achieving more refined feature selection. Compared to traditional simple splicing or equal weighting methods, this mechanism better adapts to different scene changes, improving the robustness and sensitivity of the risk warning system in complex environments.

[0054] S600: Input a preset number of fused feature vectors into a bidirectional temporal network to obtain a temporal context representation, which includes historical information and future prediction information.

[0055] Specifically, the fused feature vectors generated in the most recent L time steps are fed into a bidirectional long short-term memory (BiLSTM) network in chronological order. In the forward network, the current input and the hidden state from the previous time step work together to calculate a new forward hidden state through gating units (input gate, forget gate, output gate, and state update). In the backward network, the current input is combined with the hidden states from subsequent time steps to update the backward hidden state in reverse order. After bidirectional computation, the forward and backward hidden states at corresponding time steps are concatenated to form a temporal context representation. This vector not only contains accumulated information from the past L-1 steps but also implicitly predicts future input trends through backward propagation, enabling the system to more accurately grasp the dynamic trajectory of risk evolution.

[0056] Understandably, bidirectional temporal networks, when processing multimodal fusion features, can capture the cumulative impact of past events (such as the emergence of anomalous events across multiple consecutive frames) and utilize backward information to predict potential risk peaks (such as the potential for sustained increases in wind speed or vibration in the coming frames). This allows them to incorporate a synergistic perspective of "history, present, and future" when generating temporal context representations. Compared to unidirectional networks, this representation can better model the temporal distribution characteristics of risk signals, effectively improving the timeliness and accuracy of risk scoring, especially in the initial stages of emergencies, providing richer and more complete spatiotemporal information for subsequent alarm decisions.

[0057] S700: Perform a fully connected mapping on the temporal context representation and obtain the risk probability score through the Sigmoid function.

[0058] Specifically, the temporal context representation output by BiLSTM is input into a single-layer or multi-layer fully connected network (MLP). This network can contain one or two hidden layers, each employing ReLU activation and Dropout regularization to prevent overfitting. The last layer outputs a single real number with added bias. The mapping formula is as follows:

[0059] in, For temporal context representation, This is the weight vector of the fully connected layer. For bias. Then for... Applying the Sigmoid function, we obtain the risk probability score:

[0060] in, This can represent the probability of risk occurring at the current moment. In this step, the MLP layer and the sigmoid function work together to map from the high-dimensional time series representation to the single-dimensional risk score, and learn the weights and biases by minimizing the binary cross-entropy loss during the training phase.

[0061] Understandably, by combining fully connected mapping with Sigmoid activation, the system can compress complex temporal context information into intuitive and easy-to-use probability scores, and use the continuity of probability values ​​to express the severity and urgency of risks. In practical applications, risk probabilities can not only be used for threshold judgment, but also combined with historical risk distributions for trend analysis. This helps maintenance personnel make preventative scheduling or maintenance decisions before the risk level reaches the alarm threshold, further improving the safe operation capability of the power supply system.

[0062] S800: Calculate the exponentially weighted average of the risk probability scores based on the risk probability scores, and generate an alarm signal when the exponentially weighted average continuously exceeds a preset threshold and the duration exceeds a preset time.

[0063] Specifically, the risk scores over multiple consecutive time steps are smoothed using an exponentially weighted moving average (EWMA), calculated as follows:

[0064] in, The smoothing coefficient is relatively large. Tendency to respond quickly to new changes, smaller Tendency to track long-term trends This is the initial value. The system also records... Since the first time the preset alarm threshold is exceeded. Duration from the point in time .when and ( When the preset minimum continuous alarm duration (e.g., 5 seconds) is reached, the system triggers a complete alarm signal generation process. The alarm signal content includes: the accurate PTP timestamp of the trigger time; the unique identifier of the monitoring unit and its geographical coordinates; the current smoothed risk score and corresponding risk level (mapped to "General", "Important", or "Urgent" based on the score range); and the system's preset standardized handling suggestion text (e.g., "Check the insulation of the corresponding equipment" or "Start the backup power supply"), which is simultaneously pushed to the command platform and maintenance personnel terminals via multiple channels such as WebSocket, SMS, and email.

[0065] Understandably, the EWMA smoothing mechanism can filter out short-term high scores caused by occasional noise, and also amplify early warning signals in a timely manner when risks accumulate. This ensures that alarm decisions are neither frequently triggered falsely due to occasional fluctuations, nor delayed due to a momentary drop after a single peak. The design of the duration threshold D balances rapid response to sudden and abrupt risks with robust identification of slowly evolving risks, ensuring that system alarms are both agile and reliable, providing a reasonably balanced triggering logic for power supply equipment anomaly early warning and emergency dispatch.

[0066] In this embodiment, the visual frame sequence is input into a lightweight MobileViT architecture visual feature extraction network to obtain visual feature vectors. This includes: normalizing and Gaussian denoising each frame in the visual frame sequence to obtain a preprocessed frame set. The normalization formula is as follows:

[0067] in, These are the normalized pixel values. For pixel values, For the average pixel value of the training set, The standard deviation of the training set pixels.

[0068] The formula for Gaussian denoising is:

[0069] in, These are the denoised video frames. For the size of a Gaussian kernel, This is a Gaussian operation.

[0070] The preprocessed frame set is input into the visual feature extraction network of the lightweight MobileViT architecture to obtain the visual features after global average pooling. The formula for calculating the visual features is as follows:

[0071] in, Let k be the visual feature vector of the k-th time window. A lightweight visual feature extractor; For global average pooling, is the dimension of the visual feature vector.

[0072] Specifically, each frame of the visual frame sequence is first acquired and then subjected to pixel-level normalization and Gaussian denoising to eliminate inconsistencies in brightness, contrast, and noise under different acquisition environments, and to provide a stable and standardized signal for the subsequent input of the lightweight MobileViT network. Normalization is performed according to the formula above. Here, μ and σ are the global pixel mean and standard deviation obtained statistically from the large-scale power grid monitoring image dataset, ensuring that all pixel values ​​after processing are distributed within an approximately standard normal distribution range. Subsequently, a Gaussian convolution kernel of size k is used for smoothing filtering to complete denoising. The frame set consisting of all N processed images is sequentially input into the pre-pruned, quantized, and fused local self-attention and convolution lightweight MobileViT architecture. The MobileViT network first divides each frame image into fixed-size blocks using the PatchEmbed module, and then rapidly extracts spatial features within the local window using Depthwise+Pointwise convolution. Finally, a lightweight Transformer Block completes cross-window self-attention to capture global contextual information. Finally, the network applies Global Average Pooling (GAP) to multi-scale feature maps at different levels, and further normalizes and prevents overfitting through BatchNorm and Dropout. While maintaining high expressive power, the entire processing flow, after network pruning and 8-bit quantization, can complete feature extraction on edge computing devices with a latency of less than 20ms / frame, meeting the real-time monitoring needs of large-scale, multi-channel concurrent video streams.

[0073] Understandably, this visual feature extraction process not only fully considers the practical challenges of power grid monitoring environments, such as changes in illumination, occlusion interference, and noise impact, but also achieves dual attention to global semantics and local details by combining a lightweight MobileViT network. In the normalization stage, after standardizing pixel values ​​using global statistics, the outputs of different cameras at each monitoring point can be mapped to the same standard space, thus avoiding distribution drift caused by device differences. Gaussian denoising effectively filters out pixel-level camera jitter and electronic noise, preserving key texture features and edge information, providing clearer input for the Transformer self-attention module. In the deep network, local window self-attention and cross-window information interaction ensure efficient fusion between local features and the global context, enhancing the ability to sensitively capture small-scale abnormal signals such as foreign object occlusion and electric arc flashes. The final global average pooling summarizes multi-scale, multi-level features into a fixed-dimensional vector, providing a compact and information-rich visual representation for subsequent multimodal fusion. This design balances accuracy and efficiency while meeting the stringent end-to-end latency requirements of power grid online monitoring scenarios, enabling the system to provide highly reliable and low-latency risk warnings under complex operating conditions.

[0074] In this embodiment of the application, after performing a short-time Fourier transform on the audio signal, it is input into an acoustic feature extraction network based on a convolutional neural network to obtain an acoustic feature vector. This includes: calculating a short-time Fourier transform on continuous audio segments to obtain a time-spectrum image, wherein the formula for the short-time Fourier transform is:

[0075] in, This represents the time-spectrum graph of the k-th time window. For frequency index, For time frame indexing, For the sample index within the window, The short-time Fourier transform window length. For time The original audio sample value at that location, For frame shift, .

[0076] The time-frequency spectrum is input into a convolutional neural network (CNN) consisting of multiple convolutional and pooling layers. The CNN extracts the time-frequency features of the audio through local receptive fields and weight sharing mechanisms, generating a fixed-dimensional acoustic feature representation. The output formula of the CNN is:

[0077] in, For the first The acoustic feature vector of the window, for The feature map after convolution and pooling operations, To flatten the feature map into a one-dimensional vector, This is the weight matrix of the fully connected layer. For the bias vector of the fully connected layer, is the dimension of the acoustic feature vector.

[0078] Specifically, in this embodiment, the system first divides the acquired continuous audio segments into several overlapping frames according to a preset window length and frame shift parameters, for example, each frame is 64ms long and the frame shift is 32ms, to ensure sufficient capture of short-time dynamic signals. Each audio frame is first pre-emphasized and filtered to enhance high-frequency components and suppress low-frequency noise. Then, a short-time Fourier transform (STFT) is performed on the pre-emphasized time-domain signal to calculate the time-spectrum. This time-spectrum not only preserves the spectral distribution information of the sound but also achieves the continuity of the time series through frame shift. Subsequently, the system inputs this time-spectrum into a lightweight convolutional neural network composed of alternating convolutional and pooling layers. The convolutional layers utilize local receptive fields to locally aggregate features in the frequency and time dimensions, while the pooling layers compress the spectrum size through downsampling and enhance translation invariance. The intermediate layers of the network often employ small-sized convolutional kernels and a sliding strategy with a stride of 1 to capture fine frequency domain textures and instantaneous energy fluctuations. Pooling layers can use either max pooling or average pooling to balance the need to preserve salient features with the need to reduce computational cost. The feature maps after convolution and pooling are flattened and then passed through a fully connected layer and activation function to finally generate fixed-dimensional acoustic feature vectors for subsequent fusion and temporal modeling.

[0079] Understandably, this acoustic feature extraction process maps the temporal information of audio frames to a frequency spectrum, and then uses the hierarchical structure of a convolutional neural network to automatically learn abnormal energy patterns in key frequency bands and time windows, achieving efficient perception of acoustic anomalies in power grid facilities. For example, the high-frequency humming sound caused by sudden current changes, the clanging impact sound during switching operations, or the continuous low-frequency vibration during a sudden increase in wind speed can all form specific energy distributions in the time spectrum. The convolutional layer will refine these distribution patterns layer by layer, while the pooling operation will highlight abnormal peaks while suppressing background noise. Flattening and fully connected layer mapping not only compress high-dimensional features into manageable vectors, but also introduce nonlinear discriminative capabilities through activation functions, so that the final acoustic feature vector achieves a balance between representational power and dimensional controllability. Combined with features from visual and environmental modalities, acoustic features provide the system with an independent and reliable risk clue, especially in low-light or visually obstructed scenarios, where sound signals can fill the blind spots of visual information, further improving the accuracy and robustness of overall risk warning.

[0080] In this embodiment, environmental sensor data is input into an environmental feature extraction network composed of fully connected layers to obtain an environmental feature vector. This includes: performing interpolation and moving average processing on the environmental parameter sequence channel by channel, wherein the formula for calculating the moving average is:

[0081] in, This represents the smoothed value of the j-th environmental parameter channel at time point t. Represents the original measurement value. This is the smoothing coefficient.

[0082] Calculate the average value of the smoothed environment parameters within the window, input the average value into the fully connected transform layer, and generate an environment feature vector through a nonlinear activation function. The expression for the nonlinear activation function is:

[0083] in, Let be the environmental feature vector for the k-th time window. It is the ReLU activation function. This is the weight matrix, with dimensions r×d (d is the number of environmental parameter channels). Let r be the bias vector, and r be the dimension of the environmental feature vector.

[0084] Specifically, the system first preprocesses the data for each channel of the environmental parameter sequence (such as temperature, humidity, wind speed, air pressure, and PMU phasor). Preprocessing includes two steps: interpolation and moving average. The interpolation step uses effective readings from adjacent time points to linearly or spline-fill missing data caused by occasional frame drops or communication jitter, ensuring that the data for each channel remains complete and continuous within the window. Subsequently, the system applies a moving average filter to the interpolated data sequence to smooth short-term abrupt noise. The moving average uses a recursive calculation method, weighting and fusing the current sampled value with the filtering result from the previous time point according to a preset weight ratio β, thus enabling rapid response to environmental changes while suppressing instantaneous measurement noise. After filtering, for the N smoothed data within each window, the system calculates the channel mean and variance, and concatenates them into a statistical vector. This statistical vector is then mapped through a two-layer fully connected network, each layer equipped with ReLU nonlinear activation and Batch Normalization to ensure stable feature distribution and model convergence. The mapped output is regularized by Dropout to generate an environmental feature vector of length r. This vector takes into account both the long-term trend and sudden fluctuation characteristics of environmental variables and can be used as a quantitative input for multimodal fusion.

[0085] Understandably, the aforementioned environmental feature extraction process aims to transform massive and easily disturbed physical parameters into low-dimensional feature representations with discriminative capabilities. On one hand, interpolation and moving averages ensure the integrity and smoothness of sensor time-series data, avoiding misjudgments caused by signal loss or measurement jitter in certain time periods. On the other hand, by combining statistical indicators (mean, variance) with nonlinear mapping, key patterns that may indicate equipment anomalies in environmental changes can be extracted. For example, continuous temperature increases may be related to excessive load, and sudden increases in wind speed may cause external insulation damage. The introduction of multilayer fully connected layers and ReLU activation further enhances the nonlinearity of feature representation, enabling different environmental factors to reflect their impact on risk with appropriate weights during multimodal fusion. The resulting environmental feature vector, when used in conjunction with visual and acoustic features, not only provides the system with dynamic external operating condition information but also enhances the overall model's perception of non-visual and non-acoustic events, thereby significantly improving the comprehensiveness and accuracy of power supply risk warnings.

[0086] In this embodiment, the weights of the visual feature vector, acoustic feature vector, and environmental feature vector are calculated separately using a gating attention mechanism, and a fused feature vector is generated by weighted summation. This includes concatenating the visual feature vector, acoustic feature vector, and environmental feature vector into a fused vector; and calculating the gating weight for each modality using the gating attention mechanism, with the following formula:

[0087] in, Let be the gated weight vector for mode m in the k-th time window, representing the reliability weight of this mode under the current environmental conditions. The weight matrix is ​​a learnable matrix with dimension . , This represents the total dimension of the concatenated vectors. , For bias vectors, This represents the Sigmoid activation function. Represents the fusion vector. This represents each mode.

[0088] The fused feature vector is generated by weighted summation, and the formula for calculating the weighted summation is as follows:

[0089] in, To fuse feature vectors, the dimension is , This indicates element-wise multiplication. Let m be the eigenvector of mode m. express 3D real space.

[0090] Specifically, the feature vectors from the visual, acoustic, and environmental sub-networks are first concatenated, connecting the output vectors of each modality sequentially along the channel dimension to form a fusion vector. Then, to assess the reliability of each modality based on the dynamic conditions of the real-time acquired environment, the system uses an independent gated attention sub-network to calculate the gate weight vector for each modality. After obtaining the three gate weights, the system combines them with the corresponding modal features using element-wise multiplication to generate weighted modal features. The entire gated attention module is executed in parallel with the backbone network, completing sequential concatenation, weight generation, and weighted summation in a single forward propagation, ensuring high efficiency in the computational process. During the training phase, the parameters of this module... and Together with other model parameters, the network is adjusted based on multimodal labeled data through backpropagation and gradient optimization, enabling the network to automatically learn to allocate the optimal attention to visual, acoustic and environmental modalities under different lighting, noise and environmental changes.

[0091] Understandably, the gated attention mechanism provides an interpretable and dynamically adaptive feature selection strategy for multimodal fusion. Compared to simple feature-level concatenation or equal weighting, it can dynamically assign different importance to each input based on the real-time modal quality and contextual information. When a modality experiences signal attenuation or distortion due to insufficient lighting, overlapping occlusions, or wind noise, the corresponding gated weight vector is automatically reduced, thereby suppressing the negative impact of that modal noise on the overall fused representation. Conversely, when a modality provides obvious anomalous precursors (such as sparks in a video frame, detected arcing sounds in audio, or sudden changes in environmental parameters), its corresponding weight is significantly increased, amplifying its contribution to the fused features. Through element-wise fine-grained control, this mechanism can also identify more diagnostically valuable feature dimensions within a single modality to further enhance important signals. The resulting fusion feature vector combines the complementary advantages of multi-source information, providing more robust and richer inputs in subsequent time-series models. This significantly improves the sensitivity and accuracy of the overall risk warning system under complex conditions, while maintaining efficient computation to meet the performance requirements of edge real-time inference scenarios.

[0092] In this embodiment, a predetermined number of fused feature vectors are input into a bidirectional temporal network to obtain a temporal context representation, including: calculating the forward hidden state and the backward hidden state based on the bidirectional temporal network. The formula for calculating the forward hidden state is as follows:

[0093] in, This represents the forward hidden state at time t. The computation function representing the feedforward LSTM network; The fused feature vector at time t, This represents the forward hidden state at time t-1.

[0094] The formula for calculating the backward hidden state is:

[0095] in, This represents the backward hidden state at time t. This represents the computation function of the feedforward LSTM network. This represents the backward hidden state at time t+1. The forward hidden state and the backward hidden state are merged into a temporal context representation.

[0096] Specifically, the system first extracts the fused feature vectors from the most recent L time steps from the fused feature sequence and uses this sequence as input. Simultaneously, a bidirectional long short-term memory (BiLSTM) operation is performed to extract temporal context information. During forward computation, the system sequentially feeds the fused features of the current time step and the forward hidden state of the previous time step into the forward LSTM unit. Through gating mechanisms such as input gates, forget gates, and output gates, combined with the internal candidate memory computation function, the forward hidden state is updated. Simultaneously, to acquire the ability to predict future time information, the system also uses the same feature sequence as reverse input, feeding the current fused features and the backward hidden state of the next time step into the backward LSTM unit. The backward hidden state is calculated through its internal gating and state update functions. This bidirectional computation is performed in parallel at the same time step, encoding the past and future context of the input sequence respectively. After completing the forward and backward computations, the system concatenates or weights and fuses the hidden state vectors at the corresponding time steps to form the final temporal context representation.

[0097] Understandably, bidirectional temporal context aggregation fully leverages the advantages of BiLSTM in processing sequence data: the forward hidden state accurately depicts changes in risk characteristics that may have accumulated in the past, such as continuous abnormal fluctuations or intermittent high-risk signals; the backward hidden state, without violating causality, simulates and predicts the potential risk evolution trend in the next few steps to a certain extent. This allows the system to consider both the impact of events that have already occurred and potential risks that are about to occur when making risk scores. By concatenating the forward and backward hidden states at each time step, the final temporal context representation has insights into both the past and the future, greatly improving the temporal sensitivity and prediction accuracy of multimodal early warning signals for power grid equipment.

[0098] In practical applications, this bidirectional aggregation mechanism can effectively handle various complex operating conditions. For example, when monitoring a high-voltage line, occasional visual flashes may be accompanied by sudden changes in wind speed or mechanical vibrations. A unidirectional model may struggle to accurately determine the risk level upon first detecting a flash. However, the bidirectional model, leveraging the predictive capabilities of the backward hidden state, can correlate flashes and vibrations with potential arcing or temperature increases that may occur in the next frame or more, triggering warnings more promptly. Furthermore, when risk signals appear intermittently across multiple time steps, the forward hidden state ensures the cumulative memory of these intermittent patterns, preventing the loss of alarm clues due to brief disappearances. Overall, the design of bidirectional temporal aggregation enables the system to respond quickly to sudden risks while maintaining high sensitivity to slowly progressive risks, providing richer and more accurate temporal feature inputs for subsequent risk scoring and alarm decisions.

[0099] In this embodiment, the method further includes: acquiring multiple alarm signals. The alarm time, alarm type, and alarm region of the multiple alarm signals are acquired. If the alarm type and alarm region are the same, and the difference between the alarm times is within a preset range, then the multiple alarm signals are merged into one alarm signal.

[0100] Specifically, when a new alarm signal is detected, all unarchived signals are first retrieved from the current active alarm list, and their types and regions are compared. If the new alarm is exactly the same as some signals in the list in terms of type and region, it is further determined whether the trigger time interval of these signals is within the window range configured by the operation management. As long as these conditions are met, the system will merge these related signals into the same alarm entry and uniformly update the start and end times of the trigger period, the cumulative number of triggers, and the index of the key camera frames involved in the record of the entry. After the merging is completed, the original multiple duplicate alarms are marked as merged records and removed from the active list, leaving only one alarm message reflecting the aggregated event.

[0101] Understandably, this merging strategy effectively avoids frequent repetitive alarms caused by threshold jitter or short-term multi-frame triggering within the same monitoring area, greatly reducing the interference of alarm noise on maintenance personnel. By flexibly configuring the merging time window, it can balance timely response to rapid and continuous anomalies with smooth filtering of long-term oscillating anomalies; at the same time, retaining the frequency statistics before merging can also support the subsequent "alarm aggregation" analysis module, providing classifications such as "10 similar alarms were triggered within 10 minutes, now summarized into 1," helping maintenance personnel quickly understand the alarm density and perform graded processing based on alarm level mapping.

[0102] In this embodiment of the application, the method further includes: generating a risk evolution trend based on the exponentially weighted average of risk probability scores. When the risk evolution trend shows a continuous upward trend for more than a preset time period, an alarm signal is generated.

[0103] Specifically, after generating a smoothed risk score, the system continuously monitors the score's trajectory over time and uses built-in trend analysis logic to assess the score's rise over consecutive periods. If the risk score is found to be continuously increasing over multiple sampling periods and the cumulative duration exceeds the preset minimum time limit, the system considers the potential risk to be escalating and generates a special "rising trend" type alarm. The alarm will indicate the trend's start time, duration, and preventative maintenance or scheduling recommendations based on the current rate of increase in the notification content.

[0104] Understandably, this trend analysis module supplements risk scoring with monitoring of both "evolution speed" and "duration," enabling the system to not only handle single-point sudden risks but also provide early warnings for slowly accumulating but still dangerous risks. For example, regarding leakage risks caused by aging insulation in power grid equipment, the risk score may fluctuate at the edge for a long time, but the upward trend is slow and continuous. When this slow rise continues to a foreseeable risk level, the system can remind maintenance personnel to perform early repairs, avoiding the "chronic" fault evolution that traditional threshold alarms cannot capture. In addition, by outputting slope information and trend duration, the severity of trend alarms can be further classified, providing more accurate timing judgments and quantitative basis for command decisions and resource scheduling.

[0105] This method achieves comprehensive and multi-layered perception of power grid operation risks through deep fusion of visual, acoustic, and environmental modalities, significantly improving the detection accuracy and early warning sensitivity of abnormal events. It employs a three-path feature extraction network using lightweight MobileViT, convolutional neural networks, and fully connected mappings, combined with a gated attention mechanism to dynamically allocate modal weights, effectively suppressing single-path noise interference and enhancing the ability to capture weak signals under complex operating conditions. The bidirectional temporal network retains past risk accumulation information while making preliminary predictions of future trends. Combined with exponentially weighted smoothing and duration verification, it can quickly respond to sudden emergencies and filter out occasional fluctuations, reducing false alarm rates. The entire process is completed in real-time at the edge, and model pruning and quantization ensure a single-frame latency of less than 100ms, achieving low-latency, high-concurrency online deployment. In the cloud, digital twins and Monte Carlo simulations are used to periodically calibrate the model and thresholds, further enhancing the system's robustness and adaptability. This method not only greatly reduces repetitive alarms and maintenance costs, but also generates multi-dimensional, visualized trend reports and handling suggestions, helping maintenance personnel to intervene earlier and develop precise response plans, providing efficient, intelligent, and scalable security for power grid supply.

[0106] Figure 2 This is a schematic diagram of a power supply risk early warning system module based on multimodal data fusion provided in an embodiment of this application. Figure 2 The power supply risk early warning system 10 based on multimodal data fusion shown includes at least the following components: a data acquisition module 11, a feature extraction module 12, a feature fusion module 13, a bidirectional time series network 14, a risk probability score acquisition module 15, and an alarm module 16.

[0107] In this embodiment, the data acquisition module 11 is used to synchronously acquire visual frame sequences, audio signals, and environmental sensor data around the power supply equipment using a precise time protocol. Please refer to the documentation for details on the acquisition method. Figure 1 The details and their corresponding descriptions are not repeated here.

[0108] In this embodiment, the feature extraction module 12 is used to input the visual frame sequence into a lightweight MobileViT architecture visual feature extraction network to obtain visual feature vectors; it is also used to perform a short-time Fourier transform on the audio signal and then input it into an acoustic feature extraction network based on a convolutional neural network to obtain acoustic feature vectors; and it is also used to input environmental sensor data into an environmental feature extraction network composed of fully connected layers to obtain environmental feature vectors. For specific acquisition methods, please refer to the accompanying documentation. Figure 1 The details and their corresponding descriptions are not repeated here.

[0109] In this embodiment, the feature fusion module 13 is used to calculate the weights of the visual feature vector, acoustic feature vector, and environmental feature vector respectively through a gated attention mechanism, and generate a fused feature vector by weighted summation. For details on how to obtain the fused feature vector, please refer to the relevant documentation. Figure 1 The details and their corresponding descriptions are not repeated here.

[0110] In this embodiment, the bidirectional temporal network 14 is used to accept a predetermined number of fused feature vectors to obtain a temporal context representation. The temporal context representation includes historical information and future prediction information. Please refer to the documentation for details on how to obtain the representation. Figure 1 The details and their corresponding descriptions are not repeated here.

[0111] In this embodiment, the risk probability score acquisition module 15 is used to perform a fully connected mapping on the temporal context representation and obtain the risk probability score through the Sigmoid function. For details on the acquisition method, please refer to the relevant documentation. Figure 1 The details and their corresponding descriptions are not repeated here.

[0112] In this embodiment, the alarm module 16 is used to calculate the exponentially weighted average of the risk probability scores based on the risk probability scores, and to generate an alarm signal when the exponentially weighted average continuously exceeds a preset threshold and the duration exceeds a preset time. For details on how to obtain the alarm signal, please refer to the following documentation. Figure 1 The details and their corresponding descriptions are not repeated here.

[0113] Figure 3 This is an example of an alarm device 20 provided in one embodiment of this application. For example... Figure 3 As shown, the alarm device 20 includes at least the following components: processor 21 and memory 22.

[0114] In this embodiment, the memory 22 is used to store executable instructions of the processor 21, which, when configured to execute instructions, implement... Figure 1 The power supply risk early warning method based on multimodal data fusion is shown.

[0115] In one embodiment of this application, the program operating in the alarm device 20 can be a program that controls a central processing unit (CPU) or similar device to achieve the functions described in the above-described embodiments of the present invention (a program that enables the computer to function). The information processed by these devices is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (Flash ROM) or hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0116] It should be noted that a portion of the alarm device 20 described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0117] It should be noted that the "computer" mentioned here refers to the computer built into the alarm device 20, which uses hardware including an operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard drives built into the computer.

[0118] Furthermore, a "computer-readable recording medium" can include: a medium that dynamically stores a program for a short period of time, such as a communication line used when transmitting a program via a network such as the Internet or a communication line such as a telephone line; or a medium that stores a program for a fixed period of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the program described above can be a program used to implement the above-described functions, or it can be a program that can implement the above-described functions by combining it with a program already recorded in a computer. Furthermore, the alarm device 20 in the above embodiment can also be implemented as an assembly (device group) composed of multiple devices. Each device constituting the device group can possess some or all of the functions or functional blocks of the alarm device 20 in the above embodiment. As a device group, it is sufficient to have all the functions or functional blocks of the alarm device 20.

[0119] It is understood that the power supply risk early warning method, device, and system based on multimodal data fusion provided in this application significantly improves the accuracy of risk identification and reduces false alarms and missed alarms caused by single sensor failure or interference by using multimodal data fusion to collaboratively analyze visual, acoustic, and environmental sensor information. Real-time time-series aggregation and gating mechanisms effectively smooth short-term fluctuations, ensuring greater stability and reliability of alarms. Secondly, lightweight edge-side inference combined with cloud-based digital twin simulation achieves a closed loop of "local early warning + cloud calibration," meeting the real-time early warning requirements of low latency and high concurrency while dynamically and adaptively adjusting thresholds and model parameters, improving the overall system's robustness and maintainability. Thirdly, the system integrates an intelligent decision engine, utilizing reinforcement learning and graph neural networks to automatically generate scheduling and emergency response plans, assisting maintenance personnel in quickly locating faults and optimizing resource allocation, significantly shortening fault response and handling time, and achieving lean and intelligent power supply services. Finally, the solution is highly modular and scalable, supports hierarchical visual alarm management and batch configuration, and can be smoothly integrated into third-party GIS, SCADA and other systems, and has good promotional value and benchmark demonstration effect.

[0120] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A power supply risk early warning method based on multimodal data fusion, applied to monitoring power supply equipment, characterized in that, The method includes: Visual frame sequences, audio signals, and environmental sensor data around the power supply equipment are synchronously acquired using a precise time protocol. The visual frame sequence is input into the visual feature extraction network of the lightweight MobileViT architecture to obtain visual feature vectors; After performing a short-time Fourier transform on the audio signal, it is input into an acoustic feature extraction network based on a convolutional neural network to obtain acoustic feature vectors. The environmental sensor data is input into an environmental feature extraction network composed of fully connected layers to obtain environmental feature vectors. The weights of the visual feature vector, the acoustic feature vector, and the environmental feature vector are calculated using a gating attention mechanism, and a fused feature vector is generated by weighted summation. A predetermined number of fused feature vectors are input into a bidirectional temporal network to obtain a temporal context representation, which includes historical information and future prediction information. The temporal context representation is fully connected and mapped, and the risk probability score is obtained through the Sigmoid function; The risk probability score is calculated as an exponentially weighted average, and an alarm signal is generated when the exponentially weighted average continuously exceeds a preset threshold for a duration exceeding a preset time.

2. The power supply risk early warning method based on multimodal data fusion according to claim 1, characterized in that, The step of inputting the visual frame sequence into the visual feature extraction network of the lightweight MobileViT architecture to obtain visual feature vectors includes: Each frame in the visual frame sequence is normalized and Gaussian denoising is performed to obtain a preprocessed frame set. The normalization formula is as follows: ； in, These are the normalized pixel values. For pixel values, For the average pixel value of the training set, The standard deviation of the training set pixels; The formula for Gaussian denoising is: ； in, These are the denoised video frames. For the size of a Gaussian kernel, Gaussian operation; The preprocessed frame set is input into the visual feature extraction network of the lightweight MobileViT architecture to obtain visual features after global average pooling. The calculation formula for the visual features is as follows: ； in, Let k be the visual feature vector of the k-th time window. A lightweight visual feature extractor; For global average pooling, is the dimension of the visual feature vector.

3. The power supply risk early warning method based on multimodal data fusion according to claim 2, characterized in that, After performing a short-time Fourier transform on the audio signal, the signal is input into an acoustic feature extraction network based on a convolutional neural network to obtain acoustic feature vectors, including: Calculate the short-time Fourier transform for a continuous audio segment to obtain a time-spectrum graph. The formula for the short-time Fourier transform is: ； in, This represents the time-spectrum graph of the k-th time window. For frequency index, For time frame indexing, For the sample index within the window, The short-time Fourier transform window length. For time The original audio sample value at that location, For frame shift, ; The time-frequency spectrum is input into a convolutional neural network (CNN) consisting of multiple convolutional and pooling layers. The CNN extracts the time-frequency features of the audio through local receptive fields and weight sharing mechanisms, generating a fixed-dimensional acoustic feature representation. The output formula of the CNN is: ； in, For the first The acoustic feature vector of the window, for The feature map after convolution and pooling operations, To flatten the feature map into a one-dimensional vector, This is the weight matrix of the fully connected layer. For the bias vector of the fully connected layer, is the dimension of the acoustic feature vector.

4. The power supply risk early warning method based on multimodal data fusion according to claim 3, characterized in that, The step of inputting the environmental sensor data into an environmental feature extraction network composed of fully connected layers to obtain an environmental feature vector includes: The environmental parameter sequence is processed by interpolation and moving average for each channel. The formula for calculating the moving average is as follows: ； in, This represents the smoothed value of the j-th environmental parameter channel at time point t. Represents the original measurement value. For smoothing coefficients; Calculate the average value of the smoothed environment parameters within the window, input the average value into a fully connected transform layer, and generate an environment feature vector through a nonlinear activation function. The expression of the nonlinear activation function is as follows: ； in, Let be the environmental feature vector for the k-th time window. It is the ReLU activation function. This is the weight matrix, with dimensions r×d (d is the number of environmental parameter channels). Let r be the bias vector, and r be the dimension of the environmental feature vector.

5. The power supply risk early warning method based on multimodal data fusion according to claim 4, characterized in that, The process involves calculating the weights of the visual feature vector, the acoustic feature vector, and the environmental feature vector using a gated attention mechanism, and generating a fused feature vector through a weighted summation, including: The visual feature vector, the acoustic feature vector, and the environmental feature vector are concatenated into a fusion vector. The gating weights for each modality are calculated using a gating attention mechanism, and the formula is as follows: ； in, Let be the gated weight vector for mode m in the k-th time window, representing the reliability weight of this mode under the current environmental conditions. The weight matrix is ​​a learnable matrix with dimension . , This represents the total dimension of the concatenated vectors. , For bias vectors, This represents the Sigmoid activation function. Represents the fusion vector. Represent each mode; A fused feature vector is generated by weighted summation, and the formula for calculating the weighted summation is as follows: ； in, To fuse feature vectors, the dimension is , This indicates element-wise multiplication. Let m be the eigenvector of mode m. express 3D real space.

6. The power supply risk early warning method based on multimodal data fusion according to claim 5, characterized in that, The step of inputting a continuously preset number of fused feature vectors into a bidirectional temporal network to obtain a temporal context representation includes: The forward hidden state and backward hidden state are calculated based on the bidirectional temporal network. The formula for calculating the forward hidden state is as follows: ； in, This represents the forward hidden state at time t. The computation function representing the feedforward LSTM network; The fused feature vector at time t, This is the forward hidden state at time t-1; The formula for calculating the backward hidden state is: ； in, This represents the backward hidden state at time t. This represents the computation function of the feedforward LSTM network. This represents the backward hidden state at time t+1; The forward hidden state and the backward hidden state are merged into the temporal context representation.

7. The power supply risk early warning method based on multimodal data fusion according to claim 1, characterized in that, The method further includes: Acquire multiple alarm signals; Obtain the alarm time, alarm type, and alarm area of ​​multiple alarm signals; If the alarm type and alarm area are the same, and the difference between the alarm times is within a preset range, then the multiple alarm signals will be merged into one alarm signal.

8. The power supply risk early warning method based on multimodal data fusion according to claim 1, characterized in that, The method further includes: A risk evolution trend is generated based on the exponentially weighted average of the risk probability scores; An alarm signal is generated when the risk evolution trend shows a continuous upward trend for more than a preset time.

9. An alarm device, characterized in that, include: processor; as well as A memory having computer-readable instructions stored thereon for controlling the processor to execute the power supply risk early warning method based on multimodal data fusion as described in any one of claims 1 to 8.

10. A power supply risk early warning system based on multimodal data fusion, applied to the power supply risk early warning method based on multimodal data fusion as described in any one of claims 1 to 8, the system comprising: The data acquisition module is used to synchronously collect visual frame sequences, audio signals, and environmental sensor data around the power supply equipment via a precision time protocol. The feature extraction module is used to input the visual frame sequence into a lightweight MobileViT architecture visual feature extraction network to obtain visual feature vectors; it is also used to perform a short-time Fourier transform on the audio signal and input it into an acoustic feature extraction network based on a convolutional neural network to obtain acoustic feature vectors; it is also used to input the environmental sensor data into an environmental feature extraction network composed of fully connected layers to obtain environmental feature vectors. The feature fusion module is used to calculate the weights of the visual feature vector, the acoustic feature vector, and the environmental feature vector respectively through a gated attention mechanism, and generate a fused feature vector by weighted summation. A bidirectional temporal network is used to accept a predetermined number of fused feature vectors to obtain a temporal context representation, which includes historical information and future prediction information. The risk probability score acquisition module is used to perform a fully connected mapping on the temporal context representation and obtain the risk probability score through the Sigmoid function. An alarm module is used to calculate the exponentially weighted average of the risk probability score based on the risk probability score, and to generate an alarm signal when the exponentially weighted average continuously exceeds a preset threshold and the duration exceeds a preset time.

Images