A power grid operation and maintenance multi-modal data anomaly detection method and system

Abstract

The present application relates to the technical field of power operation and maintenance management, and particularly relates to a power grid operation and maintenance multi-modal data anomaly detection method and system, comprising feature extraction and class imbalance correction on power grid operation and maintenance inspection text data to obtain a text deep semantic feature vector; pseudo-label extraction and richness perception weak supervision analysis on power grid operation and maintenance inspection audio and video data to obtain visual feature representation; multi-modal cross-layer attention fusion of the text deep semantic feature vector and the visual feature representation to obtain cross-modal joint feature representation; semantic acoustic multi-modal collaborative decoding on the cross-modal joint feature representation to generate operation and maintenance event comprehensive analysis result, anomaly detection according to the operation and maintenance event comprehensive analysis result to generate power grid operation and maintenance multi-modal anomaly detection result. The present application realizes deep fusion of multi-modal data and accurate quantitative evaluation of abnormal state in the power grid operation and maintenance scene, and improves the accuracy and robustness of power grid operation and maintenance anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of power operation and maintenance management technology, and in particular to a method and system for detecting anomalies in multimodal data of power grid operation and maintenance. Background Technology

[0002] With the expansion of the power industry and the increasing complexity of the power grid structure, the amount of operation and maintenance data has exploded. Power grid operation and maintenance data covers multimodal data such as text, audio and video, which together constitute the key information carrier of power grid operation and maintenance ledgers. However, existing power grid operation and maintenance anomaly detection technologies have obvious defects in processing these multimodal data and are difficult to meet the needs of accurate operation and maintenance decision-making.

[0003] Power grid operation and maintenance text data suffers from numerous categories, high and sparse feature dimensions, and uneven sample distribution. Traditional text processing methods based on bag-of-words models or shallow machine learning struggle to capture long-distance semantic dependencies and effectively extract deep semantic features, leading to insufficient information utilization and impacting the accuracy of operation and maintenance decisions. Regarding audio and video data, while equipment operation sound monitoring and video surveillance can provide on-site information, audio and video parsing faces challenges such as difficulty in obtaining labels and high annotation costs. Manual annotation is almost impractical on large-scale datasets, severely limiting the training of supervised learning models. Existing technologies often fail to efficiently identify abnormal signals in audio or key events in video under unsupervised or weakly supervised conditions, hindering operation and maintenance personnel from achieving high efficiency. The difficulty in timely locating equipment anomalies is compounded by the fact that multimodal data in power grid operation and maintenance are interconnected and complementary. For example, the fault description in a text report may point to the same operation and maintenance event as abnormal sounds in audio or equipment images in video. However, existing technologies are usually limited to single-modal processing and lack effective means to integrate the inherent connections and complementarities of multimodal information. This results in insufficient cross-modal feature representation capabilities, leading to one-sided information integration and ultimately making operation and maintenance decisions lack comprehensiveness and accuracy. In summary, existing technologies have many shortcomings in the processing of multimodal data in power grid operation and maintenance ledgers, such as insufficient deep semantic extraction, making it difficult to achieve truly effective multimodal information complementarity and joint representation, resulting in a low detection rate of abnormal events. Summary of the Invention

[0004] To address the above technical problems, this invention provides a method and system for detecting anomalies in multimodal data during power grid operation and maintenance.

[0005] In a first aspect, the present invention provides a method for detecting anomalies in multimodal data of power grid operation and maintenance, the method comprising the following steps: The multimodal operation and maintenance data collected in the power grid operation and maintenance scenario are preprocessed to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data; Feature extraction is performed on the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features of the text, and class imbalance correction is performed on the long-distance contextual semantic features of the text to obtain deep semantic feature vectors of the text. Pseudo-tags are extracted from the audio and video data of the power grid operation and maintenance inspection to obtain segment-level visual pseudo-tags and video-level visual pseudo-tags; The fragment-level visual pseudo-labels and the video-level visual pseudo-labels are subjected to richness-aware weakly supervised parsing to obtain visual feature representations; The deep semantic feature vector of the text and the visual feature representation are fused through multimodal cross-layer attention to obtain a cross-modal joint feature representation; Semantic-acoustic multimodal collaborative decoding is performed on the cross-modal joint feature representation to generate comprehensive analysis results of operation and maintenance events. Anomaly detection is then performed based on the comprehensive analysis results of operation and maintenance events to generate multimodal anomaly detection results for power grid operation and maintenance.

[0006] In a further implementation, the step of extracting features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features, and performing class imbalance correction on the long-distance contextual semantic features to obtain a deep semantic feature vector of the text includes: The power grid operation and maintenance inspection text data is segmented into words to obtain a word sequence, and the word sequence is then subjected to distributed word vector embedding processing to obtain a word vector matrix; The word vector matrix is ​​subjected to residual block hierarchical extraction and deformed convolution processing using a convolutional layer composed of two residual blocks to obtain a local text feature matrix; Based on the local text feature matrix, channel-space attention weighting is applied to the key semantic regions of the text using an attention mechanism to obtain an attention-enhanced local feature matrix. Bidirectional cyclic semantic modeling is performed on the long-distance contextual dependencies of the attention-enhanced local feature matrix to obtain long-distance contextual semantic features of the text. Based on the long-distance contextual semantic features of the text, the deep semantic feature vector of the text is obtained by optimizing the Gaussian weight adaptive loss for the imbalance of the number of abnormal category samples by improving the focus loss constraint.

[0007] In a further embodiment, each of the residual blocks includes a residual convolution block, a deformed convolution block, a first convolution unifying block, an attention block, a residual stacking block, and an activation block connected in sequence. The residual convolutional block comprises a first convolutional layer, a normalization layer, and an activation layer connected in sequence; the deformable convolutional block comprises a deformable convolutional layer, a normalization layer, and an activation layer connected in sequence; the first convolutional normalization block comprises a second convolutional layer and a normalization layer connected in sequence; the deformable convolutional block adopts a fourth-generation deformable convolutional network structure, and the deformable convolutional layer adds a text contour geometric deformation offset to each sampling point of the convolutional kernel.

[0008] In a further implementation scheme, the step of extracting pseudo-tags from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo-tags and video-level visual pseudo-tags includes: Each video sample in the power grid operation and maintenance inspection audio and video data is uniformly divided into several video segments to obtain the image features of the video segments. The power grid operation and maintenance inspection event categories are embedded into preset text prompt templates, and the category text features of each power grid operation and maintenance inspection event category are extracted. Calculate the normalized cosine similarity between the video segment image features and the category text features, and generate a segment-level score mask based on the normalized cosine similarity and a preset similarity probability threshold; Based on the segment-level score mask and the preset audio-video joint label, the visual event categories existing in each video segment are filtered out to obtain the segment-level visual pseudo-label for each video segment; The fragment-level visual pseudo-tags are traversed, and the fragment-level visual pseudo-tags are aggregated according to the power grid operation and maintenance inspection event categories to obtain video-level visual pseudo-tags.

[0009] In a further implementation, the step of performing richness-aware weakly supervised parsing on the fragment-level visual pseudo-tags and the video-level visual pseudo-tags to obtain visual feature representations includes: The pseudo-label category richness is obtained by calculating the ratio of the number of power grid operation and maintenance inspection event categories in each video segment to the total number of all power grid operation and maintenance inspection event categories based on the segment-level visual pseudo-labels. The pseudo-label segment richness is obtained by calculating the ratio of the number of video segments for a single power grid operation and maintenance inspection event category to the total number of video segments. Visual prediction is performed on the audio and video data of the power grid operation and maintenance inspection to obtain the visual prediction result. The visual prediction result, the richness of the pseudo-label category, and the richness of the pseudo-label segment are aligned to obtain the richness perception loss. The video-level visual pseudo-labels are used as supervision information, and a video-level basic loss is constructed based on the visual prediction results. Based on the richness perception loss and the video-level basic loss, forward propagation loss analysis is performed on the segment-level visual pseudo-labels to obtain the localization results of abnormally high loss segments. Based on the location results of the abnormally high loss segments, the segment-level visual pseudo-labels are reversed and corrected to obtain optimized pseudo-labels. Based on the optimized pseudo-labels, weakly supervised audiovisual analysis is performed on the power grid operation and maintenance inspection audio and video data to obtain visual feature representations.

[0010] In a further implementation, the step of performing multimodal cross-layer attention fusion of the deep semantic feature vector of the text and the visual feature representation to obtain a cross-modal joint feature representation includes: The deep semantic feature vector of the text and the visual feature representation are respectively subjected to projection transformation to obtain semantic embedding sequence and visual embedding sequence of the same dimension; The semantic embedding sequence and the visual embedding sequence are concatenated in the time dimension to form a multimodal fusion sequence, and positional encoding information is superimposed on the multimodal fusion sequence to obtain a multimodal fusion embedding sequence. The degree of information interaction between different positions in the multimodal fusion embedding sequence is analyzed by a multi-head self-attention mechanism to obtain the multi-head self-attention matrix; Based on the multi-head self-attention matrix, a two-dimensional convolutional evolution is performed on the cross-layer attention distribution to obtain the cross-layer evolved attention matrix; The cross-layer attention information is dynamically fused based on the cross-layer evolutionary attention matrix and the multi-head self-attention matrix to obtain the cross-layer joint attention matrix; The multimodal fusion embedding sequence is nonlinearly transformed using the cross-layer joint attention matrix to obtain a cross-modal joint feature representation.

[0011] In a further implementation, the steps of performing semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representation to generate a comprehensive analysis result of the operation and maintenance event, and performing anomaly detection based on the comprehensive analysis result of the operation and maintenance event to generate a multimodal anomaly detection result for power grid operation and maintenance include: Based on the cross-modal joint feature representation, the text semantic features and acoustic visual features are projected and embedded to obtain a multimodal collaborative embedding sequence; Based on the multimodal cooperative embedding sequence, semantic acoustic multimodal cooperative decoding is performed on the cross-modal anomaly semantic information to obtain the anomaly pattern probability distribution; Based on the probability distribution of the abnormal patterns, abnormal events are identified in the operating status of power equipment to obtain comprehensive analysis results of operation and maintenance events. Based on the comprehensive analysis results of the operation and maintenance events, the severity of the anomalies is quantitatively assessed to determine the risk level, and multimodal anomaly detection results for power grid operation and maintenance are generated.

[0012] Secondly, the present invention provides a multimodal data anomaly detection system for power grid operation and maintenance, the system comprising: The data acquisition module is used to preprocess the multimodal operation and maintenance data collected in the power grid operation and maintenance scenario to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data. The feature extraction module is used to extract features from the power grid operation and maintenance inspection text data, obtain long-distance contextual semantic features of the text, and perform class imbalance correction on the long-distance contextual semantic features of the text to obtain deep semantic feature vectors of the text. The tag extraction module is used to extract pseudo tags from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo tags and video-level visual pseudo tags. The visual analysis module is used to perform richness-aware weakly supervised parsing on the fragment-level visual pseudo-tags and the video-level visual pseudo-tags to obtain visual feature representations. The cross-layer fusion module is used to perform multimodal cross-layer attention fusion of the deep semantic feature vector of the text and the visual feature representation to obtain a cross-modal joint feature representation; The anomaly detection module is used to perform semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representation, generate comprehensive analysis results of operation and maintenance events, and perform anomaly detection based on the comprehensive analysis results of operation and maintenance events to generate multimodal anomaly detection results for power grid operation and maintenance.

[0013] In a further implementation, the feature extraction module is specifically used for: The power grid operation and maintenance inspection text data is segmented into words to obtain a word sequence, and the word sequence is then subjected to distributed word vector embedding processing to obtain a word vector matrix; The word vector matrix is ​​subjected to residual block hierarchical extraction and deformed convolution processing using a convolutional layer composed of two residual blocks to obtain a local text feature matrix; Based on the local text feature matrix, channel-space attention weighting is applied to the key semantic regions of the text using an attention mechanism to obtain an attention-enhanced local feature matrix. Bidirectional cyclic semantic modeling is performed on the long-distance contextual dependencies of the attention-enhanced local feature matrix to obtain long-distance contextual semantic features of the text. Based on the long-distance contextual semantic features of the text, the deep semantic feature vector of the text is obtained by optimizing the Gaussian weight adaptive loss for the imbalance of the number of abnormal category samples by improving the focus loss constraint.

[0014] In a further embodiment, each of the residual blocks includes a residual convolution block, a deformed convolution block, a first convolution unifying block, an attention block, a residual stacking block, and an activation block connected in sequence. The residual convolutional block comprises a first convolutional layer, a normalization layer, and an activation layer connected in sequence; the deformable convolutional block comprises a deformable convolutional layer, a normalization layer, and an activation layer connected in sequence; the first convolutional normalization block comprises a second convolutional layer and a normalization layer connected in sequence; the deformable convolutional block adopts a fourth-generation deformable convolutional network structure, and the deformable convolutional layer adds a text contour geometric deformation offset to each sampling point of the convolutional kernel.

[0015] This invention provides a method and system for anomaly detection in multimodal data of power grid operation and maintenance. The method includes: preprocessing multimodal operation and maintenance data collected in power grid operation and maintenance scenarios to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data; extracting features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features, and performing class imbalance correction on the long-distance contextual semantic features to obtain deep semantic feature vectors; extracting pseudo-labels from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo-labels and video-level visual pseudo-labels; performing richness-aware weakly supervised parsing on the segment-level visual pseudo-labels and video-level visual pseudo-labels to obtain visual feature representations; performing multimodal cross-layer attention fusion on the deep semantic feature vectors of the text and the visual feature representations to obtain cross-modal joint feature representations; performing semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representations to generate comprehensive analysis results of operation and maintenance events, and performing anomaly detection based on the comprehensive analysis results of operation and maintenance events to generate multimodal anomaly detection results for power grid operation and maintenance. Compared with existing technologies, this method achieves deep fusion of multimodal data and accurate quantitative assessment of abnormal states in power grid operation and maintenance scenarios through the organic synergy of deep semantic extraction of text and adaptive category equalization, thereby improving the accuracy and robustness of anomaly detection and providing intelligent decision support for the safe and stable operation of the power grid. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the process for detecting anomalies in multimodal data of power grid operation and maintenance provided in an embodiment of the present invention; Figure 2 This is a block diagram of the power grid operation and maintenance multimodal data anomaly detection system provided in an embodiment of the present invention.

[0017] Figure labeling: 101, Data acquisition module; 102, Feature extraction module; 103, Label extraction module; 104, Visual analysis module; 105, Cross-layer fusion module; 106, Anomaly detection module. Detailed Implementation

[0018] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0019] Figure 1 This is a schematic flowchart of a multi-modal data anomaly detection method for power grid operation and maintenance provided in an embodiment of the present invention. The present invention provides a multi-modal data anomaly detection method for power grid operation and maintenance, such as... Figure 1 As shown, the method includes the following steps: S1. Preprocess the multimodal operation and maintenance data collected in the power grid operation and maintenance scenario to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data.

[0020] Specifically, this embodiment extracts text data such as inspection reports, fault records, and operation instructions associated with the target power equipment from the power grid operation and maintenance ledger, along with corresponding video surveillance footage and voiceprint monitoring signals (i.e., equipment operating sounds), forming multimodal operation and maintenance data in the power grid operation and maintenance scenario. Since Chinese text lacks natural word boundaries, this embodiment uses a trained Jieba word segmentation method to segment the original text data (such as power grid operation and maintenance inspection reports, fault records, and operation instructions) into independent word sets, obtaining a word sequence. The length of this word sequence is equal to the total number of words in the input text data. This segmentation process, based on a trained word segmentation dictionary, can effectively identify power-related proprietary terms such as relay protection devices and voltage transformers, thus obtaining a word sequence with a length corresponding to the number of words in the input text. Then, this embodiment inputs the segmented word sequence into the embedding layer for word processing. In the vector embedding process, the embedding layer employs a distributed word vector training method to map each word into a low-dimensional dense numerical vector. Specifically, this embodiment utilizes a dictionary pre-trained on a power grid operation and maintenance corpus to replace each word in the word sequence with its corresponding word vector, thereby converting the entire text into a two-dimensional text word vector matrix. The number of rows in this text word vector matrix equals the number of words in the text, and the number of columns equals the dimension of the word vectors. The dimension of the word vectors is set according to the scale of the power field corpus. Since the power grid operation and maintenance text data is large in volume and contains a large number of professional terms, the dictionary used in the word embedding part is trained using the model training dataset itself to ensure accurate representation of power field terminology. After the embedding layer processing, the original text data is converted into a text word vector sequence that is easy for subsequent calculation and processing. Each word vector in this sequence contains the semantic information of the corresponding word. In this embodiment, all text word vector matrices are integrated to obtain power grid operation and maintenance inspection text data.

[0021] Meanwhile, since video data is usually a continuous sequence of image frames, in order to reduce the computational load of subsequent processing and extract representative visual information, this embodiment performs time-axis uniform segmentation processing on the monitoring video data of the same target power equipment during the operation of the power grid equipment at fixed time intervals (e.g., every 1 second or every 2 seconds). Each video sample is uniformly divided into several video segments according to a preset time length. Each video segment corresponds to a fixed time window, and keyframe sampling is performed on each video segment. One frame is sampled from all image frames of each video segment as the visual representative frame of that video segment. The image features of the visual representative frame characterize the visual content of the entire segment. The sampling strategy can be to take the middle frame, the first frame, or random frames. This embodiment prioritizes selecting the middle frame of each segment to take into account the visual information before and after the segment. All visual representative frames are arranged in chronological order to form a video data feature sequence. In addition, for the same target device, this embodiment divides its voiceprint monitoring signal according to the same fixed time interval as the video to obtain audio segments that correspond one-to-one with the video segments, ensuring that the audio and video are strictly aligned in time. Short-time Fourier transform is used to analyze each audio segment. This embodiment extracts acoustic features from audio segments. Specifically, it converts the time-domain waveform of each audio segment into a time-frequency domain representation to obtain a spectrogram. Based on this, the spectrogram is converted into Mel-frequency cepstral coefficients to form a two-dimensional acoustic feature map. The horizontal axis of this acoustic feature map corresponds to the time step, and the vertical axis corresponds to the frequency channel. To reduce the length of the time dimension while retaining key acoustic information, this embodiment uses a two-layer two-dimensional convolutional neural network to downsample the acoustic feature map. The convolutional kernel has a set size and stride in the time and frequency directions, respectively. Through a large stride convolution operation in the time direction, The number of time steps is reduced, and the joint features of adjacent time-frequency units are extracted by utilizing the local receptive field of the convolution kernel. The output after downsampling is a compressed acoustic feature sequence. This acoustic feature sequence retains the abnormal sound features of the equipment while significantly reducing the consumption of computing resources. In this embodiment, the video data feature sequence and the acoustic feature sequence are integrated to obtain power grid operation and maintenance inspection audio and video data. Among them, the text word vector sequence represents the semantic information of equipment operating status, maintenance history and abnormal conditions, and the audio and video segment frame sequence represents the spatiotemporal information of equipment appearance status, operating environment and abnormal sound.

[0022] S2. Extract features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features of the text, and perform class imbalance correction on the long-distance contextual semantic features of the text to obtain deep semantic feature vector of the text.

[0023] In some implementations, the step of extracting features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features, and then performing class imbalance correction on the long-distance contextual semantic features to obtain a deep semantic feature vector of the text includes: The power grid operation and maintenance inspection text data is segmented into words to obtain a word sequence, and the word sequence is then subjected to distributed word vector embedding processing to obtain a word vector matrix; The word vector matrix is ​​subjected to residual block hierarchical extraction and deformed convolution processing using a convolutional layer composed of two residual blocks to obtain a local text feature matrix; Based on the local text feature matrix, channel-space attention weighting is applied to the key semantic regions of the text using an attention mechanism to obtain an attention-enhanced local feature matrix. Bidirectional cyclic semantic modeling is performed on the long-distance contextual dependencies of the attention-enhanced local feature matrix to obtain long-distance contextual semantic features of the text. Based on the long-distance contextual semantic features of the text, the deep semantic feature vector of the text is obtained by optimizing the Gaussian weight adaptive loss for the imbalance of the number of abnormal category samples by improving the focus loss constraint.

[0024] Specifically, in this embodiment, the input text data of power grid operation and maintenance inspection is processed by a convolutional layer consisting of two concatenated residual blocks for local feature residual extraction. Each residual block can introduce original input features in a timely manner to alleviate the gradient degradation problem in deep network training. In this embodiment, each residual block includes a residual convolutional block, a deformable convolutional block, a first convolutional normalization block, an attention block, a residual stacking block, and an activation block connected in sequence, forming an end-to-end structure for feature enhancement, noise suppression, and detail preservation. The residual convolutional block includes a first convolutional layer, a normalization layer, and an activation layer connected in sequence, and is used to extract basic local features from the input feature matrix. The deformable convolutional block includes a deformable convolutional layer, a normalization layer, and an activation layer connected in sequence. This deformable convolutional layer adopts a fourth-generation deformable convolutional network structure. DCNv4 (Deformed Convolutional Layer 4) replaces traditional fixed-size convolutions. This deformed convolutional layer adds a text contour geometric deformation offset to each sampling point of the convolutional kernel, enabling the convolutional kernel to adaptively adapt to the geometric deformation of word positions in the text sequence. In this embodiment, the text contour geometric deformation offset can dynamically adjust the receptive field according to the geometric deformation of the word sequence in the text (such as irregular structures like curved arrangement or local breaks), thereby more effectively capturing non-fixed-form local semantic features and solving the problem of insufficient capture of irregular text structures by fixed convolutional kernels. The first convolutional block includes a second convolutional layer and a normalization layer connected in sequence, used to further regularize the feature distribution. The attention block includes cascaded channel attention and spatial attention, which are used to weight and enhance the feature matrix in terms of channel and spatial dimensions. In some embodiments, the residual block also includes a second convolutional block, which includes a third convolutional layer and a normalization layer connected in sequence. The second convolutional block convolves the original input features of the residual block with the output features of the attention block, adjusting the number of channels of the original input features to match the number of channels of the backbone output features for residual stacking. The residual stacking block performs element-wise addition on the output features of the attention block and the output features of the second convolutional block, and performs non-linear activation on the stacked features through activation blocks to output the residual block feature matrix. In this embodiment, the residual block stacks the input matrix with the feature matrix after convolution. Convolution can set the feature output dimension, and the dimension of the output of each layer will gradually decrease, finally obtaining a low-dimensional local text feature matrix. In this embodiment, by processing the two residual blocks in sequence, the dimension of the word vector matrix is ​​gradually reduced, and the local features of the text are effectively extracted, finally outputting the local text feature matrix, thereby achieving the effect of dimensionality reduction, and at the same time, the original features are introduced in a timely manner to alleviate the network degradation problem.

[0025] In the aforementioned residual block, to enhance the model's attention to key semantic regions, this embodiment inputs the local text feature matrix output by the residual block into the attention block for channel-space attention weighting. This attention block consists of two sub-modules connected in series: Channel Attention (CA) and Spatial Attention (SA). Channel Attention extracts channel importance weights through global average pooling and max pooling, and weights the original feature channels, thereby strengthening text channel features that are highly relevant to fault descriptions. Spatial Attention generates a spatial weight map based on the results of channel pooling, focusing on the importance of different word positions in the text. This embodiment uses this dual attention mechanism to assign higher weights to important features in text semantics, enabling the model to focus on and capture key abnormal content in the deep semantics of the text, resulting in an attention-enhanced local feature matrix.

[0026] Next, since convolutional neural networks are primarily adept at capturing local features and have limited ability to model long-distance contextual dependencies in longer texts, this embodiment inputs the attention-enhanced local feature matrix into a Bidirectional Long Short-Term Memory (BiLSTM) network layer for bidirectional long-distance semantic modeling. Compared to standard recurrent neural networks, this BiLSTM network layer has bidirectional learning advantages and long-distance contextual feature learning capabilities, ensuring that the input features do not decay with network iterations. In this embodiment, the BiLSTM network unit includes an input gate, an output gate, and a forget gate. Each gate is equivalent to a fully connected layer, controlling the input, output, and whether the input vector is forgotten. The forward unit calculates the sequence from front to back, and the backward unit calculates the sequence from back to front. At each time step, the network compares the current input with the hidden state of the previous time step. The system combines the states, and through the control of the input gate, forget gate, and output gate, it determines the retention and updating of information, thereby effectively avoiding the gradient decay problem in long-distance dependencies. Finally, this embodiment combines the outputs of the forward and backward directions according to time steps to obtain the long-distance contextual semantic features of the text sequence in both the forward and backward directions. The long-distance contextual semantic features of the text are then input into the output layer, and the classification result is obtained by using the normalized exponential function method. Each element in the output vector represents the probability that the sample data belongs to a certain category. When the probability is greater than a certain threshold, the category is retained. When multiple categories are retained, the model predicts that the sample belongs to multiple categories.

[0027] Building upon the above, since the number of samples for different fault categories in power grid operation and maintenance text data often exhibits a severe imbalance (e.g., texts about oil leaks far outnumber those about winding deformation), this embodiment employs an improved Focal Loss (FL) function in the output stage to replace the standard cross-entropy loss function for adaptive class weight balancing. This improved Focal Loss function adds a Gaussian weight parameter to the standard Focal Loss, with the value decreasing as the number of class samples increases. Specifically, abnormal categories with relatively small sample sizes have larger weights, and categories with smaller data volumes have larger feature weights, while normal categories with larger data volumes have relatively smaller weights. By adjusting the Gaussian function parameter, the loss weight range for each category can be precisely controlled, allowing the model training to focus more on feature learning of small-sample abnormal categories, thus highlighting the role of small-class anomalies and preventing the model from ignoring rare but crucial anomaly patterns. Finally, the output layer uses the Softmax function to calculate the weight of each abnormal category. The predicted probability of each category is used, and the model is optimized by combining the improved focus loss function mentioned above. This yields a deep semantic feature vector that effectively reflects the semantics of power grid operation and maintenance text while taking into account class imbalance. In summary, this embodiment completes local feature residual extraction, deformable convolution adaptation, channel spatial attention weighting, bidirectional long-distance semantic modeling, and adaptive balancing of class weights through the end-to-end text feature extraction process described above. The output is a deep semantic anomaly feature vector that represents the deep semantic anomaly pattern of equipment operation status. This effectively solves the problems of class imbalance and semantic feature extraction, improves the accuracy and robustness of text anomaly detection, and provides a text modality basis for subsequent cross-modal anomaly correlation analysis.

[0028] S3. Extract pseudo-tags from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo-tags and video-level visual pseudo-tags.

[0029] In some implementations, the step of extracting pseudo-tags from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo-tags and video-level visual pseudo-tags includes: Each video sample in the power grid operation and maintenance inspection audio and video data is uniformly divided into several video segments to obtain the image features of the video segments. The power grid operation and maintenance inspection event categories are embedded into preset text prompt templates, and the category text features of each power grid operation and maintenance inspection event category are extracted. Calculate the normalized cosine similarity between the video segment image features and the category text features, and generate a segment-level score mask based on the normalized cosine similarity and a preset similarity probability threshold; Based on the segment-level score mask and the preset audio-video joint label, the visual event categories existing in each video segment are filtered out to obtain the segment-level visual pseudo-label for each video segment; The fragment-level visual pseudo-tags are traversed, and the fragment-level visual pseudo-tags are aggregated according to the power grid operation and maintenance inspection event categories to obtain video-level visual pseudo-tags.

[0030] Specifically, this embodiment extracts visual representative frames from the video data feature sequence in the power grid operation and maintenance inspection audio and video data to obtain video segment image features. Simultaneously, this embodiment predefines a set of abnormal event categories that may occur in power grid operation and maintenance, such as partial discharge, abnormal mechanical vibration, and oil leakage, and constructs a fixed-format text prompt template. This text prompt template embeds replaceable placeholders, such as a photo containing [category name]. For each abnormal event category, this embodiment replaces the category placeholders in the preset text prompt template with the name of each power grid operation and maintenance inspection event category, generating a corresponding text description. The text descriptions of all categories are then input into a pre-trained text encoder to extract the text feature vector of each category, obtaining the category text features of each power grid operation and maintenance inspection event category. The pre-trained text encoder... A bidirectional encoder representation model based on a transformer can be used. Then, for the image features of a certain video segment, this embodiment normalizes the video segment image features by dividing them by their own modulus (i.e., L2 norm) to obtain normalized video segment image features. Simultaneously, the text feature vectors of each category are normalized by dividing them by their modulus to obtain normalized category text features. Then, this embodiment calculates the cosine similarity between the normalized video segment image features and the normalized category text features. To ensure numerical stability, a soft maximization operation can be applied to the similarity score vector to obtain the normalized cosine similarity. At the same time, this embodiment sets a similarity threshold. If the normalized cosine similarity is not less than the similarity threshold, the corresponding position is output as 1; otherwise, it is output as 0. Thus, by setting a similarity threshold, high-confidence categories are filtered and a binary segment-level score mask is generated.

[0031] Each video sample is assigned a corresponding coarse-grained audio-video joint label, which only indicates whether various abnormal events exist in the entire video, without indicating the specific time of the event. In this embodiment, the audio-video joint label is multiplied element-wise with the above-mentioned segment-level score mask to filter out the visual event categories present in each video segment, generating the initial visual pseudo-label vector of the current segment. The position of 1 in the initial visual pseudo-label vector indicates that the segment may have this type of anomaly according to the image-text similarity, and the anomaly category also exists in the video-level label. The above process is repeated for all video segments to obtain the segment-level visual pseudo-label of each segment. The pseudo-labels of all segments are arranged in chronological order to form segment-level visual pseudo-labels. In this embodiment, all segment-level visual pseudo-labels are traversed and aggregated according to event category. If a certain category exists in at least one segment, it is included in the video-level visual pseudo-label. For example, for each anomaly category, if the pseudo-label value corresponding to at least one segment in all segments is 1, then the value of that category in the video-level pseudo-label is 1; otherwise, it is 0. This is how the video-level visual pseudo-label is obtained.

[0032] S4. Perform richness-aware weakly supervised parsing on the fragment-level visual pseudo-labels and the video-level visual pseudo-labels to obtain visual feature representations.

[0033] In some implementations, the step of performing richness-aware weakly supervised parsing on the fragment-level visual pseudo-tags and the video-level visual pseudo-tags to obtain visual feature representations includes: The pseudo-label category richness is obtained by calculating the ratio of the number of power grid operation and maintenance inspection event categories in each video segment to the total number of all power grid operation and maintenance inspection event categories based on the segment-level visual pseudo-labels. The pseudo-label segment richness is obtained by calculating the ratio of the number of video segments for a single power grid operation and maintenance inspection event category to the total number of video segments. Visual prediction is performed on the audio and video data of the power grid operation and maintenance inspection to obtain the visual prediction result. The visual prediction result, the richness of the pseudo-label category, and the richness of the pseudo-label segment are aligned to obtain the richness perception loss. The video-level visual pseudo-labels are used as supervision information, and a video-level basic loss is constructed based on the visual prediction results. Based on the richness perception loss and the video-level basic loss, forward propagation loss analysis is performed on the segment-level visual pseudo-labels to obtain the localization results of abnormally high loss segments. Based on the location results of the abnormally high loss segments, the segment-level visual pseudo-labels are reversed and corrected to obtain optimized pseudo-labels. Based on the optimized pseudo-labels, weakly supervised audiovisual analysis is performed on the power grid operation and maintenance inspection audio and video data to obtain visual feature representations.

[0034] Specifically, this embodiment calculates the ratio of the number of anomaly categories with a pseudo-label value of 1 in a segment to the total number of anomaly categories in the audio-video joint label based on the segment-level visual pseudo-labels, thus obtaining the category richness vector for all segments. The more anomaly categories a segment contains, the higher its category richness. Simultaneously, for each event category, this embodiment calculates the ratio of the number of segments containing that event category to the total number of video segments, thus obtaining the segment richness vector for all categories. For example, for the c-th anomaly category, its segment richness is defined as the ratio of the number of segments with a pseudo-label value of 1 in that category to the total number of segments T, thus obtaining the frequency of that anomaly on the timeline. Given the current fragment ratio, this embodiment calculates the fragment richness vector for all categories. Simultaneously, this embodiment performs visual prediction on the power grid operation and maintenance inspection audio and video data through a basic visual prediction network to obtain visual prediction results. From the visual prediction results, the predicted category richness and predicted fragment richness can be calculated separately in the same way. The predicted category richness is aligned with the pseudo-label category richness and the predicted fragment richness is aligned with the pseudo-label fragment richness through a binary cross-entropy loss function, thereby aligning the consistency between visual prediction and pseudo-label in terms of category richness and fragment richness. The sum of the two is the richness perception loss.

[0035] Next, this embodiment uses video-level visual pseudo-labels as supervision signals. Based on the video-level visual pseudo-labels and visual prediction results, a binary cross-entropy loss is calculated to obtain the visual prediction loss. The visual prediction loss, the audio prediction loss between audio modality prediction and audio labels, and the audio-video joint prediction loss between audio-video joint prediction and joint labels are weighted and summed to construct the video-level basic loss. The richness perception loss is then weighted and combined with the video-level basic loss to obtain the overall objective loss function used for training audiovisual parsing. This embodiment balances the contributions of the richness perception loss and the video-level basic loss by adjusting the weight parameters, enabling the model to perceive segment-level event distribution differences based on video-level supervision. Due to the initial... Pseudo-labels are generated by unsupervised or weakly supervised methods, inevitably containing noise. Therefore, this embodiment uses the overall objective loss function to calculate the binary cross-entropy loss value between the predicted probability of each category in each segment and the initial pseudo-label, obtaining the element-wise forward propagation loss matrix between the visual prediction result and the segment-level visual pseudo-labels. Each column in this element-wise forward propagation loss matrix represents the cross-entropy loss value of a specific event category across all segments, and its element value represents the model's fitting error on that segment and that category. Simultaneously, this embodiment sets the loss value corresponding to event categories that do not exist in the video to zero, ensuring that denoising is only performed on necessary event categories. Considering that video-level visual pseudo-labels have already indicated the possible event categories in the entire video, for video-level... For categories with a value of zero in the pseudo-labels (i.e., abnormal categories that do not exist in the entire video), the forward loss at the segment level is not meaningful for denoising. Therefore, for each event category, this embodiment selects the average of the first few minimum loss values ​​and multiplies it by a scaling factor as an adaptive threshold. The cross-entropy loss value of each segment is compared with this adaptive threshold to generate a binary mask vector matrix. This binary mask vector matrix is ​​then multiplied element-wise with the element-wise forward propagation loss matrix to mask the loss values ​​of irrelevant categories. For each abnormal category (i.e., each column), this embodiment extracts the forward loss value of that column across all segments and calculates the average loss of the K segments with the smallest loss value in that column, then multiplies it by a scaling factor (greater than 1) to obtain the loss value for that column. The denoising threshold is determined, and then the loss value of each segment is compared with the denoising threshold. If the loss value is greater than the denoising threshold, the initial pseudo-label of the segment in that category is considered to be incorrect and is marked as to be reversed; otherwise, it is retained, and the abnormal high loss segment localization result is obtained. Finally, the pseudo-labels corresponding to the marked abnormal segment positions are flipped according to the abnormal high loss segment localization result, and the erroneous pseudo-labels are corrected to the opposite state, that is, the values ​​of the corresponding reverse positions in the initial segment-level pseudo-label matrix are flipped (1 becomes 0, 0 becomes 1), and optimized pseudo-labels are obtained. This denoising process is based on the characteristic that the pre-trained model has a small loss on correctly labeled segments and a large loss on incorrectly labeled segments. It identifies and corrects noise labels through adaptive thresholding, thereby improving the quality of pseudo-labels.

[0036] Finally, this embodiment performs weakly supervised audio-visual analysis on power grid operation and maintenance inspection audio and video data based on the above-mentioned optimized pseudo-labels. Visual and audio features of the audio and video data are extracted through the basic network. Model training is carried out under the supervision of optimized pseudo-labels, enabling the model to learn to distinguish between normal and abnormal video segments without manual frame-by-frame annotation. The model outputs reliable segment-level audio and video anomaly feature representations and obtains visual feature representations. These visual feature representations characterize the spatiotemporal information of abnormal equipment images and sounds, providing an audio and video modal foundation for subsequent multimodal fusion.

[0037] Compared with existing technologies that rely on extensive manual annotation or simply use video-level labels for supervised learning, this embodiment automatically generates segment-level pseudo-labels through image-text similarity calculation, achieving weakly supervised activation with zero manual intervention. At the same time, it introduces a two-dimensional loss of category richness and segment richness, enabling the model to perceive the differences in the number of abnormal events in different segments and the differences in the distribution of different abnormal events in segments, thereby enhancing the model's sensitivity to sparse abnormal patterns.

[0038] S5. Perform multimodal cross-layer attention fusion on the deep semantic feature vector of the text and the visual feature representation to obtain a cross-modal joint feature representation.

[0039] In some implementations, the step of performing multimodal cross-layer attention fusion of the deep semantic feature vector of the text and the visual feature representation to obtain a cross-modal joint feature representation includes: The deep semantic feature vector of the text and the visual feature representation are respectively subjected to projection transformation to obtain semantic embedding sequence and visual embedding sequence of the same dimension; The semantic embedding sequence and the visual embedding sequence are concatenated in the time dimension to form a multimodal fusion sequence, and positional encoding information is superimposed on the multimodal fusion sequence to obtain a multimodal fusion embedding sequence. The degree of information interaction between different positions in the multimodal fusion embedding sequence is analyzed by a multi-head self-attention mechanism to obtain the multi-head self-attention matrix; Based on the multi-head self-attention matrix, a two-dimensional convolutional evolution is performed on the cross-layer attention distribution to obtain the cross-layer evolved attention matrix; The cross-layer attention information is dynamically fused based on the cross-layer evolutionary attention matrix and the multi-head self-attention matrix to obtain the cross-layer joint attention matrix; The multimodal fusion embedding sequence is nonlinearly transformed using the cross-layer joint attention matrix to obtain a cross-modal joint feature representation.

[0040] Specifically, in this embodiment, the deep semantic feature vectors and visual feature representations of the text are input into two independent linear projection layers. A learnable projection matrix maps them to a unified latent feature space, ensuring that heterogeneous modal data have the same feature dimension. Specifically, the text projection layer maps the text feature vectors to a semantic embedding sequence, and the visual projection layer maps the visual feature representations to a visual embedding sequence. Both are projected onto the same feature dimension space for subsequent cross-modal interaction. Next, this embodiment concatenates the semantic and visual embedding sequences along the time dimension to form a multimodal fusion sequence. Since the time steps of the semantic and visual features are different, to preserve the positional information of each element in the sequence, this embodiment superimposes positional encoding information onto the multimodal fusion sequence. The positional encoding is generated using sine and cosine functions, corresponding to even and odd positions respectively, to explicitly mark the absolute or relative position of each element in the sequence. After superimposing the positional encoding, a multimodal fusion embedding sequence with spatiotemporal position awareness is obtained. Then, the multimodal fusion embedding sequence... The multi-head self-attention mechanism is used for information interaction computation. Specifically, in this embodiment, the multimodal fusion embedding sequence is projected into a query sequence, a key sequence, and a value sequence through a learnable projection matrix. The query sequence represents the abnormal information needs of the current layer, the key sequence provides a global abnormal context reference, and the value sequence carries the abnormal feature information to be aggregated. To enhance the expressive power of the model, this embodiment splits the attention computation into multiple independent attention heads, each focusing on a different abnormal feature subspace. An attention score matrix is ​​calculated by the dot product operation of the query sequence and the key sequence. This attention score matrix determines how the model allocates attention resources, enabling each position to aggregate important abnormal information from the global context. Each element in the attention score matrix represents the correlation strength between the query position and the key position. The attention matrices of multiple attention heads are concatenated along the feature dimension to obtain a multi-head self-attention matrix, which reflects the degree of abnormal information interaction between different positions in the multimodal fusion embedding sequence.

[0041] In the standard Transformer architecture, attention maps are generated independently at each layer, and attention patterns between different layers are not explicitly shared. This leads to deeper layers gradually losing fine-grained local anomaly information captured by lower layers. To address this issue, this embodiment constructs a transport and convergence self-attention module. This module uses two-dimensional convolution operations to evolve the log-probability of the attention from the previous layer, treating attention as a two-dimensional feature map with multi-channel feature dimensions. By sliding the convolution kernel across the attention head dimension and the spatiotemporal dimension, the attention distribution of the previous layer adapts to the anomaly feature modeling requirements of the current layer, alleviating the mismatch problem between inter-layer anomaly feature distributions and obtaining cross-layer evolutionary attention. Specifically, in this embodiment, a learnable two-dimensional convolutional kernel is used to perform sliding convolutions in the height and width directions (i.e., sequence position dimensions) and the channel direction (i.e., attention head dimensions). Each parameter of the convolutional kernel is multiplied by the attention value at the corresponding position, summed, and then a learnable bias term is added to obtain the cross-layer evolutionary attention matrix. This two-dimensional convolutional operation can adjust the spatial distribution and channel weights of the previous layer's attention according to the modeling requirements of the current layer, so that its feature distribution matches that of the current layer, thereby mitigating the information transmission attenuation caused by differences in inter-layer feature distribution. Then, this embodiment uses a dynamic selection fusion strategy based on gating factors to dynamically fuse the cross-layer evolutionary attention matrix with the current layer's multi-head self-attention matrix. Specifically, this embodiment introduces learnable or preset weight parameters to multiply the cross-layer evolutionary attention matrix of the previous layer by the weight parameters to obtain the cross-layer evolutionary attention weighted matrix of the previous layer. Then, the cross-layer evolutionary attention matrix of the current layer is multiplied by the difference between the current layer and the weight parameters to obtain the cross-layer evolutionary attention weighted matrix of the current layer. The two cross-layer evolutionary attention weighted matrices are then added element-wise. A larger weight parameter emphasizes preserving the local details of the previous layer, while a smaller weight parameter emphasizes the global modeling features of the current layer. In this embodiment, the attention matrices of the previous and current layers are concatenated along the attention head dimension to obtain a new feature map with twice the number of channels. Then, a two-dimensional convolutional layer is used to locally redistribute the new feature map. The number of output channels of the convolutional kernel is the same as the number of original attention heads. The convolution operation can simultaneously consider the attention patterns of adjacent positions to achieve smooth fusion. Next, in this embodiment, the previous layer attention matrix is ​​linearly transformed through a learnable projection matrix, and then a gating factor matrix is ​​generated by the Sigmoid activation function. The element values ​​of the gating factor matrix are between 0 and 1. The final fusion result is equal to the gating factor matrix multiplied element by element by the previous layer attention matrix, plus 1 minus the difference of the gating factor matrix, and then multiplied element by element by the current layer attention matrix. The gating mechanism enables the model to dynamically determine the contribution ratio of the information of the previous layer and the current layer according to different positions.

[0042] Next, in this embodiment, the attention matrices of the previous layer and the current layer are added element by element to obtain a temporary matrix. This temporary matrix is ​​then subjected to global average pooling to compress it in the sequence length dimension, resulting in a global feature vector. The global feature vector is then upscaled to a higher dimension through a linear layer. Two attention weight vectors are then generated through two different linear projections. After Softmax normalization, the dynamic weights of the previous layer and the current layer are obtained. Finally, the fusion matrix is ​​equal to the weights of the previous layer multiplied by the previous layer attention matrix plus the weights of the current layer multiplied by the current layer attention matrix, thus obtaining a cross-layer joint attention matrix. This fusion process allows the attention distribution of the current layer to inherit the fine local anomaly information captured by the previous layer and combine the relatively global anomaly features modeled by the current layer, thereby enhancing the synergistic effect of attention information from different layers.

[0043] Then, in this embodiment, the cross-layer joint attention matrix is ​​divided by the square root of the query feature dimension to prevent the Softmax output from being too extreme due to excessively large values, resulting in a scaled cross-layer joint attention matrix. This stabilizes the gradient and improves training stability. In this embodiment, the scaled cross-layer joint attention matrix is ​​used to generate the final cross-modal attention map through the normalized exponential function Softmax. The attention map is then multiplied by the value matrix and mapped back to the original feature dimension through a linear transformation to obtain the output feature of the current encoder layer. This output feature is then input into the feedforward neural network, which contains two linear transformation layers. The first linear transformation layer expands the feature dimension to a higher intermediate dimension and then activates it through a linear rectified function. The second linear transformation layer then compresses the dimension back to the original hidden dimension. Residual connections and layer normalization are interspersed to stabilize training. After processing by multiple encoders, the cross-modal joint anomaly feature representation is finally obtained. This cross-modal joint anomaly feature representation realizes the deep alignment and correlation modeling of text semantic anomalies and audio-visual spatiotemporal anomalies in a unified feature space, and it represents the multimodal complementary information of device abnormal states.

[0044] S6. Perform semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representation to generate a comprehensive analysis result of operation and maintenance events, and perform anomaly detection based on the comprehensive analysis result of operation and maintenance events to generate a multimodal anomaly detection result of power grid operation and maintenance.

[0045] In some implementations, the step of performing semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representation to generate a comprehensive analysis result of the operation and maintenance event, and performing anomaly detection based on the comprehensive analysis result of the operation and maintenance event to generate a multimodal anomaly detection result for power grid operation and maintenance includes: Based on the cross-modal joint feature representation, the text semantic features and acoustic visual features are projected and embedded to obtain a multimodal collaborative embedding sequence; Based on the multimodal cooperative embedding sequence, semantic acoustic multimodal cooperative decoding is performed on the cross-modal anomaly semantic information to obtain the anomaly pattern probability distribution; Based on the probability distribution of the abnormal patterns, abnormal events are identified in the operating status of power equipment to obtain comprehensive analysis results of operation and maintenance events. Based on the comprehensive analysis results of the operation and maintenance events, the severity of the anomalies is quantitatively assessed to determine the risk level, and multimodal anomaly detection results for power grid operation and maintenance are generated.

[0046] Specifically, to fully utilize the complementary information of audio and text modalities, this embodiment achieves deep fusion of acoustic and semantic information in the decoder. The textual semantic features and acoustic-visual features in the cross-modal joint feature representation are dimensionally mapped through semantic projection layers and acoustic projection layers, respectively, so that the two modal features are projected onto a common subspace of the same dimension to match the feature dimension of the decoder. Then, the acoustic embedding and semantic embedding are concatenated along the temporal dimension and superimposed with positional encoding to retain the positional information of the input sequence, generating a multimodal co-embedding sequence containing textual anomaly semantic descriptions and acoustic-visual anomaly state information. This positional encoding enables the model to explicitly perceive the temporal dependencies of different modal features, improving the alignment effect of cross-modal information. Next, this embodiment inputs the multimodal co-embedding sequence into the acoustic and semantic information co-decoder. This decoder employs a multi-head attention module and a feedforward neural network module, and uses pre-normalization and residual connection structures to ensure gradient stability and training convergence. During the decoding process, the semantic anomaly features interact with the acoustic anomaly features provided by the encoder through the multi-head attention mechanism to ensure full utilization of low-level anomaly information in the acoustic signal during anomaly pattern recognition; simultaneously, semantic anomaly features are fused. The semantic anomaly features predicted at previous time steps enable the decoder to model long-range contextual dependencies during anomaly pattern recognition. Acoustic anomaly features dynamically update their own information under the same attention mechanism, optimizing the temporal representation of acoustic features through the guidance of semantic anomaly information to better meet the decoder's anomaly pattern modeling needs, thereby reducing acoustic ambiguity in the anomaly recognition process. A unified multi-head attention mechanism achieves deep fusion of acoustic and semantic information, allowing the decoder to fully utilize the relevant information of both modalities when reasoning about the current anomaly semantic state. It should be noted that semantic features interact in parallel with acoustic features and historically predicted semantic features through the multi-head attention mechanism. On the one hand, they interact with the acoustic features provided by the encoder to fully utilize the low-level acoustic information of the speech signal; on the other hand, they fuse with the semantic features predicted at previous time steps to model long-range contextual dependencies. Simultaneously, this embodiment introduces an internal self-interaction mechanism for acoustic features, enabling them to dynamically adapt to the semantic information of the text during decoding, rather than being statically provided with fixed representations by the encoder. The temporal representation of acoustic features is optimized through the guidance of semantic information to better meet the decoder's language modeling needs, thereby reducing acoustic ambiguity in the anomaly recognition process.

[0047] To reduce computational complexity, this embodiment only performs full feedforward network and self-attention updates on semantic features during the decoding process. The multimodal collaborative embedding sequence only participates in the calculation as a key and value in the multi-head attention mechanism, and is not updated in the feedforward neural network or self-attention. Although acoustic features no longer directly participate in the update calculation, they still serve as background information to assist semantic reasoning in the multimodal attention calculation, allowing the decoder to retain the guiding role of acoustic information during the prediction process. This optimization significantly reduces the overall time complexity and effectively reduces the computational burden while maintaining the ability of joint speech-text reasoning.

[0048] It should be noted that during the model training phase, this embodiment uses either the connection-time classification loss function or the cross-entropy loss function to train and optimize the decoder structure. A joint loss strategy is employed, utilizing both connection-time classification loss for frame-level constraints and cross-entropy loss for decoding optimization. In this embodiment, the connection-time classification loss function flexibly handles the misalignment of input and output sequence lengths by introducing whitespace and repetition mechanisms, eliminating the need for explicit frame-level alignment annotations beforehand. For the transformer decoder using an attention-based encoding / decoding structure, this embodiment uses the cross-entropy loss function for training. This loss function measures the difference between the model's output probability distribution and the true target probability distribution, progressively guiding the model to learn the predicted probability distribution at each time step. Simultaneously, a joint optimization strategy is used, weighting the connection-time classification loss and the cross-entropy loss. By adjusting hyperparameters to balance the contributions of the two losses, the connection-time classification loss is fully utilized for frame-level constraints, while cross-entropy is used for autoregressive decoding optimization, thereby improving the robustness and accuracy of anomaly detection.

[0049] Then, this embodiment performs anomaly probability quantization calculation on the decoded output, generating anomaly occurrence probability distributions for different device components at different time steps. Anomaly levels are determined based on anomaly probability thresholds and preset anomaly judgment thresholds. When the probability exceeds the preset anomaly judgment threshold, it is determined to be an abnormal state; otherwise, it is determined to be a normal state. Furthermore, the spatiotemporal locations where the anomaly probability exceeds the preset anomaly judgment threshold are back-located, associated with fault description statements in the text modality and abnormal segment frames in the audio / video modality, achieving multimodal tracing of anomaly evidence. Anomaly spatial location information is obtained to determine the device location or area where the anomaly occurred. This embodiment integrates the above information into a comprehensive analysis result of the operation and maintenance event. This comprehensive analysis result includes the category, confidence level, time, and spatial information of all detected anomaly events. The preset anomaly risk level is queried based on the anomaly category label in the comprehensive analysis result of the operation and maintenance event. A mapping table maps different anomaly categories to corresponding risk levels. Anomaly events within the same risk level are ranked according to their anomaly confidence scores; higher confidence scores indicate more severe anomalies. The urgency of the anomaly is assessed by combining the difference between the anomaly's timestamp and the current system time; the more recent the anomaly, the higher the urgency. The impact range of the anomaly is assessed based on its spatial location information; anomalies involving critical equipment have a larger impact range. A comprehensive risk score is calculated for each anomaly event, considering all four dimensions: risk level, confidence score, urgency, and impact range. The anomaly events are then ranked from highest to lowest according to their comprehensive risk scores to generate a multimodal anomaly detection result for power grid operation and maintenance. This result includes the risk level, comprehensive risk score, and urgency ranking of each anomaly event. This result can be directly pushed to the operation and maintenance alarm system, allowing maintenance personnel to quickly locate equipment faults and take appropriate measures.

[0050] In summary, this embodiment enables deep models to retain low-level fine-grained local anomaly information through the dynamic evolution and selective fusion of cross-layer attention information, thereby enhancing the collaborative modeling capability of multi-scale anomaly features. At the same time, it introduces an internal self-interaction mechanism within acoustic features, allowing acoustic features to dynamically adapt to semantic information during the decoding process. Semantic information also guides the temporal optimization of acoustic features, achieving true bidirectional deep fusion rather than simple splicing.

[0051] This invention provides a method for detecting anomalies in multimodal data of power grid operation and maintenance. The method includes: preprocessing multimodal operation and maintenance data collected in a power grid operation and maintenance scenario to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data; extracting features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features, and performing class imbalance correction on the long-distance contextual semantic features to obtain a deep semantic feature vector; extracting pseudo-labels from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo-labels and video-level visual pseudo-labels; performing richness-aware weakly supervised parsing on the segment-level visual pseudo-labels and video-level visual pseudo-labels to obtain visual feature representations; performing multimodal cross-layer attention fusion on the deep semantic feature vectors of the text and the visual feature representations to obtain cross-modal joint feature representations; performing semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representations to generate a comprehensive analysis result of operation and maintenance events, and performing anomaly detection based on the comprehensive analysis result of operation and maintenance events to generate a multimodal anomaly detection result for power grid operation and maintenance. Compared with existing technologies, this method achieves deep fusion of multimodal data and accurate quantitative assessment of abnormal states in power grid operation and maintenance scenarios through the organic synergy of deep semantic extraction of text and adaptive category equalization, thereby improving the accuracy and robustness of anomaly detection and providing intelligent decision support for the safe and stable operation of the power grid.

[0052] It should be noted that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0053] In one embodiment, such as Figure 2 As shown in the figure, this embodiment of the invention provides a multimodal data anomaly detection system for power grid operation and maintenance, the system comprising: The data acquisition module 101 is used to preprocess the multimodal operation and maintenance data collected in the power grid operation and maintenance scenario to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data. The feature extraction module 102 is used to extract features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features of the text, and to perform class imbalance correction on the long-distance contextual semantic features of the text to obtain deep semantic feature vectors of the text. The tag extraction module 103 is used to extract pseudo tags from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo tags and video-level visual pseudo tags. The visual analysis module 104 is used to perform richness-aware weakly supervised parsing on the fragment-level visual pseudo-labels and the video-level visual pseudo-labels to obtain visual feature representations. The cross-layer fusion module 105 is used to perform multimodal cross-layer attention fusion of the deep semantic feature vector of the text and the visual feature representation to obtain a cross-modal joint feature representation; The anomaly detection module 106 is used to perform semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representation, generate a comprehensive analysis result of operation and maintenance events, and perform anomaly detection based on the comprehensive analysis result of operation and maintenance events to generate a multimodal anomaly detection result of power grid operation and maintenance.

[0054] In some implementations, the feature extraction module is specifically used for: The power grid operation and maintenance inspection text data is segmented into words to obtain a word sequence, and the word sequence is then subjected to distributed word vector embedding processing to obtain a word vector matrix; The word vector matrix is ​​subjected to residual block hierarchical extraction and deformed convolution processing using a convolutional layer composed of two residual blocks to obtain a local text feature matrix; Based on the local text feature matrix, channel-space attention weighting is applied to the key semantic regions of the text using an attention mechanism to obtain an attention-enhanced local feature matrix. Bidirectional cyclic semantic modeling is performed on the long-distance contextual dependencies of the attention-enhanced local feature matrix to obtain long-distance contextual semantic features of the text. Based on the long-distance contextual semantic features of the text, the deep semantic feature vector of the text is obtained by optimizing the Gaussian weight adaptive loss for the imbalance of the number of abnormal category samples by improving the focus loss constraint.

[0055] In some implementations, each residual block includes a residual convolutional block, a deformable convolutional block, a first convolutional unifying block, an attention block, a residual stacking block, and an activation block connected in sequence; wherein, the residual convolutional block includes a first convolutional layer, a normalization layer, and an activation layer connected in sequence; the deformable convolutional block includes a deformable convolutional layer, a normalization layer, and an activation layer connected in sequence; the first convolutional unifying block includes a second convolutional layer and a normalization layer connected in sequence; the deformable convolutional block adopts a fourth-generation deformable convolutional network structure, and the deformable convolutional layer adds a text contour geometric deformation offset to each sampling point of the convolutional kernel.

[0056] For specific limitations regarding a multimodal data anomaly detection system for power grid operation and maintenance, please refer to the above-described limitations regarding a multimodal data anomaly detection method for power grid operation and maintenance, which will not be repeated here. Those skilled in the art will recognize that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0057] This invention provides a multimodal data anomaly detection system for power grid operation and maintenance. The system preprocesses multimodal operation and maintenance data collected in power grid operation and maintenance scenarios through a data acquisition module to obtain power grid operation and maintenance inspection text data and audio / video data. A feature extraction module extracts features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features, and performs class imbalance correction on these features to obtain a deep semantic feature vector. A label extraction module extracts pseudo-labels from the power grid operation and maintenance inspection audio / video data to obtain segment-level visual pseudo-labels and video-level visual pseudo-labels. A visual analysis module performs richness-aware weakly supervised parsing on the segment-level and video-level visual pseudo-labels to obtain visual feature representations. A cross-layer fusion module performs multimodal cross-layer attention fusion on the deep semantic feature vectors and visual feature representations to obtain cross-modal joint feature representations. An anomaly detection module performs semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representations to generate a comprehensive analysis result of operation and maintenance events, and performs anomaly detection based on the comprehensive analysis result to generate a multimodal anomaly detection result for power grid operation and maintenance. Compared with existing technologies, this system achieves deep fusion of multimodal data and accurate quantitative assessment of abnormal states in power grid operation and maintenance scenarios through the organic synergy of technologies such as deep semantic extraction of text and adaptive category balance. This improves the accuracy and robustness of anomaly detection and provides intelligent decision support for the safe and stable operation of the power grid.

[0058] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A method for detecting anomalies in multimodal data during power grid operation and maintenance, characterized in that, Includes the following steps: The multimodal operation and maintenance data collected in the power grid operation and maintenance scenario are preprocessed to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data; Feature extraction is performed on the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features of the text, and class imbalance correction is performed on the long-distance contextual semantic features of the text to obtain deep semantic feature vectors of the text. Pseudo-tags are extracted from the audio and video data of the power grid operation and maintenance inspection to obtain segment-level visual pseudo-tags and video-level visual pseudo-tags; The fragment-level visual pseudo-labels and the video-level visual pseudo-labels are subjected to richness-aware weakly supervised parsing to obtain visual feature representations; The deep semantic feature vector of the text and the visual feature representation are fused through multimodal cross-layer attention to obtain a cross-modal joint feature representation; Semantic-acoustic multimodal collaborative decoding is performed on the cross-modal joint feature representation to generate comprehensive analysis results of operation and maintenance events. Anomaly detection is then performed based on the comprehensive analysis results of operation and maintenance events to generate multimodal anomaly detection results for power grid operation and maintenance.

2. The method for detecting anomalies in multimodal data of power grid operation and maintenance as described in claim 1, characterized in that, The steps of extracting features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features, and performing class imbalance correction on the long-distance contextual semantic features to obtain a deep semantic feature vector of the text include: The power grid operation and maintenance inspection text data is segmented into words to obtain a word sequence, and the word sequence is then subjected to distributed word vector embedding processing to obtain a word vector matrix; The word vector matrix is ​​subjected to residual block hierarchical extraction and deformed convolution processing using a convolutional layer composed of two residual blocks to obtain a local text feature matrix; Based on the local text feature matrix, channel-space attention weighting processing is performed on the key semantic regions of the text through an attention mechanism to obtain an attention-enhanced local feature matrix. Bidirectional cyclic semantic modeling is performed on the long-distance contextual dependencies of the attention-enhanced local feature matrix to obtain long-distance contextual semantic features of the text. Based on the long-distance contextual semantic features of the text, the deep semantic feature vector of the text is obtained by optimizing the Gaussian weight adaptive loss for the imbalance of the number of abnormal category samples by improving the focus loss constraint.

3. The method for detecting anomalies in multimodal data of power grid operation and maintenance as described in claim 2, characterized in that: Each residual block includes a residual convolution block, a deformed convolution block, a first convolution block, an attention block, a residual stacking block, and an activation block connected in sequence. The residual convolutional block comprises a first convolutional layer, a normalization layer, and an activation layer connected in sequence; the deformable convolutional block comprises a deformable convolutional layer, a normalization layer, and an activation layer connected in sequence; the first convolutional normalization block comprises a second convolutional layer and a normalization layer connected in sequence; the deformable convolutional block adopts a fourth-generation deformable convolutional network structure, and the deformable convolutional layer adds a text contour geometric deformation offset to each sampling point of the convolutional kernel.

4. The method for detecting anomalies in multimodal data of power grid operation and maintenance as described in claim 1, characterized in that, The step of extracting pseudo-tags from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo-tags and video-level visual pseudo-tags includes: Each video sample in the power grid operation and maintenance inspection audio and video data is uniformly divided into several video segments to obtain the image features of the video segments. The power grid operation and maintenance inspection event categories are embedded into preset text prompt templates, and the category text features of each power grid operation and maintenance inspection event category are extracted. Calculate the normalized cosine similarity between the video segment image features and the category text features, and generate a segment-level score mask based on the normalized cosine similarity and a preset similarity probability threshold; Based on the segment-level score mask and the preset audio-video joint label, the visual event categories existing in each video segment are filtered out to obtain the segment-level visual pseudo-label for each video segment; The fragment-level visual pseudo-tags are traversed, and the fragment-level visual pseudo-tags are aggregated according to the power grid operation and maintenance inspection event categories to obtain video-level visual pseudo-tags.

5. The method for detecting anomalies in multimodal data of power grid operation and maintenance as described in claim 4, characterized in that, The step of performing richness-aware weakly supervised parsing on the fragment-level visual pseudo-tags and the video-level visual pseudo-tags to obtain visual feature representations includes: The pseudo-label category richness is obtained by calculating the ratio of the number of power grid operation and maintenance inspection event categories in each video segment to the total number of all power grid operation and maintenance inspection event categories based on the segment-level visual pseudo-labels. The pseudo-label segment richness is obtained by calculating the ratio of the number of video segments for a single power grid operation and maintenance inspection event category to the total number of video segments. Visual prediction is performed on the audio and video data of the power grid operation and maintenance inspection to obtain the visual prediction result. The visual prediction result, the richness of the pseudo-label category, and the richness of the pseudo-label segment are aligned to obtain the richness perception loss. The video-level visual pseudo-labels are used as supervision information, and a video-level basic loss is constructed based on the visual prediction results. Based on the richness perception loss and the video-level basic loss, forward propagation loss analysis is performed on the segment-level visual pseudo-labels to obtain the localization results of abnormally high loss segments. Based on the location results of the abnormally high loss segments, the segment-level visual pseudo-labels are reversed and corrected to obtain optimized pseudo-labels. Based on the optimized pseudo-labels, weakly supervised audiovisual analysis is performed on the power grid operation and maintenance inspection audio and video data to obtain visual feature representations.

6. The method for detecting anomalies in multimodal data of power grid operation and maintenance as described in claim 1, characterized in that, The step of performing multimodal cross-layer attention fusion of the deep semantic feature vector of the text and the visual feature representation to obtain a cross-modal joint feature representation includes: The deep semantic feature vector of the text and the visual feature representation are respectively subjected to projection transformation to obtain semantic embedding sequence and visual embedding sequence of the same dimension; The semantic embedding sequence and the visual embedding sequence are concatenated in the time dimension to form a multimodal fusion sequence, and positional encoding information is superimposed on the multimodal fusion sequence to obtain a multimodal fusion embedding sequence. The degree of information interaction between different positions in the multimodal fusion embedding sequence is analyzed by a multi-head self-attention mechanism to obtain the multi-head self-attention matrix; Based on the multi-head self-attention matrix, a two-dimensional convolutional evolution is performed on the cross-layer attention distribution to obtain the cross-layer evolved attention matrix; The cross-layer attention information is dynamically fused based on the cross-layer evolutionary attention matrix and the multi-head self-attention matrix to obtain the cross-layer joint attention matrix; The multimodal fusion embedding sequence is nonlinearly transformed using the cross-layer joint attention matrix to obtain a cross-modal joint feature representation.

7. The method for detecting anomalies in multimodal data of power grid operation and maintenance as described in claim 1, characterized in that, The steps of performing semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representation to generate a comprehensive analysis result of operation and maintenance events, and performing anomaly detection based on the comprehensive analysis result of operation and maintenance events to generate a multimodal anomaly detection result for power grid operation and maintenance include: Based on the cross-modal joint feature representation, the text semantic features and acoustic visual features are projected and embedded to obtain a multimodal collaborative embedding sequence; Based on the multimodal cooperative embedding sequence, semantic acoustic multimodal cooperative decoding is performed on the cross-modal anomaly semantic information to obtain the anomaly pattern probability distribution; Based on the probability distribution of the abnormal patterns, abnormal events are identified in the operating status of power equipment to obtain comprehensive analysis results of operation and maintenance events. Based on the comprehensive analysis results of the operation and maintenance events, the severity of the anomalies is quantitatively assessed to determine the risk level, and multimodal anomaly detection results for power grid operation and maintenance are generated.

8. A multimodal data anomaly detection system for power grid operation and maintenance, characterized in that, The system includes: The data acquisition module is used to preprocess the multimodal operation and maintenance data collected in the power grid operation and maintenance scenario to obtain power grid operation and maintenance inspection text data and power grid operation and maintenance inspection audio and video data. The feature extraction module is used to extract features from the power grid operation and maintenance inspection text data to obtain long-distance contextual semantic features of the text, and to perform class imbalance correction on the long-distance contextual semantic features of the text to obtain deep semantic feature vectors of the text. The tag extraction module is used to extract pseudo tags from the power grid operation and maintenance inspection audio and video data to obtain segment-level visual pseudo tags and video-level visual pseudo tags. The visual analysis module is used to perform richness-aware weakly supervised parsing on the fragment-level visual pseudo-labels and the video-level visual pseudo-labels to obtain visual feature representations. The cross-layer fusion module is used to perform multimodal cross-layer attention fusion of the deep semantic feature vector of the text and the visual feature representation to obtain a cross-modal joint feature representation; The anomaly detection module is used to perform semantic-acoustic multimodal collaborative decoding on the cross-modal joint feature representation, generate comprehensive analysis results of operation and maintenance events, and perform anomaly detection based on the comprehensive analysis results of operation and maintenance events to generate multimodal anomaly detection results for power grid operation and maintenance.

9. The power grid operation and maintenance multimodal data anomaly detection system as described in claim 8, characterized in that, The feature extraction module is specifically used for: The power grid operation and maintenance inspection text data is segmented into words to obtain a word sequence, and the word sequence is then subjected to distributed word vector embedding processing to obtain a word vector matrix; The word vector matrix is ​​subjected to residual block hierarchical extraction and deformed convolution processing using a convolutional layer composed of two residual blocks to obtain a local text feature matrix; Based on the local text feature matrix, channel-space attention weighting processing is performed on the key semantic regions of the text through an attention mechanism to obtain an attention-enhanced local feature matrix. Bidirectional cyclic semantic modeling is performed on the long-distance contextual dependencies of the attention-enhanced local feature matrix to obtain long-distance contextual semantic features of the text. Based on the long-distance contextual semantic features of the text, the deep semantic feature vector of the text is obtained by optimizing the Gaussian weight adaptive loss for the imbalance of the number of abnormal category samples by improving the focus loss constraint.

10. The power grid operation and maintenance multimodal data anomaly detection system as described in claim 9, characterized in that: Each residual block includes a residual convolution block, a deformed convolution block, a first convolution block, an attention block, a residual stacking block, and an activation block connected in sequence. The residual convolutional block comprises a first convolutional layer, a normalization layer, and an activation layer connected in sequence; the deformable convolutional block comprises a deformable convolutional layer, a normalization layer, and an activation layer connected in sequence; the first convolutional normalization block comprises a second convolutional layer and a normalization layer connected in sequence; the deformable convolutional block adopts a fourth-generation deformable convolutional network structure, and the deformable convolutional layer adds a text contour geometric deformation offset to each sampling point of the convolutional kernel.

Images