Video behavior identification method and system based on transformer substation monitoring

Abstract

The invention discloses a video behavior recognition method and system based on transformer substation monitoring, and relates to the technical field of behavior recognition, and the method comprises the steps: collecting a video monitored by a transformer substation, and carrying out the extraction of a video frame sequence; inputting the extracted video frame sequence into a key frame extraction module based on spatial-temporal characteristics for key frame screening; the key frames obtained through screening are input into a behavior classification decoder, multi-level feature interaction and dynamic weight distribution are conducted on the key frames, and behavior classification is conducted by combining a self-attention mechanism and a grouping decoding strategy and capturing the relevance of key frame time; and finally outputting a behavior recognition result. According to the method, an end-to-end video behavior recognition framework is constructed, so that the dependence of a traditional method on a static image is avoided; a key frame extraction method based on spatio-temporal characteristics is designed, so that the video reasoning efficiency is remarkably improved; a high-definition self-adaptive visual encoder is innovatively provided, and fine-grained behavior characteristics can be effectively captured.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and more particularly to a video behavior recognition method and system based on substation monitoring. Background Art

[0002] As a vital infrastructure in modern society, the safe and stable operation of power systems is directly linked to the normal functioning of society's production and daily life. Substations, as core components of power systems, perform crucial functions such as voltage conversion and current distribution, making their operational safety paramount. However, substations operate in complex environments, and the operational behavior of personnel directly impacts the safety of equipment. Statistics show that safety incidents caused by illegal or abnormal operator behavior are common, posing a serious threat not only to the safety of operators but also to the stable operation of the power system.

[0003] Traditional methods for identifying power personnel behavior primarily rely on machine learning algorithms such as support vector machines (SVMs) and multi-layer perceptrons (MLPs). These methods process static images and extract features such as color, texture, and shape for classification. However, these methods have significant limitations. First, static images cannot capture the temporal dynamics of behavior, resulting in insufficient fine-grained behavior recognition. Second, traditional methods experience significant degradation in recognition accuracy in complex scenarios (such as those affected by changing lighting and weather), making them difficult to meet the demands of real-world video surveillance scenarios.

[0004] In recent years, with the development of deep learning technology, video-based behavior recognition methods have gradually become a research hotspot. For example, Wang et al. proposed a lightweight detection model based on an improved YOLOv4. By introducing MobileNetV3 and the efficient channel attention mechanism (ECA), they significantly improved the accuracy and real-time performance of detecting abnormal behavior of substation personnel. Furthermore, Chen et al. used an improved Faster R-CNN algorithm combined with Retinex image enhancement technology to achieve high-precision recognition of helmet wearing, with a mean average precision (mAP) of 94.3%. Although these methods have made significant progress in specific tasks, they still suffer from slow detection speed and insufficient real-time performance. Moreover, they often focus on a single behavior (such as helmet detection) and fail to fully cover the diverse behaviors in substation scenarios.

[0005] Therefore, how to propose a video behavior recognition method and system based on substation monitoring, how to avoid the traditional method's reliance on static images, and significantly improve the efficiency of video reasoning are issues that technical personnel in this field urgently need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a video behavior recognition method and system based on substation monitoring, which effectively avoids the traditional method's reliance on static images and significantly improves the efficiency of video reasoning. In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A video behavior recognition method based on substation monitoring, comprising:

[0008] Collect substation monitoring videos and extract video frame sequences;

[0009] The extracted video frame sequence is input into the key frame extraction module based on spatiotemporal features for key frame screening;

[0010] The filtered key frames are input into the behavior classification decoder, which performs multi-level feature interaction and dynamic weight assignment on the key frames. The self-attention mechanism and group decoding strategy are combined to capture the correlation of key frame time and perform behavior classification.

[0011] Finally, the behavior recognition results are output.

[0012] Optionally, the key frame extraction module based on spatiotemporal features includes:

[0013] Use 3D convolutional neural network to extract spatiotemporal features of video frame sequences;

[0014] Calculate the feature difference between frames of the extracted spatiotemporal features to obtain a feature difference sequence;

[0015] The key frame screening method based on adaptive threshold is used to screen the key frames of feature difference sequences;

[0016] The K-means clustering algorithm is used to remove redundant frames with similar content in key frames.

[0017] Optionally, the extracting spatiotemporal features of the video frame sequence using a 3D convolutional neural network includes:

[0018] Given an input video frame sequence V = {v1, v2, ..., v T}, where v t Represents the t-th frame image, and the convolution operation of 3D-CNN is expressed as:

[0019]

[0020] Among them, F i，j，k is the output feature map, W m，n，p is the 3D convolution kernel, b is the bias term, and M×N×P is the size of the convolution kernel;

[0021] Through multi-layer 3D convolution and pooling operations, the spatiotemporal feature representation of the video is finally obtained F = {f1, f2, ..., f t}, where f t Represents the spatiotemporal feature vector of the t-th frame.

[0022] Optionally, performing inter-frame feature difference calculation on the extracted spatiotemporal features to obtain a feature difference sequence includes:

[0023] For the spatiotemporal feature sequence F, define the feature difference D between the t-th frame and the t-1-th frame t for:

[0024] D t =||f t -f t-1 || 2 ;

[0025] Among them, ||·||2 represents the Euclidean distance. By calculating the feature differences of all adjacent frames, the difference sequence D={D2, D3, ..., D T}.

[0026] Optionally, the key frame screening method based on the adaptive threshold for screening key frames of the feature difference sequence includes:

[0027] Based on the characteristic difference sequence D, calculate the mean μ of the difference sequence D and standard deviation σ D :

[0028]

[0029] Set the adaptive threshold τ to:

[0030] τ=μ D +α·σ D ;

[0031] Among them, α is an adjustable parameter used to control the strictness of key frame screening. For each frame, if its difference value D t If it is greater than the threshold τ, it is marked as a candidate key frame.

[0032] Optionally, the step of removing redundant frames with similar content from key frames using a K-means clustering algorithm includes:

[0033] The feature vectors of all candidate key frames {fk1, fk2, ..., fk K} as input, where K is the number of candidate key frames;

[0034] Use the K-means algorithm to cluster the feature vectors into C clusters, where C is the preset number of cluster centers;

[0035] For each cluster, the frame closest to the cluster center is selected as the final keyframe;

[0036] Finally, the optimized key frame set K = {k1, k2, ..., k C}, where C<<T.

[0037] Optionally, the behavior classification decoder includes: inputting the spatiotemporal features of the key frames obtained by the spatiotemporal feature extractor and the behavior category query vector composed of learnable parameters, and generating the final classification output through a feature interaction module, a dynamic weight allocation module, a nonlinear mapping module and an output generation module in sequence.

[0038] Optionally, the feature interaction module calculates the similarity between query vectors, dynamically assigns weights, and extracts global features; the dynamic weight assignment module interacts the query vector with the spatiotemporal feature sequence, dynamically assigns weights, and captures the spatial correlation between key frames; the nonlinear mapping module performs nonlinear transformation on the features through a multi-layer perceptron to enhance the expressiveness of the features; and the output generation module maps high-dimensional features to the probability distribution of behavior categories through a pooling operation based on the nonlinear mapping module to generate the final classification output.

[0039] Optionally, it also includes: adding a group decoding strategy, dividing the behavior categories into several groups through the group decoding strategy, each group shares a query vector, and generating outputs of multiple categories through affine transformation and pooling operations, reducing the computational complexity to linear.

[0040] Optionally, a video behavior recognition system based on substation monitoring includes:

[0041] Acquisition module: used to collect substation monitoring videos and extract video frame sequences;

[0042] Key frame screening module: used to input the extracted video frame sequence into the key frame extraction module based on spatiotemporal features for key frame screening;

[0043] Behavior classification module: This module inputs the filtered keyframes into the behavior classification decoder, performs multi-level feature interaction and dynamic weight assignment on the keyframes, and combines the self-attention mechanism with the group decoding strategy to capture the temporal correlation of the keyframes and perform behavior classification.

[0044] Output module: used to finally output behavior recognition results.

[0045] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for video behavior recognition based on substation monitoring, which has the following beneficial effects:

[0046] A video behavior recognition method based on substation monitoring involves: acquiring substation monitoring videos and extracting video frame sequences; inputting the extracted video frame sequences into a keyframe extraction module based on spatiotemporal features to filter keyframes; inputting the filtered keyframes into a behavior classification decoder, performing multi-level feature interaction and dynamic weight assignment on the keyframes, and combining a self-attention mechanism with a group decoding strategy to capture the temporal correlation of the keyframes for behavior classification; and finally outputting behavior recognition results. This paper proposes a behavior recognition method for substation video monitoring scenarios. By constructing an end-to-end video behavior recognition framework, this method directly processes video data, avoiding the reliance on static images in traditional methods. Furthermore, a keyframe extraction method based on spatiotemporal features is designed, significantly improving video inference efficiency. Furthermore, an innovative high-definition adaptive visual encoder is proposed to effectively capture fine-grained behavior features. Experimental results demonstrate that the proposed algorithm achieves an overall recognition rate of 98.1% on a substation work scene video dataset, significantly improving both recognition accuracy and processing speed compared to traditional methods, validating its effectiveness and engineering value in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0048] Figure 1 A flowchart of a video behavior recognition method based on substation monitoring provided by the present invention.

[0049] Figure 2 This is a schematic diagram of the structure of the spatiotemporal feature encoder provided by the present invention.

[0050] Figure 3 This is a schematic diagram of the decoder structure provided by the present invention.

[0051] Figure 4 Schematic diagram of the behavior recognition accuracy mixing matrix provided by the present invention.

[0052] Figure 5 This is a schematic diagram of the power personnel behavior recognition effect provided by the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The embodiment of the present invention discloses a video behavior recognition method based on substation monitoring, comprising:

[0055] Collect substation monitoring videos and extract video frame sequences;

[0056] The extracted video frame sequence is input into the key frame extraction module based on spatiotemporal features for key frame screening;

[0057] The filtered key frames are input into the behavior classification decoder, which performs multi-level feature interaction and dynamic weight assignment on the key frames. The self-attention mechanism and group decoding strategy are combined to capture the correlation of key frame time and perform behavior classification.

[0058] Finally, the behavior recognition results are output.

[0059] Furthermore, the key frame extraction module based on spatiotemporal features includes:

[0060] Use 3D convolutional neural network to extract spatiotemporal features of video frame sequences;

[0061] Calculate the feature difference between frames of the extracted spatiotemporal features to obtain a feature difference sequence;

[0062] The key frame screening method based on adaptive threshold is used to screen the key frames of feature difference sequences;

[0063] The K-means clustering algorithm is used to remove redundant frames with similar content in key frames.

[0064] Furthermore, the extraction of spatiotemporal features of a video frame sequence using a 3D convolutional neural network includes:

[0065] Given an input video frame sequence V = {v1, v2, ..., v T}, where v t Represents the t-th frame image, and the convolution operation of 3D-CNN is expressed as:

[0066]

[0067] Among them, F i，j，k is the output feature map, W m，n，p is the 3D convolution kernel, b is the bias term, and M×N×P is the size of the convolution kernel;

[0068] Through multi-layer 3D convolution and pooling operations, the spatiotemporal feature representation of the video is finally obtained F = {f1, f2, ..., f t}, where f t Represents the spatiotemporal feature vector of the t-th frame.

[0069] Furthermore, the step of calculating the inter-frame feature differences of the extracted spatiotemporal features to obtain a feature difference sequence includes:

[0070] For the spatiotemporal feature sequence F, define the feature difference D between the t-th frame and the t-1-th frame t for:

[0071] D t =||f t -f t-1 ||2;

[0072] Among them, ||·||2 represents the Euclidean distance. By calculating the feature differences of all adjacent frames, the difference sequence D={D2, D3, ..., D T}.

[0073] Furthermore, the key frame screening method based on the adaptive threshold for screening key frames of the feature difference sequence includes:

[0074] Based on the characteristic difference sequence D, calculate the mean μ of the difference sequence D and standard deviation σ D :

[0075]

[0076] Set the adaptive threshold τ to:

[0077] τ=μ D +α·σ D ;

[0078] Among them, α is an adjustable parameter used to control the strictness of key frame screening. For each frame, if its difference value D t If it is greater than the threshold τ, it is marked as a candidate key frame.

[0079] Furthermore, the use of the K-means clustering algorithm to remove redundant frames with similar content in key frames includes:

[0080] The feature vectors of all candidate key frames {fk1, fk2, ..., fk K} as input, where K is the number of candidate key frames;

[0081] Use the K-means algorithm to cluster the feature vectors into C clusters, where C is the preset number of cluster centers;

[0082] For each cluster, the frame closest to the cluster center is selected as the final keyframe;

[0083] Finally, the optimized key frame set K = {k1, k2, ..., k C}, where C<<T.

[0084] Furthermore, the behavior classification decoder includes: inputting the spatiotemporal features of the key frames obtained by the spatiotemporal feature extractor and the behavior category query vector composed of learnable parameters, and generating the final classification output through the feature interaction module, dynamic weight allocation module, nonlinear mapping module and output generation module in sequence.

[0085] Furthermore, the feature interaction module calculates the similarity between query vectors, dynamically assigns weights, and extracts global features; the dynamic weight assignment module interacts the query vector with the spatiotemporal feature sequence, dynamically assigns weights, and captures the spatial correlation between key frames; the nonlinear mapping module performs nonlinear transformation on the features through a multi-layer perceptron to enhance the expressive power of the features; and the output generation module maps high-dimensional features to the probability distribution of behavior categories through a pooling operation based on the nonlinear mapping module to generate the final classification output.

[0086] Furthermore, it also includes: adding a group decoding strategy, dividing the behavior categories into several groups through the group decoding strategy, each group shares a query vector, and generating outputs of multiple categories through affine transformation and pooling operations, reducing the computational complexity to linear.

[0087] In a specific embodiment, a video behavior recognition system based on substation monitoring includes:

[0088] Acquisition module: used to collect substation monitoring videos and extract video frame sequences;

[0089] Key frame screening module: used to input the extracted video frame sequence into the key frame extraction module based on spatiotemporal features for key frame screening;

[0090] Behavior classification module: This module inputs the filtered keyframes into the behavior classification decoder, performs multi-level feature interaction and dynamic weight assignment on the keyframes, and combines the self-attention mechanism with the group decoding strategy to capture the temporal correlation of the keyframes and perform behavior classification.

[0091] Output module: used to finally output behavior recognition results.

[0092] In a specific implementation, a video behavior recognition method based on substation monitoring realizes direct processing and recognition of video data; a key frame extraction method based on spatiotemporal features significantly improves the efficiency of video reasoning; a behavior classification decoder effectively improves the ability to capture fine-grained behavior features, such as Figure 1 As shown in the figure, we propose an end-to-end video action recognition system that directly processes video stream data and outputs action recognition results. It consists of two core modules: a keyframe extraction module based on spatiotemporal features and an action classification module. These modules work together to implement a complete process from raw video input to action classification output.

[0093] Specifically, the key frame extraction module based on spatiotemporal features includes:

[0094] Keyframe extraction is a critical step in video behavior recognition. Its goal is to extract the most representative frames from a video stream, reducing redundant computations and preserving key behavioral information. Traditional keyframe extraction methods, which rely on inter-frame differences or fixed-time sampling, struggle to effectively capture the dynamic characteristics of behaviors in complex scenarios. To address this issue, this paper proposes a keyframe extraction method based on spatiotemporal features. By incorporating the spatiotemporal information of the video, this method enables intelligent selection of keyframes.

[0095] S11: Spatiotemporal Feature Extraction

[0096] Spatiotemporal features are important information for describing the dynamic changes of behaviors in videos. 3D convolutional neural network (3D-CNN) is used to extract spatiotemporal features of videos. The specific structure is as follows: Figure 2 As shown in Figure 2, 3D-CNN can capture feature changes in both spatial and temporal dimensions by extending the traditional 2D convolution kernel in the temporal dimension. Given an input video frame sequence V = {v1, v2, ..., v T}, where v t Represents the t-th frame image, the convolution operation of 3D-CNN can be expressed as:

[0097]

[0098] Among them, F i，j，k is the output feature map, W m，n，p is the 3D convolution kernel, b is the bias term, and M×N×P is the size of the convolution kernel. Through multiple layers of 3D convolution and pooling operations, the spatiotemporal feature representation of the video is finally obtained: F = {f1, f2, ..., f T}, where f t Represents the spatiotemporal feature vector of the t-th frame.

[0099] S12: Calculation of feature differences between frames

[0100] For the spatiotemporal feature sequence F, define the feature difference D between the t-th frame and the t-1-th frame t for:

[0101] D t =||f t -f t-1 || 2 ;

[0102] Among them, ||·||2 represents the Euclidean distance. By calculating the feature differences of all adjacent frames, we can get the difference sequence D={D2, D3, ..., D T}.

[0103] S13: Keyframe Filtering

[0104] Based on the feature difference sequence D, a key frame screening method with adaptive threshold is designed. First, the mean μ of the difference sequence is calculated. D and standard deviation σ D :

[0105]

[0106] Then, the adaptive threshold τ is set as:

[0107] τ=μ D +α·σ D ;

[0108] Among them, α is an adjustable parameter used to control the strictness of key frame screening. For each frame, if its difference value D t If it is greater than the threshold τ, it is marked as a candidate key frame.

[0109] S14: Redundant frame removal

[0110] To further optimize the key frame set, the K-means clustering algorithm is used to remove redundant frames with similar content. The specific steps are as follows:

[0111] 1) The feature vectors of all candidate key frames {fk1, fk2, ..., fk K} as input, where K is the number of candidate key frames.

[0112] 2) Use the K-means algorithm to cluster the feature vectors into C clusters, where C is the preset number of cluster centers.

[0113] 3) For each cluster, select the frame closest to the cluster center as the final key frame. Through the above steps, the optimized key frame set K = {k1, k2, ..., k C}, where C<<T, which significantly reduces the amount of video data.

[0114] Specifically, the behavior classification decoder includes:

[0115] This paper designs a behavior classification decoder (Decoder) based on an attention mechanism. This decoder aims to improve the accuracy and robustness of power personnel behavior recognition through multi-level feature interaction and dynamic weight allocation. By combining a self-attention mechanism with a group decoding strategy, the decoder effectively captures the temporal correlation of key frames and maintains low computational complexity for large-scale behavior classification tasks.

[0116] S21: Decoder Architecture Design

[0117] The core idea of this embodiment is to achieve accurate classification of human behavior through multi-level spatiotemporal feature extraction and dynamic weight allocation. Figure 3 As shown in Figure 1, its input consists of two parts: the spatiotemporal features of the keyframes obtained by the spatiotemporal feature extractor, and the behavior category query vector (Query, Q) composed of learnable parameters. The decoder's processing flow is divided into four main stages: feature interaction, dynamic weight allocation, nonlinear mapping, and output generation, specifically including:

[0118] 1) Feature Interaction Module

[0119] This module uses the self-attention mechanism to capture the potential correlations between different behavior categories. Specifically, the feature interaction module calculates the similarity between query vectors and dynamically assigns weights to extract global features. Its mathematical expression is as follows:

[0120]

[0121] Among them, Q is the query vector, K is the key vector, d is the dimension of the vector, and the SoftMax function is used to normalize the weights.

[0122] 2) Dynamic Weight Allocation Module

[0123] Based on feature interaction, this module further interacts the query vector with the spatiotemporal feature sequence and dynamically assigns weights to capture the spatial correlation between key frames. The calculation formula is as follows:

[0124]

[0125] Among them, F is the spatiotemporal feature sequence, d f is the feature dimension.

[0126] 3) Nonlinear Mapping Module

[0127] This module uses a multi-layer perceptron (MLP) to perform nonlinear transformation on features, further enhancing the expressiveness of features. Its mathematical expression is as follows:

[0128] Mapping(x)=ReLU(W1x+b1)W2+b2;

[0129] Among them, W1 and W2 are learnable weight matrices, b1 and b2 are bias terms, and ReLU is the activation function.

[0130] 4) Output Generation Module

[0131] This module maps high-dimensional features to the probability distribution of behavior categories through pooling operations to generate the final classification output. Its calculation formula is as follows:

[0132] Output(x)=MeanPooling(x, dim=D);

[0133] Among them, D is the feature dimension, and MeanPooling means mean pooling along the feature dimension.

[0134] S22: Multi-head attention decoding strategy

[0135] In order to deal with the computational complexity problem caused by the increase in the number of behavior categories, this paper proposes a multi-head attention decoding strategy. In the traditional decoder, each query vector corresponds to a behavior category. As the number of categories increases, the computational complexity increases quadratically O(n 2 The group decoding strategy divides the behavior categories into several groups, each group shares a query vector, and generates outputs of multiple categories through affine transformation and pooling operations, thereby reducing the computational complexity to linear O(n). Specifically:

[0136] 1) Grouped Query

[0137] The behavior categories are randomly divided into several groups, each containing g categories. Each group corresponds to a query vector, and the number of query vectors is K = N / g, where N is the total number of categories.

[0138] 2) Affine Transformation

[0139] Through the affine transformation layer, each query vector is expanded into g category scores (logits). Its mathematical expression is as follows:

[0140] Li =W k q k +b k ;

[0141] Among them, q k is the kth query vector, W k and b k are the learnable weight matrices and bias terms.

[0142] 3) Pooling Operation

[0143] After the affine transformation, a pooling operation is performed along the feature dimension to generate the final classification output. The computational cost is N×D multiplications, where D is the feature dimension.

[0144] Through the group decoding strategy, the Decoder significantly reduces the computational overhead while maintaining high accuracy, making it suitable for large-scale behavior classification tasks.

[0145] In a specific embodiment, a video behavior recognition method based on substation monitoring is tested, and the results and analysis are as follows:

[0146] S31: Dataset

[0147] To ensure the model's generalization and robustness in real-world power scenarios, we constructed a high-quality private dataset specifically for the power personnel behavior recognition task. This dataset covers typical power scenarios, such as substations, and fully simulates the complexity and diversity of real-world working environments.

[0148] Dataset construction:

[0149] 1) Data collection:

[0150] Data collection was conducted at multiple substations within the China Southern Power Grid, using high-resolution cameras to capture the daily work activities of power workers. The data collection process took into account factors such as varying lighting conditions, weather variations, and equipment obstruction to ensure data diversity and representativeness.

[0151] 2) Data annotation:

[0152] Each image is manually annotated with key body points (such as the head, shoulders, elbows, and knees) and behavioral category labels. This annotation work is completed by a professional team and undergoes multiple rounds of verification to ensure an accuracy of over 99%.

[0153] 3) Data augmentation:

[0154] In order to further improve the diversity of the dataset and the generalization ability of the model, a variety of data enhancement techniques were adopted, including random flipping, brightness adjustment, contrast adjustment, and saturation adjustment.

[0155] Dataset statistics:

[0156] The private dataset contains 20,000 behavioral examples, covering a variety of typical behaviors of power personnel in substation scenarios. The number and distribution of samples in the private dataset are shown in Table 1. It is important to note that the number of examples represents the number of instances of the behavior in a video. The long video duration is mainly due to the fact that the majority of the video consists of normal behavior, with only a few segments containing abnormal or specific behaviors.

[0157] Table 1. Number of samples in private datasets

[0158] Behavior Category Number of examples Total video length (hours) Climbing 2321 213 Crossing 2442 283 Calling 2121 509 Falling 2486 236 Carrying 2776 247 Smoking 653 80 Not wearing a helmet (No Helmet) 4321 599 No Uniform 2545 360

[0159] S32: Implementation Details

[0160] This example was carried out in the following experimental environment: the computing platform was equipped with an AMD EPYC 7742 processor (with 1TB of memory) and four NVIDIA Tesla V100 GPUs (with 32GB of video memory), the operating system was CentOS 7.6, and the deep learning framework was built based on TensorFlow 2.5.

[0161] In the model pre-training phase, a phased training strategy was adopted. The encoder network was optimized first, while the decoder parameters remained unchanged. The input image size was adjusted to 416×320 pixels, and the batch size was set to 32. The optimizer used the RMSprop algorithm, and the initial learning rate was set to 2×10 -4 , and adopted the Reduce LR On Plateau learning rate scheduling strategy. When the validation loss did not decrease for 5 consecutive epochs, the learning rate was reduced to 20% of the original. The entire pre-training process lasted for 250 epochs, and an early stopping mechanism (patience = 15) was introduced during training to prevent overfitting. During the model fine-tuning phase, a parameter freezing strategy was adopted to fix the encoder weights and only perform end-to-end optimization on the decoder. The initial learning rate in this phase was set to 5×10 -6 , using the cyclic learning rate scheduling method, the batch size is adjusted to 64, and the weight decay is set to 5×10 -7The fine-tuning process lasted 50 epochs and employed gradient clipping to limit the gradient norm to less than 1.0 to ensure training stability. Furthermore, label smoothing (label smoothing = 0.1) and mixed-precision training were incorporated into the fine-tuning phase to improve model performance.

[0162] S33: Experimental Results

[0163] The classification models compared in the experiment include traditional machine learning algorithms KNN and SVM, deep learning algorithm MLP and the model proposed in this invention. The model proposed in this invention has the following recognition accuracy for the five common types of power personnel behaviors: the mixed matrix Top1 accuracy is as follows: Figure 4 The results of power personnel behavior recognition are shown in Table 2. From the classification accuracy point of view, it is easy to identify the two categories of climbing and crossing. Figure 5 This is also verified by the diagram of the power personnel behavior recognition effect. These two actions have richer recognition. The accuracy is slightly lower for easily confused behaviors such as smoking and not wearing a helmet (No Helmet), which may be because the visual area occupied by these two actions is too small. However, compared with traditional classification algorithms and deep learning-based algorithms, the proposed method has achieved the best behavior recognition accuracy.

[0164] Table 2 Results of power personnel behavior recognition

[0165] Behavior Category Accuracy / % Climbing 93.8 Crossing 94.8 Calling 92.7 Falling 93.5 Carrying 91.7 Smoking 90.8 Not wearing a helmet (No Helmet) 91.0 No Uniform 93.6

[0166] Table 3 Algorithm comparison experimental results

[0167]

[0168] In order to verify the superior performance of the proposed method in the behavior classification task, extensive ablation studies were conducted involving different encoders and decoders. The algorithm comparison experimental results can be seen from the data in Table 3. In the video processing stage, the key frame sampling method of the present invention achieves a significant improvement in inference speed compared to uniform sampling, while ensuring that the accuracy is not compromised. In addition, the spatiotemporal feature encoder based on 3D convolution outperforms OpenPose in terms of accuracy. This result also reflects the advantages of the present invention in encoder design. Finally, the decoder proposed in the present invention also surpasses the traditional behavior decoder in performance, further proving the effectiveness of the new method.

[0169] Behavior recognition of power personnel is one of the key technologies to ensure the safe operation and maintenance of power systems, and plays a vital role in ensuring the safe operation and maintenance of substations. The present invention addresses the problems of insufficient feature extraction granularity and low processing efficiency in traditional machine learning algorithms in behavior recognition. For example, the current mainstream recognition algorithms are mostly based on traditional machine learning methods such as support vector machines (SVM) and multi-layer perceptrons (MLP). They have limitations such as insufficient granularity in behavior feature extraction and being limited to static image processing, making it difficult to meet the application requirements of actual video surveillance scenarios. Therefore, a behavior recognition method for substation video surveillance scenarios is proposed. This method directly processes video data by constructing an end-to-end video behavior recognition framework, thus getting rid of the dependence on static images; designs a key frame extraction method based on spatiotemporal features, significantly improving the efficiency of video inference; and innovatively proposes a behavior classification decoder, which effectively improves the ability to capture fine-grained behavior features and effectively improves the accuracy of behavior recognition. Experimental results show that the proposed algorithm achieves a comprehensive recognition rate of 98.1% on a substation work scene video dataset. Compared with traditional image processing methods, it achieves significant improvements in recognition accuracy and processing speed, verifying its effectiveness in practical applications and its practical engineering value.

[0170] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0171] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video behavior recognition method based on substation monitoring, characterized in that: include: Collect substation monitoring videos and extract video frame sequences; The extracted video frame sequence is input into the key frame extraction module based on spatiotemporal features for key frame screening; The filtered key frames are input into the behavior classification decoder, which performs multi-level feature interaction and dynamic weight assignment on the key frames. The self-attention mechanism and group decoding strategy are combined to capture the correlation of key frame time and perform behavior classification. Finally, the behavior recognition results are output.

2. The video behavior recognition method based on substation monitoring according to claim 1 is characterized in that: The key frame extraction module based on spatiotemporal features includes: Use 3D convolutional neural network to extract spatiotemporal features of video frame sequences; Calculate the feature difference between frames of the extracted spatiotemporal features to obtain a feature difference sequence; The key frame screening method based on adaptive threshold is used to screen the key frames of feature difference sequences; The K-means clustering algorithm is used to remove redundant frames with similar content in key frames.

3. The video behavior recognition method based on substation monitoring according to claim 2 is characterized in that: The method of extracting spatiotemporal features of a video frame sequence using a 3D convolutional neural network includes: Given an input video frame sequence V = {v1, v2, ..., v T }, where v t Represents the t-th frame image, and the convolution operation of 3D-CNN is expressed as: Among them, F i，j，k is the output feature map, W m，n，p is the 3D convolution kernel, b is the bias term, and M×N×P is the size of the convolution kernel; Through multi-layer 3D convolution and pooling operations, the spatiotemporal feature representation of the video is finally obtained F = {f1, f2, ..., f t }, where f t Represents the spatiotemporal feature vector of the t-th frame.

4. The video behavior recognition method based on substation monitoring according to claim 2 is characterized in that: The step of calculating the inter-frame feature differences of the extracted spatiotemporal features to obtain a feature difference sequence includes: For the spatiotemporal feature sequence F, define the feature difference D between the t-th frame and the t-1-th frame t for: D t =||f t -f t-1 ||2; Among them, ||·||2 represents the Euclidean distance. By calculating the feature differences of all adjacent frames, the difference sequence D={D2, D3, ..., D T }.

5. The video behavior recognition method based on substation monitoring according to claim 2 is characterized in that: The key frame screening method based on the adaptive threshold for screening the key frames of the feature difference sequence includes: Based on the characteristic difference sequence D, calculate the mean μ of the difference sequence D and standard deviation σ D : Set the adaptive threshold τ to: τ=μ D +a·s D ; Among them, α is an adjustable parameter used to control the strictness of key frame screening. For each frame, if its difference value D t If it is greater than the threshold τ, it is marked as a candidate key frame.

6. The video behavior recognition method based on substation monitoring according to claim 2 is characterized in that: The method of using the K-means clustering algorithm to remove redundant frames with similar content in key frames includes: The feature vectors of all candidate key frames {fk1, fk2, ..., fk K } as input, where K is the number of candidate key frames; Use the K-means algorithm to cluster the feature vectors into C clusters, where C is the preset number of cluster centers; For each cluster, the frame closest to the cluster center is selected as the final keyframe; Finally, the optimized key frame set K = {k1, k2, ..., k C }, where C<<T.

7. The video behavior recognition method based on substation monitoring according to claim 1 is characterized in that: The behavior classification decoder includes: inputting the spatiotemporal features of key frames obtained by the spatiotemporal feature extractor and the behavior category query vector composed of learnable parameters, and generating the final classification output through a feature interaction module, a dynamic weight allocation module, a nonlinear mapping module and an output generation module in sequence.

8. The video behavior recognition method based on substation monitoring according to claim 7 is characterized in that: The feature interaction module calculates the similarity between query vectors, dynamically assigns weights, and extracts global features; the dynamic weight assignment module interacts the query vector with the spatiotemporal feature sequence, dynamically assigns weights, and captures the spatial correlation between key frames; The nonlinear mapping module uses a multi-layer perceptron to perform nonlinear transformation on features to enhance the expressiveness of features; Based on the nonlinear mapping module, the output generation module maps high-dimensional features to the probability distribution of behavior categories through pooling operations to generate the final classification output.

9. The video behavior recognition method based on substation monitoring according to claim 8 is characterized in that: Also includes: A group decoding strategy is added to divide the behavior categories into several groups, each group sharing a query vector, and generate outputs of multiple categories through affine transformation and pooling operations, reducing the computational complexity to linear.

10. A video behavior recognition system based on substation monitoring, characterized in that: include: Acquisition module: used to collect substation monitoring videos and extract video frame sequences; Key frame screening module: used to input the extracted video frame sequence into the key frame extraction module based on spatiotemporal features for key frame screening; Behavior classification module: This module inputs the filtered keyframes into the behavior classification decoder, performs multi-level feature interaction and dynamic weight assignment on the keyframes, and combines the self-attention mechanism with the group decoding strategy to capture the temporal correlation of the keyframes and perform behavior classification. Output module: used to finally output behavior recognition results.

Images