Electric power overhaul video motion detection method based on multi-scale state space

Abstract

The invention discloses an electric power overhaul video action detection method based on a multi-scale state space, and the method comprises the steps: firstly collecting long video data in an electric power overhaul process, segmenting the long video data into a plurality of time sequence segments, and carrying out the spatial feature coding and high-dimensional mapping of the plurality of time sequence segments, and obtaining a time sequence token sequence; then, an electric power overhaul video action detection network is constructed and trained, and the electric power overhaul video action detection network comprises a time sequence multi-scale coding module, a scale perception state fusion device and a multi-label prediction layer; and finally, performing action detection on a time sequence token sequence corresponding to the electric power overhaul long video data to be detected by adopting the trained electric power overhaul video action detection network to obtain an action category detection result. According to the method, timing sequence multi-scale coding, state space modeling and a scale perception feature fusion mechanism are fused, the short-time sudden action and the long-range dependency relationship can be captured at the same time, and the recognition capability and the positioning precision of the complex concurrent action in the electric power overhaul video are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of power maintenance data processing technology, specifically a power maintenance video motion detection method based on multi-scale state space. Background Technology

[0002] With the rapid development of the Global Energy Internet and the continuous improvement of the intelligence and automation level of power systems, power equipment maintenance has become a crucial link in ensuring the safe and stable operation of the power grid. To this end, the National "Interim Measures for Power Reliability Management" explicitly requires power companies to establish a reliability management system covering the entire process of planning, construction, production, and operation, providing fundamental guidance for the quality and standards of maintenance work. In recent years, video surveillance technology has been widely used in power maintenance. By collecting real-time video data from the field, the maintenance process can be recorded and analyzed, achieving comprehensive recording of work behavior, equipment status, and the field environment. However, power maintenance videos are generally characterized by long durations, dense actions, and complex backgrounds. Developing time-series action recognition technology for power maintenance scenarios has become a key support for achieving automated supervision and quality assessment of the maintenance process, and is of significant practical importance for ensuring the reliable operation of the power grid.

[0003] While video surveillance provides abundant visual data support for power maintenance processes, existing analysis methods still have significant limitations when dealing with the complex and diverse action patterns in power maintenance videos. First, the scale of actions in maintenance operations varies greatly, ranging from short, sudden actions like instrument readings and switch switching to long-term, systematic operations. Traditional methods struggle to capture action characteristics at different time scales within a single model, easily losing crucial information at different temporal granularities. Second, work sites often involve dense and concurrent actions. If multiple workers are operating collaboratively, or if a single person is simultaneously holding tools and observing instruments, existing detection methods may miss or misjudge detections. Furthermore, the power maintenance environment is complex, with factors such as changing lighting, equipment obstruction, and background interference, resulting in long action context dependencies, further increasing the difficulty of action recognition. Traditional models cannot flexibly model complex dynamic changes.

[0004] In recent years, deep learning-based action recognition and spatiotemporal action detection technologies have made significant progress. In the field of temporal action detection, mainstream methods can be divided into two categories: segment-level detection and frame-level detection. Segment-level methods locate actions by constructing a three-dimensional spatiotemporal pipeline, focusing on utilizing the temporal continuity of videos, but have limitations in classification accuracy. Frame-level methods, on the other hand, borrow from image object detection approaches, generating action bounding boxes and classifying them frame by frame, providing more granular temporal localization capabilities. Building upon this foundation, researchers have introduced various advanced network architectures to improve model performance. For example, the SlowFast network constructs a dual-stream architecture with slow and fast paths to capture spatial semantic information and rapidly changing temporal dynamic information in videos, respectively. The Transformer-based encoder-decoder structure models the global context through a self-attention mechanism and uses the decoder to generate accurate action localization and classification results. The YOWO series frameworks, by fusing the spatial features of two-dimensional convolutional neural networks and the spatiotemporal features of three-dimensional convolutional neural networks, achieve action localization and recognition in videos within a single forward propagation. While these methods perform well on publicly available datasets, they still have significant shortcomings when dealing with specific scenarios such as power maintenance. On the one hand, most existing models are designed for general actions and do not fully consider the strong temporal dependencies, multi-scale concurrency, and semantically dense characteristics of actions in power maintenance. On the other hand, most methods fail to effectively integrate local details and global contextual information during feature extraction, resulting in insufficient modeling capabilities for short-term bursts of actions and long-term dependencies. Therefore, there is an urgent need for a power maintenance video action detection method that can adaptively integrate multi-scale temporal information, accurately identify concurrent actions, and possess strong generalization capabilities. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for motion detection in power maintenance videos based on multi-scale state space. It integrates temporal multi-scale coding, state space modeling and scale-aware feature fusion mechanism, which can simultaneously capture short-term sudden actions and long-term dependencies, and significantly improve the recognition ability and positioning accuracy of complex concurrent actions in power maintenance videos.

[0006] The technical solution of this invention is as follows: A method for motion detection in power maintenance videos based on multi-scale state space, specifically including the following steps: (1) Collect long video data during power maintenance, then divide the long video data into multiple continuous and non-overlapping time segments, and then perform spatial feature encoding and high-dimensional mapping on the multiple time segments to obtain a time token sequence. (2) Construct a power maintenance video action detection network for detecting action categories in power maintenance videos. The power maintenance video action detection network includes a time-series multi-scale coding module, a scale-aware state fusion unit, and a multi-label prediction layer. The time-series multi-scale coding module performs multi-level time modeling on the input time-series token sequence to capture action features at different time scales. The scale-aware state fusion unit adopts a progressive cross-scale propagation mechanism to perform cross-level interaction and adaptive aggregation on the multi-scale features output by the time-series multi-scale coding module to generate a unified time-series representation. The multi-label prediction layer maps the unified time-series representation to the confidence of each action category on each time segment, thereby achieving end-to-end detection output for dense concurrent actions. (3) Construct a loss function to train the power maintenance video action detection network to obtain a trained power maintenance video action detection network. Use the trained power maintenance video action detection network to perform action detection on the time-series token sequence corresponding to the power maintenance long video data to be detected, and obtain the action category detection results.

[0007] The multiple time segments are spatially encoded using an image encoder in a pre-trained contrastive language image pre-training model to obtain a time-series feature sequence. , Represents the total number of time-series segments. Representing the The feature vectors corresponding to each time segment are mapped to hidden dimensions through a learnable linear projection layer. Obtain the time-series token sequence ,in Represents the batch dimension.

[0008] The aforementioned time-series multi-scale coding module is composed of The system consists of cascaded temporal coding layers, each containing a temporal convolution module and an expanded state space module. The processing steps of each temporal coding layer specifically include the following: S11. The temporal convolution module first expands the channel dimension through pointwise linear projection, as shown in the following equation (1): (1); In equation (1), Representing the The feature sequence output by the layered temporal coding layer after dimensional expansion. No. The output of the layered temporal coding layer, For channel-by-channel projection weights, For bias terms, The channel dimension is That is, the dimension of the feature sequence is in a fixed proportion. To expand; S12. The temporal convolution module performs dilation convolution operation, as shown in equation (2) below: (2); In equation (2), The output of a one-dimensional dilated convolution at time step ,aisle The value at which the kernel size of the one-dimensional dilated convolution is... Step size is Expansion rate ; Representing the In the sequential coding layer, the one-dimensional dilated convolutional kernel processes the channels. When considering the characteristics of a location, its internal first... Weight parameters corresponding to each position; Representative in the passage Up, according to step size and expansion rate Calculated Feature values ​​at each position; S13. The temporal convolution module performs a linear transformation, as shown in equation (3) below: (3); In equation (3), Representing the The output features of the temporal convolutional module of the temporal coding layer are used as the first... Input to the extended state space module of the layer-time coding layer; and These represent the weights and biases of the one-dimensional dilated convolution kernel, respectively. This represents the output tensor of a one-dimensional dilated convolution. S14. The state space expansion module first performs an expansion operation. The input feature sequence is divided into non-overlapping subsequences , see the following formula (4) for details: (4); In equation (4), Representing the The state space dilation rate of the temporal coding layer's dilated state space module is similar to the dilation rate of the temporal convolution module. equal; Represents the effective batch dimension after expansion; expansion operation Through intervals Sampling is performed on each token to divide the input feature sequence into... The input feature sequence has independent subsequences and a temporal length of . The temporal length of the partitioned subsequence It is an integer; S15. The extended state space module processes each subsequence through a state space transformation layer. The process is performed to obtain the output sequence corresponding to each subsequence. For subsequences At each time step Input token Its discrete-time state-space equation is shown in equation (5) below: (5); In equation (5), It is the discretized state transition matrix; and Both are projection matrices; Represents time step The hidden state, The dimension of the hidden state is used to cyclically fuse historical information. Representative input token The feature vector output by the mapping; Equation (5) is executed recursively over the sequence length, thereby transforming the input subsequence Mapped to output sequence ; S16, the expanded state space module will The output sequence corresponding to each subsequence By reverse operation Perform a re-merge and restore the original time sequence to its original state. The sequence length and batch dimension are shown in the following formula (6): (6); In equation (6), Representing the The output of the extended state-space module of the layer-time coding layer, i.e., the first... The output of the layered temporal coding layer; The output of the aforementioned temporal multi-scale coding module is Output set of layer-time coding layer .

[0009] The processing procedure of the scale-aware state fusion device specifically includes the following steps: S21. First, perform linear projection, as shown in equation (7) below: (7); In equation (7), and These are the weights and biases of the linear projection, respectively. It is the unified latent space dimension mapped to after linear projection; the linear projection operation is applied independently along the time dimension to the feature at each time step, making The multi-scale feature set obtained after projection ; S22. Extract global temporal statistics for each scale feature. , see the following formula (8) for details: (8); In equation (8), Represents global average pooling over the time dimension. For time step eigenvectors; S23. Calculate the fusion weight according to the following formula (9). ; (9); In equation (9), Representing the The original importance scores corresponding to the features at each scale; The representative will use global time statistics and Perform the splicing operation; , These are the weights and biases of the learnable linear transformation, respectively; Representative to The original importance scores corresponding to each scale feature are applied to the Softmax function to obtain a set of normalized fusion weights that sum to 1. ; S24. Based on the fusion weights Calculate the initial fusion sequence , see the following formula (10) for details: (10); In equation (10), Representing the Preliminary fusion characteristics at each time step; S25. Introduce a bidirectional expanding state space update mechanism. First, define the time expansion rate of the fusion phase as... And through reversible operation The initial fusion characteristics are divided into time series. There are several non-overlapping subsequences, as shown in equation (11): (11); In equation (11), The first part obtained by the division Subsequences represent and Divide by The remainders are the same. Then for each subsequence Apply bidirectional state space updates, as shown in equation (12): (12); In equation (12), subscript Representing a subsequence Time step index in the middle; and These are the forward and backward state transition matrices, respectively; and Both are input projection matrices; To output the projection matrix; and These represent the forward and backward hidden states, respectively. represent and Perform the splicing operation. To apply a bidirectional state space update to the output; Finally The output sequence corresponding to each subsequence Reverse operation Perform a re-merge, restore the original time sequence, and recover to The sequence length and batch dimension are shown in the following formula (13): (13); In equation (13), This is the fusion sequence after a bidirectional expansion of the state space update; S26, will and splicing along the channel, via Convolution and Sigmoid activation function generate attention matrix , see the following formula (14) for details: (14); In equation (14), The representative will and Splice along the channel dimensions. represent convolution, Represents the Sigmoid activation function; Then, a unified temporal representation is obtained through residual weighted fusion. , see the following formula (15) for details: (15); In equation (15), This indicates element-wise multiplication.

[0010] The multi-label prediction layer maps the unified temporal representation to the original confidence scores of each action category in each temporal segment. , see the following formula (16) for details: (16); In equation (16), and These are the weight matrix and bias term of the multi-label prediction layer. It is the output of the multi-label prediction layer, for the first label in the batch. Individual samples, time steps Vector of time Each element in , Represents the prediction of the multi-label prediction layer. Each action category in time step The original confidence level that occurred.

[0011] The loss function See the following formula (17) for details: (17); In equation (17), Represents batch size; Represents timing length; The total number of action categories; For real labels, indicating the first In the nth sample Each action category in time step The above indicates whether it has occurred; 1 represents that it has occurred, and 0 represents that it has not occurred. This is the detection result of the power maintenance video motion detection network, namely the first... In the nth sample Each action category in time step The original confidence level occurred on; It is the Sigmoid activation function, which will Mapped to probability values, i.e. .

[0012] The continuous values ​​output by the power maintenance video action detection network are transformed into specific power maintenance action results through a threshold-based action category decision logic. Specifically, this involves converting the raw confidence scores output by the trained power maintenance video action detection network into... Applying the Sigmoid activation function It is mapped to a probability space and a preset decision threshold is applied. Perform binarization to generate an action mask matrix. The calculation process is shown in the following formula (18): (18); In equation (18), Representing the Action type in time step It happened on the surface. Representing the Action type in time step This did not happen; Finally, based on the action mask matrix Perform category index decoding, and for each time-series segment in the long video data of power maintenance, output the time step. A collection of names of all currently occurring action categories. , see the following formula (19) for details: (19); In equation (19), The name representing the action category.

[0013] Advantages of this invention: (1) High sensitivity detection capability for short-term burst actions: The present invention adopts a temporal convolution module in the temporal multi-scale coding module, which focuses on short-term burst behaviors and subtle changes in maintenance actions, effectively capturing instantaneous events that are easily missed by traditional methods. The temporal convolution module realizes local interaction through dilation convolution operation and increases the dilation rate layer by layer to expand the temporal receptive field, so that the early layers focus on the fine-grained features of neighboring frames, and the deep layers introduce contextual information from a greater distance, which enhances the ability to distinguish fine-grained actions and significantly improves the temporal positioning accuracy of short-term actions.

[0014] (2) Global long-term action dependency modeling capability: This invention addresses the long-term dependency relationship of cross-time period actions in power maintenance by introducing an extended state space module in the temporal multi-scale coding module. Combined with reversible extended sampling and dynamic state space update mechanism, the input sequence is divided into multiple sub-sequences, and forward and backward bidirectional state propagation is performed on each sub-sequence to achieve global dependency modeling across frames. This mechanism overcomes the limitation of traditional state space models that can only process tokens sequentially and effectively models the semantic association between long-term actions.

[0015] (3) Adaptive multi-level temporal information fusion mechanism: In order to integrate multi-granularity action features from instantaneous operation to continuous operation in power maintenance videos, this invention sets up a scale-aware state fusion device, realizes semantic alignment and adaptive weighted fusion of multi-level features through a progressive cross-scale propagation mechanism, uses deep global features to guide the dynamic weight allocation of temporal representations of each layer, and combines bidirectional state updates and attention enhancement to suppress redundant feature responses and improve the robust recognition capability of multi-temporal granular actions, providing a key feature foundation for the accurate detection of dense and concurrent actions in power maintenance videos.

[0016] (4) High efficiency in recognizing dense concurrent actions: This invention targets the typical scenario of multiple personnel working together and single person performing multiple tasks concurrently in power maintenance videos. It adopts a multi-label prediction layer to directly map the fused unified temporal features to the action category space, and predicts all possible action categories at each time step. The end-to-end design avoids the complex post-processing steps in traditional methods, has higher computational efficiency and real-time performance, and improves the recognition ability of concurrent actions. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] A method for motion detection in power maintenance videos based on multi-scale state space, specifically including the following steps: (1) Collect long video data during power maintenance, and then evenly divide the long video data into multiple continuous and non-overlapping time segments. Each time segment contains a fixed number of 8 or 16 frames of images. Multiple time segments are spatially encoded using the image encoder in the trained contrastive language image pre-training model (CLIP model) to obtain a time feature sequence. , Represents the total number of time-series segments. Representing the The feature vectors corresponding to each time segment are mapped to hidden dimensions through a learnable linear projection layer. Obtain the time-series token sequence ,in Represents the batch dimension; (2) Construct a power maintenance video motion detection network for detecting motion categories in power maintenance videos. The power maintenance video motion detection network includes a time-series multi-scale coding module, a scale-aware state fusion unit, and a multi-label prediction layer. The temporal multi-scale coding module performs multi-level temporal modeling on the input temporal token sequence to capture action features at different time scales. Specifically, it includes the following steps: S11. The temporal convolution module first expands the channel dimension through pointwise linear projection, as shown in the following equation (1): (1); In equation (1), Representing the The feature sequence output by the layered temporal coding layer after dimensional expansion. No. The output of the layered temporal coding layer, For channel-by-channel projection weights, For bias terms, The channel dimension is That is, the dimension of the feature sequence is in a fixed proportion. To expand; S12. The temporal convolution module performs dilation convolution operation, as shown in equation (2) below: (2); In equation (2), The output of a one-dimensional dilated convolution at time step ,aisle The value at which the kernel size of the one-dimensional dilated convolution is... Step size is Expansion rate ; Representing the In the sequential coding layer, the one-dimensional dilated convolutional kernel processes the channels. When considering the characteristics of a location, its internal first... Weight parameters corresponding to each position; Representative in the passage Up, according to step size and expansion rate Calculated Feature values ​​at each position; S13. The temporal convolution module performs a linear transformation, as shown in equation (3) below: (3); In equation (3), Representing the The output features of the temporal convolutional module of the temporal coding layer are used as the first... Input to the extended state space module of the layer-time coding layer; and These represent the weights and biases of the one-dimensional dilated convolution kernel, respectively. The output tensor of a one-dimensional dilated convolution; in equation (2) yes At time step ,aisle The specific value at a given location is the convolution output value at a single position, which is a scalar. S14. The state space expansion module first performs an expansion operation. The input feature sequence is divided into non-overlapping subsequences , see the following formula (4) for details: (4); In equation (4), Representing the The state space dilation rate of the temporal coding layer's dilated state space module is similar to the dilation rate of the temporal convolution module. equal; Represents the effective batch dimension after expansion; expansion operation Through intervals Sampling is performed on each token to divide the input feature sequence into... The input feature sequence has independent subsequences and a temporal length of . The temporal length of the partitioned subsequence It is an integer; if set This yields two subsequences, with their time sequences being respectively... and ; S15. The extended state space module processes each subsequence through a state space transformation layer. The process is performed to obtain the output sequence corresponding to each subsequence. For subsequences At each time step Input token Its discrete-time state-space equation is shown in equation (5) below: (5); In equation (5), It is the discretized state transition matrix; and Both are projection matrices; Represents time step The hidden state, The dimension of the hidden state is used to cyclically fuse historical information. Representative input token The feature vector of the mapping output; in order to fully capture the context, a bidirectional scan is performed on each subsequence in both forward and backward directions. The final output is a synthesis of the bidirectional states, thereby ensuring that the feature vector of each output contains all the past and future context information of its subsequence. Equation (5) is executed recursively over the sequence length, thereby transforming the input subsequence Mapped to output sequence ; S16, the expanded state space module will The output sequence corresponding to each subsequence By reverse operation Perform a re-merge and restore the original time sequence to its original state. The sequence length and batch dimension are shown in the following formula (6): (6); In equation (6), Representing the The output of the extended state-space module of the layer-time coding layer, i.e., the first... The output of the layered temporal coding layer; The output of the temporal multi-scale coding module is Output set of layer-time coding layer ; The scale-aware state fusion unit employs a progressive cross-scale propagation mechanism to perform cross-level interaction and adaptive aggregation of multi-scale features output by the temporal multi-scale coding module, generating a unified temporal representation. This process includes the following steps: S21. First, perform linear projection, as shown in equation (7) below: (7); In equation (7), and These are the weights and biases of the linear projection, respectively. It is the unified latent space dimension mapped to after linear projection; the linear projection operation is applied independently along the time dimension to the feature at each time step, making The multi-scale feature set obtained after projection ; S22. Extract global temporal statistics for each scale feature. , see the following formula (8) for details: (8); In equation (8), Represents global average pooling over the time dimension. For time step eigenvectors; S23. Calculate the fusion weight according to the following formula (9). Utilizing the deepest and most semantically rich global time statistics As a guide, to evaluate and weight features across all levels; (9); In equation (9), Representing the The original importance scores corresponding to the features at each scale; The representative will use global time statistics and Perform the splicing operation; , These are the weights and biases of the learnable linear transformation, respectively; Representative to The original importance scores corresponding to each scale feature are applied to the Softmax function to obtain a set of normalized fusion weights that sum to 1. ; S24. Based on the fusion weights Calculate the initial fusion sequence , see the following formula (10) for details: (10); In equation (10), Representing the Preliminary fusion characteristics at each time step; S25. Introduce a bidirectional expanding state space update mechanism. First, define the time expansion rate of the fusion phase as... And through reversible operation The initial fusion characteristics are divided into time series. There are several non-overlapping subsequences, as shown in equation (11): (11); In equation (11), The first part obtained by the division Subsequences represent and Divide by The remainders are the same. Then for each subsequence Apply bidirectional state space updates, as shown in equation (12): (12); In equation (12), subscript Representing a subsequence Time step index in the middle; and These are the forward and backward state transition matrices, respectively; and Both are input projection matrices; To output the projection matrix; and These represent the forward and backward hidden states, respectively. represent and Perform the splicing operation. The output is after applying a bidirectional state space update; this bidirectional expansion update preserves local details while achieving efficient propagation of long-range dependencies through sparse scanning; Finally The output sequence corresponding to each subsequence Reverse operation Perform a re-merge, restore the original time sequence, and recover to The sequence length and batch dimension are shown in the following formula (13): (13); In equation (13), This is the fusion sequence after a bidirectional expansion of the state space update; S26, will and splicing along the channel, via Convolution and Sigmoid activation function generate attention matrix , see the following formula (14) for details: (14); In equation (14), The representative will and Splice along the channel dimensions. represent convolution, Represents the Sigmoid activation function; Then, a unified temporal representation is obtained through residual weighted fusion. , see the following formula (15) for details: (15); In equation (15), This indicates element-wise multiplication; The multi-label prediction layer maps the unified temporal representation to the original confidence of each action category on each temporal segment, realizing end-to-end detection output for dense concurrent actions. The specific processing process is shown in the following formula (16): (16); In equation (16), and These are the weight matrix and bias term of the multi-label prediction layer. It is the output of the multi-label prediction layer, for the first label in the batch. Individual samples, time steps Vector of time Each element in , Represents the prediction of the multi-label prediction layer. Each action category in time step The original confidence level occurred on; (3) Construct a loss function to train the power maintenance video action detection network to obtain a trained power maintenance video action detection network. Use the trained power maintenance video action detection network to perform action detection on the time-series token sequence corresponding to the power maintenance long video data to be detected, and obtain the action category detection result. loss function See the following formula (17) for details: (17); In equation (17), Represents batch size; Represents timing length; The total number of action categories; For real labels, indicating the first In the nth sample Each action category in time step The above indicates whether it has occurred; 1 represents that it has occurred, and 0 represents that it has not occurred. This is the detection result of the power maintenance video motion detection network, namely the first... In the nth sample Each action category in time step The original confidence level occurred on; It is the Sigmoid activation function, which will Mapped to probability values, i.e. ; During network training, mini-batch stochastic gradient descent is used to update network parameters, and the parameter updates follow these rules: ; Parameters updated for the power maintenance video motion detection network, including those in the temporal convolution module. , And the kernel parameters of one-dimensional dilated convolution, in the dilated state space module , , In the fusion of hidden states and scale-aware states , , , , , , , and and multi-label prediction layers and ; Represents the current iteration number; It's the learning rate, which controls the step size for updating parameters. It is a loss function Regarding parameters The gradient is calculated using the backpropagation algorithm; During the training phase, the initial sequence of time tokens is... and its corresponding dense multi-label real values The data is input into the power maintenance video motion detection network, and the data is processed in batches. The network is organized so that the forward propagation sequentially passes through a temporal multi-scale coding module, a scale-aware state fusion unit, and a multi-label prediction layer to finally obtain the raw confidence score. Then, calculation and real labels Pointwise multi-label loss The gradient is calculated through backpropagation, and all parameters are updated using the optimizer. A cosine annealing learning rate scheduler is used, and a linear warm-up strategy is used in the early stage of training to stabilize the training process. The entire training is carried out on a computing device equipped with a GPU, and the training dataset is traversed multiple times until the performance of the power maintenance video action detection network converges on the validation set. (4) The continuous values ​​output by the power maintenance video action detection network are transformed into specific power maintenance action results through a threshold-based action category decision logic. Specifically, the original confidence scores output by the trained power maintenance video action detection network are converted into the values ​​of the original confidence scores. Applying the Sigmoid activation function It is mapped to a probability space and a preset decision threshold is applied. Perform binarization to generate an action mask matrix. The calculation process is shown in the following formula (18): (18); In equation (18), Representing the Action type in time step It happened on the surface. Representing the Action type in time step This did not happen; Finally, based on the action mask matrix Perform category index decoding, and for each time-series segment in the long video data of power maintenance, output the time step. A collection of names of all currently occurring action categories. , see the following formula (19) for details: (19); In equation (19), The name representing the action category; For example, at time step ,like and Then the time step The direct output is a text list ["tightening screws", "checking the instrument"], which can be determined based on the action mask matrix of continuous time steps. Time-series aggregation is performed to generate action segment records with start and end timestamps. In this way, the complex maintenance actions in long power maintenance videos are automated and output end-to-end, directly assisting maintenance personnel in recording work and judging compliance.

[0020] Performance Analysis: To verify the effectiveness and advancement of this invention, this embodiment constructs a real power maintenance scenario dataset for experimentation and compares and analyzes it with current mainstream time-series action detection methods.

[0021] The dataset used in this experiment comes from real-world power operation monitoring and mobile recording equipment, covering various typical actions such as substation inspection, equipment maintenance, instrument reading, and tool operation. The dataset contains 400 unedited long videos of power maintenance, totaling approximately 160 hours in length, with an average duration of 15-30 minutes. It defines 24 typical power maintenance actions, including but not limited to "wearing a safety helmet," "reading instruments," "closing the circuit breaker," and "recording." The dataset features dense multi-label annotation, allowing multiple action labels at the same time step. In the experiment, the dataset was randomly divided into training, validation, and test sets in a 7:2:1 ratio.

[0022] The power maintenance video motion detection network of this invention is implemented based on the PyTorch deep learning framework, using a single NVIDIA RTX 3090 GPU with 24GB of video memory. A pre-trained CLIP (ViT-L / 14) model is used as the visual backbone network, with a feature dimension of 768. The extracted feature sequences are mapped to the hidden dimensions through linear projection. The temporal multi-scale coding module consists of three temporal coding layers. The kernel size of the temporal convolutional module in each temporal coding layer is... Expansion rate Set as The state space expansion rate of the expanded state space module and Maintaining consistency, the dimension of the hidden state. The time expansion rate of the integration phase Training was performed using the AdamW optimizer, with an initial learning rate set to... The learning rate is decayed using a cosine annealing strategy, the batch size is set to 2, and the number of training rounds is set to 100.

[0023] To comprehensively evaluate the performance of this invention in power maintenance video motion detection tasks, the experiment used four evaluation metrics: mean Average Precision (mAP), precision, recall, and F1 score. mAP is the most commonly used comprehensive metric in the field of temporal motion detection. It measures the model's overall detection capability across multiple classes and time steps by calculating the average precision of each motion category. Precision reflects the accuracy of the model's predictions, and recall assesses the model's missed detection rate. The F1 score is the harmonic mean of precision and recall, comprehensively reflecting the model's overall detection performance.

[0024] This embodiment selects three mainstream methods as benchmark models for comparison: TGM (Temporal Gaussian Mixture) and PDAN (Pyramid Dilated Attention Network) action detection models based on convolutional networks, and MLAD (Multi-Label Action Dependencies) and MS-TCT (Multi-Scale Temporal Conv Transformer) action detection models based on Transformers. The experimental results are shown in Table 1 below: Table 1. Performance comparison results of different methods on the power maintenance dataset.

[0025] The action detection network of this invention outperforms four benchmark models in all evaluation metrics. The mAP reaches 44.2%, which is 1.7% higher than the second-best model MS-TCT. The precision, recall, and F1 score are 51.8%, 45.3%, and 46.3%, respectively. This indicates that the scale-aware state fusion device effectively suppresses background noise and significantly reduces false alarms. The ability of temporal multi-scale encoding and the extended state space module to collaboratively capture long and short-term features effectively reduces missed detections of key actions. The evaluation metrics fully demonstrate that this invention can effectively address the challenges of action-intensive and concurrently variable detection in power maintenance scenarios, exhibiting higher accuracy, stronger robustness, and better overall performance.

[0026] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for motion detection in power maintenance videos based on multi-scale state space, characterized in that: Specifically, it includes the following steps: (1) Collect long video data during power maintenance, then divide the long video data into multiple continuous and non-overlapping time segments, and then perform spatial feature encoding and high-dimensional mapping on the multiple time segments to obtain a time token sequence. (2) Construct a power maintenance video action detection network for detecting action categories in power maintenance videos. The power maintenance video action detection network includes a time-series multi-scale coding module, a scale-aware state fusion unit, and a multi-label prediction layer. The time-series multi-scale coding module performs multi-level time modeling on the input time-series token sequence to capture action features at different time scales. The scale-aware state fusion unit adopts a progressive cross-scale propagation mechanism to perform cross-level interaction and adaptive aggregation on the multi-scale features output by the time-series multi-scale coding module to generate a unified time-series representation. The multi-label prediction layer maps the unified time-series representation to the confidence of each action category on each time segment, thereby achieving end-to-end detection output for dense concurrent actions. (3) Construct a loss function to train the power maintenance video action detection network to obtain a trained power maintenance video action detection network. Use the trained power maintenance video action detection network to perform action detection on the time-series token sequence corresponding to the power maintenance long video data to be detected, and obtain the action category detection results.

2. The power maintenance video motion detection method based on multi-scale state space according to claim 1, characterized in that: The multiple time segments are spatially encoded using an image encoder in a pre-trained contrastive language image pre-training model to obtain a time-series feature sequence. , Represents the total number of time-series segments. Representing the The feature vectors corresponding to each time segment are mapped to hidden dimensions through a learnable linear projection layer. Obtain the time-series token sequence ,in Represents the batch dimension.

3. The power maintenance video motion detection method based on multi-scale state space according to claim 2, characterized in that: The aforementioned time-series multi-scale coding module is composed of The system consists of cascaded temporal coding layers, each containing a temporal convolution module and an expanded state space module. The processing steps of each temporal coding layer specifically include the following: S11. The temporal convolution module first expands the channel dimension through pointwise linear projection, as shown in the following equation (1): （1）； In equation (1), Representing the The feature sequence output by the layered temporal coding layer after dimensional expansion. No. The output of the layered temporal coding layer, For channel-by-channel projection weights, For bias terms, The channel dimension is That is, the dimension of the feature sequence is in a fixed proportion. To expand; S12. The temporal convolution module performs dilation convolution operation, as shown in equation (2) below: （2）； In equation (2), The output of a one-dimensional dilated convolution at time step ,aisle The value at which the kernel size of the one-dimensional dilated convolution is... Step size is Expansion rate ; Representing the In the sequential coding layer, the one-dimensional dilated convolutional kernel processes the channels. When considering the characteristics of a location, its internal first... Weight parameters corresponding to each position; Representative in the passage Up, according to step size and expansion rate Calculated Feature values ​​at each position; S13. The temporal convolution module performs a linear transformation, as shown in equation (3) below: （3）； In equation (3), Representing the The output features of the temporal convolutional module of the temporal coding layer are used as the first... Input to the extended state space module of the layer-time coding layer; and These represent the weights and biases of the one-dimensional dilated convolution kernel, respectively. This represents the output tensor of a one-dimensional dilated convolution. S14. The state space expansion module first performs an expansion operation. The input feature sequence is divided into non-overlapping subsequences , see the following formula (4) for details: （4）； In equation (4), Representing the The state space dilation rate of the temporal coding layer's dilated state space module is similar to the dilation rate of the temporal convolution module. equal; Represents the effective batch dimension after expansion; expansion operation Through intervals Sampling is performed on each token to divide the input feature sequence into... The input feature sequence has independent subsequences and a temporal length of . The temporal length of the partitioned subsequence It is an integer; S15. The extended state space module processes each subsequence through a state space transformation layer. The process is performed to obtain the output sequence corresponding to each subsequence. For subsequences At each time step Input token Its discrete-time state-space equation is shown in equation (5) below: （5）； In equation (5), It is the discretized state transition matrix; and Both are projection matrices; Represents time step The hidden state, The dimension of the hidden state is used to cyclically fuse historical information. Representative input token The feature vector output by the mapping; Equation (5) is executed recursively over the sequence length, thereby transforming the input subsequence Mapped to output sequence ; S16, the expanded state space module will The output sequence corresponding to each subsequence By reverse operation Perform a re-merge and restore the original time sequence to its original state. The sequence length and batch dimension are shown in the following formula (6): （6）； In equation (6), Representing the The output of the extended state-space module of the layer-time coding layer, i.e., the first... The output of the layered temporal coding layer; The output of the aforementioned temporal multi-scale coding module is Output set of layer-time coding layer .

4. The power maintenance video motion detection method based on multi-scale state space according to claim 3, characterized in that: The processing procedure of the scale-aware state fusion device specifically includes the following steps: S21. First, perform linear projection, as shown in equation (7) below: （7）； In equation (7), and These are the weights and biases of the linear projection, respectively. It is the unified latent space dimension mapped to after linear projection; the linear projection operation is applied independently along the time dimension to the feature at each time step, making The multi-scale feature set obtained after projection ; S22. Extract global temporal statistics for each scale feature. , see the following formula (8) for details: （8）； In equation (8), Represents global average pooling over the time dimension. For time step eigenvectors; S23. Calculate the fusion weight according to the following formula (9). ; （9）； In equation (9), Representing the The original importance scores corresponding to the features at each scale; The representative will use global time statistics and Perform the splicing operation; , These are the weights and biases of the learnable linear transformation, respectively; Representative to The original importance scores corresponding to each scale feature are applied to the Softmax function to obtain a set of normalized fusion weights that sum to 1. ; S24. Based on the fusion weights Calculate the initial fusion sequence , see the following formula (10) for details: （10）； In equation (10), Representing the Preliminary fusion characteristics at each time step; S25. Introduce a bidirectional expanding state space update mechanism. First, define the time expansion rate of the fusion phase as... And through reversible operation The initial fusion characteristics are divided into time series. There are several non-overlapping subsequences, as shown in equation (11): （11）； In equation (11), The first part obtained by the division Subsequences represent and Divide by The remainders are the same. Then for each subsequence Apply bidirectional state space updates, as shown in equation (12): （12）； In equation (12), subscript Representing a subsequence Time step index in the middle; and These are the forward and backward state transition matrices, respectively; and Both are input projection matrices; To output the projection matrix; and These represent the forward and backward hidden states, respectively. represent and Perform the splicing operation. To apply a bidirectional state space update to the output; Finally The output sequence corresponding to each subsequence Reverse operation Perform a re-merge, restore the original time sequence, and recover to The sequence length and batch dimension are shown in the following formula (13): （13）； In equation (13), This is the fusion sequence after a bidirectional expansion of the state space update; S26, will and splicing along the channel, via Convolution and Sigmoid activation function generate attention matrix , see the following formula (14) for details: （14）； In equation (14), The representative will and Splice along the channel dimensions. represent convolution, Represents the Sigmoid activation function; Then, a unified temporal representation is obtained through residual weighted fusion. , see the following formula (15) for details: （15）； In equation (15), This indicates element-wise multiplication.

5. The power maintenance video motion detection method based on multi-scale state space according to claim 4, characterized in that: The multi-label prediction layer maps the unified temporal representation to the original confidence scores of each action category in each temporal segment. , see the following formula (16) for details: （16）； In equation (16), and These are the weight matrix and bias term of the multi-label prediction layer. It is the output of the multi-label prediction layer, for the first label in the batch. Individual samples, time steps Vector of time Each element in , Represents the prediction of the multi-label prediction layer. Each action category in time step The original confidence level that occurred.

6. The power maintenance video motion detection method based on multi-scale state space according to claim 5, characterized in that: The loss function See the following formula (17) for details: （17）； In equation (17), Represents batch size; Represents timing length; The total number of action categories; For real labels, indicating the first In the nth sample Each action category in time step The above indicates whether it has occurred; 1 represents that it has occurred, and 0 represents that it has not occurred. This is the detection result of the power maintenance video motion detection network, namely the first... In the nth sample Each action category in time step The original confidence level occurred on; It is the Sigmoid activation function, which will Mapped to probability values, i.e. .

7. The power maintenance video motion detection method based on multi-scale state space according to claim 5, characterized in that: The continuous values ​​output by the power maintenance video action detection network are transformed into specific power maintenance action results through a threshold-based action category decision logic. Specifically, this involves converting the raw confidence scores output by the trained power maintenance video action detection network into... Applying the Sigmoid activation function It is mapped to a probability space and a preset decision threshold is applied. Perform binarization to generate an action mask matrix. The calculation process is shown in the following formula (18): （18）； In equation (18), Representing the Action type in time step It happened on the surface. Representing the Action type in time step This did not happen; Finally, based on the action mask matrix Perform category index decoding, and for each time-series segment in the long video data of power maintenance, output the time step. A collection of names of all currently occurring action categories. , see the following formula (19) for details: （19）； In equation (19), The name representing the action category.

Images