A power production violation behavior detection method, a terminal device and a storage medium

Abstract

The application relates to a power production violation behavior detection method, a terminal device and a storage medium, and the method proposes a network structure ST-SlowFast of time-space information fusion. The network constructs a third auxiliary channel based on a time-space attention mechanism on the basis of the SlowFast network structure, realizes extraction and strengthening of detailed features between different frame rate sampling streams, and captures local action changes through the attention mechanism, and enhances global space feature representation. The application solves the problem of low accuracy of the existing behavior detection method, and guarantees safe construction in the power production scene.

Description

Technical Field

[0001] This invention relates to the field of power production, and in particular to a method, terminal equipment and storage medium for detecting violations in power production. Background Technology

[0002] With the comprehensive advancement of smart grids and the rapid development of power grid infrastructure, ensuring the safety of construction workers during power production has become a key focus of power industry construction. Due to the special nature of power operations, construction workers must perform specialized work in high-risk environments with live electrical equipment at high altitudes. In such situations, even minor safety hazards can lead to serious consequences. Therefore, in addition to requiring construction workers to strictly follow operating manuals, dedicated safety supervision departments and on-site safety officers have been established to provide safety assurance. While the management method of having safety officers on-site can, to some extent, encourage construction workers to comply with safety protection rules (safety attire or safe operating procedures) and reduce construction risks, having safety officers on-site is time-consuming and labor-intensive, and it cannot guarantee 24-hour safety monitoring.

[0003] In recent years, due to the widespread adoption of high-definition surveillance and the rapid development of deep learning technology, computer vision technology, which uses high-definition cameras as a medium, has provided a wide range of solutions for real-time detection applications in industry.

[0004] Action detection, as a traditional task in the field of computer vision, has always been favored by academia and industry. Its purpose is to use computer technology to process, analyze and understand video data, and determine the human actions present in any time segment of a video. These actions include atomic actions such as clapping and high-fives, short actions composed of multiple atomic actions such as jumping and hugging, and long actions such as playing basketball, playing football, and riding a bicycle.

[0005] Deep learning-based action recognition methods can be mainly divided into two types. The first type uses two different feature extraction networks based on dual-stream convolution to model spatial and temporal information respectively. Although this solution can achieve good results for some simple atomic actions, its accuracy is low for short-term and long-term actions. The main reason is that the feature extraction of each channel only focuses on information in a single dimension, either time or space. The second type uses three-dimensional convolution to model both time and space dimensions simultaneously. Compared with dual-stream convolution, this method can extract features in two dimensions at the same time. However, three-dimensional networks generally consider relatively short time intervals, and it is often difficult to judge behavior through important local action changes when faced with long-term actions. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a method, terminal equipment, and storage medium for detecting violations in power production.

[0007] The specific plan is as follows:

[0008] A method for detecting violations in power production includes the following steps:

[0009] S1: Collect images of power production scenarios to construct a training set;

[0010] S2: Build a violation detection model and train the model using a training set;

[0011] The main network of the model uses a SlowFast network. A spatiotemporal channel is added between the slow and fast channels of the SlowFast network. The output of each layer of the slow and fast channels is input into the spatiotemporal channel for spatiotemporal attention fusion. The first spatiotemporal channel takes the spatiotemporal attention fusion result as its output. The spatiotemporal channels of other layers add the spatiotemporal attention fusion result to the output of the previous spatiotemporal channel as their output. The output of the last spatiotemporal channel, together with the outputs of the last slow and fast channels, is input into the global average pooling layer.

[0012] S3: Use the trained model to detect violations in the power production scene images to be tested.

[0013] Furthermore, the power production scene images acquired in step S1 are obtained from video clips and are preprocessed from the video clips. The preprocessing process includes: segmenting the video into segments of fixed time length; selecting one frame from each segmented video frame image as a key frame for annotation; converting the format of the annotated key frame and adding identity tags according to the target tracking method.

[0014] Furthermore, the network architecture used in the slow channel, fast channel, and spatiotemporal channel is ResNet-101, consisting of an input layer, an output layer, and 33 ResNet modules.

[0015] Furthermore, the spatiotemporal attention fusion method involves inputting the outputs of the slow channel and the fast channel into the Spatio Transformer module, inputting the outputs of the slow channel and the fast channel into the Temporal Transformer module, and adding the outputs of the Spatio Transformer module and the Temporal Transformer module to obtain the spatiotemporal attention fusion result.

[0016] Furthermore, the Spatio Transformer module includes a multi-head cross-attention layer, a normalization layer, a multi-layer perceptron layer, and residual connections. Dropout is also added during training to prevent overfitting. For input features of different dimensions, the channel dimension and time dimension are first transformed by 3D convolution to unify them. Then, the time dimension is integrated into the channel dimension by dimensional transformation, and the pixels are integrated into a one-dimensional vector.

[0017] Furthermore, compared to the Spatio Transformer module, the Temporal Transformer module re-changes the dimensions before the multi-head cross-attention layer. While not changing the temporal dimension, it transforms image pixels into one-dimensional vectors and captures time frames with drastic pixel changes through an attention mechanism, thereby locking in important time frames in the fast channel features.

[0018] A terminal device for detecting violations in power production includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the embodiments of the present invention.

[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above in the embodiments of the present invention.

[0020] The present invention adopts the above technical solution, which solves the problem of low accuracy of existing behavior detection methods and ensures safe construction in power production scenarios. Attached Figure Description

[0021] Figure 1 Therefore, this is a schematic diagram of the network structure of the model in this embodiment.

[0022] Figure 2 Therefore, this is a schematic diagram of the network structure of Spatio Transformer in this embodiment.

[0023] Figure 3 Therefore, this is a schematic diagram of the network structure of Temporal Transformer in this embodiment. Detailed Implementation

[0024] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention.

[0025] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0026] Example 1:

[0027] This invention provides a method for detecting violations in power production, the method comprising the following steps:

[0028] S1: Collect images of power production scenarios to build a training set.

[0029] In this embodiment, a hurdling behavior detection dataset based on AVA format was created using hurdling violations in a power production scenario as the background. The dataset contains a total of 132 video clips (each clip is 3 seconds long, with a frame sampling rate of 30 fps). The hurdling behavior is decomposed into fine-grained actions based on the atomic actions of human behavior. All the original actions in the dataset can be divided into three categories (standing, walking, and hurdling). Since the model input is usually in the form of images, the video data needs to be preprocessed. The process mainly includes: (1) segmenting the video into 30 frames per second; (2) selecting one frame from the middle frame of each second as a keyframe and annotating it; (3) converting the format of the annotated keyframe and adding identity labels according to the target tracking method. After data preprocessing, there are a total of 11,513 frames in the entire dataset, of which 397 frames are keyframes for action annotation.

[0030] S2: Build a violation detection model and train the model using a training set.

[0031] In this embodiment, the violation detection model uses an improved network based on the SlowFast network, named ST-SlowFast. The structure of ST-SlowFast is as follows: Figure 1 As shown, its input includes five dimensions: batch dimension B, channel dimension C, number of sampling frames T, image width W, and image height H. Figure 1 The first low frame rate channel is the Slow channel, the third high frame rate channel is the Fast channel, and the channel in between that performs feature fusion through the spatio-temporal attention module is the ST pathway (Spatio-temporal path) added in this embodiment. The specific network layer details are shown in Table 1.

[0032] Table 1

[0033]

[0034]

[0035] In this embodiment, the network structure used for the Slow, Fast, and ST channels is ResNet-101, which includes an input layer, an output layer, and 33 ResNet modules. The spatiotemporal attention module is used for the intermediate feature output layer of each ResNet output. The last column of the table is the T*W*H of the three channels output in each layer.

[0036] The Transformer is an end-to-end attention mechanism module, originally applied in the field of Neuro-Linguistic Programming (NLP) to address the limitations of RNNs in long-term dependencies and slow inference speeds. It assigns weights to fine-grained features through a global attention mechanism, thus resolving the long-term dependency problem. In image classification, the Vision Transformer (VIT) was the first to completely replace the CNN module with a Transformer. It reduces the computational cost of attention by dividing the input image into blocks and ensures the relative positions between blocks by adding learnable parameters. Inspired by this, this embodiment proposes a spatiotemporal Transformer for 3D video stream input to address the adaptive fusion of semantic action features in SlowFast scenarios.

[0037] The structure of Spatio Transformer is as follows: Figure 2 As shown, the input dimensions of the Slow and Fast channels are (B, C, T, W, H) and (B, bC, aT, W, H) respectively (b and a are hyperparameters; b is used to adjust the number of channel dimensions of the Fast channel, which is set to 0.125 in this embodiment; a is used to adjust the sampling frame rate, which is set to 4 in this embodiment). From the structural composition of the Spatio Transformer, it includes a Multi-head Cross Attention (MCA) layer, a Layer Norm (LN) layer, a Multi-Layer Perceptron (MLP) layer, and residual connections. Dropout is added during training to prevent overfitting.

[0038] For input features of different dimensions, the Spatio Transformer first transforms the channel and temporal dimensions using 3D convolution to unify them. Then, to compute the dual-channel spatial attention map, a dimensional transformation is used to integrate the temporal dimension into the channel dimension, while simultaneously integrating pixels into a one-dimensional vector. To ensure the speed of model inference, block-based computation is employed (in this embodiment, the block size is set to 14*14), using linear layers to learn a set of parameters to record the positional information of the pixel blocks. The overall process is shown below:

[0039] fast=Conv3D(fast),fast=LN(fast); (1)

[0040] slow=RS(slow),fast=RS(fast) (2)

[0041] (B,C,T,W,H)——→(W×H,B×T,C); (3)

[0042] out=MCA(slow,fast,fast); (4)

[0043] out mlp =MLP(LN(out)); (5)

[0044] out = out + out mlp (6)

[0045] In equation (2), RS represents the dimension transformation operation, and the transformation form is shown in equation (3); MCA, as a key module in Transformer, can be expressed in the following form:

[0046] Q = O slow W q (7)

[0047] K = O fast W k (8)

[0048] V=O slow W v (9)

[0049]

[0050] O in the formula slow and O fast These represent the stage-wise output features from the Slow and Fast channels, respectively, where d is the feature vector embedding dimension. Scaling operations are used to address the vanishing gradient problem caused by dot product operations, as shown in Table 1. The above equations illustrate the computational process of the attention map. The multi-head attention mechanism is the result of fusing multiple attention mechanisms after dimensional expansion, where each attention head can be represented as:

[0051] head i =Attention(O fast W i q O slow W i k O slow W i v (11)

[0052] In equation (11), the subscript i represents the head number, and the multi-head attention mechanism is expressed as:

[0053]

[0054] In equation (12) This is represented as a concat operation between features, W o These are the weights of the output layer.

[0055] The MLP layer contains two fully connected layers for performing affine transformations on the output of the multi-head attention mechanism, while the LN layer adjusts the input distribution to prevent gradient explosion. The Spatio Transformer, based on the Transformer architecture, is designed for video stream input. It uses an attention mechanism to aggregate semantic and spatial features generated by the Slow and Fast channels in pixel regions, thereby capturing key motion change regions in the Slow channel.

[0056] The structure of Temporal Transformer is as follows: Figure 3As shown, the Temporal Transformer is similar to the Spatio Transformer in terms of structure. However, in order to capture the key frames in the Fast channel frame sequence in the time dimension, the dimension needs to be changed again before the multi-head cross attention mechanism. The input form of the Temporal Transformer is the same as that of the Spatio Transformer. However, the difference is that in order to aggregate in the spatial dimension, the Fast channel features need to be downsampled in the time dimension. The Temporal Transformer, on the other hand, transforms the image pixels into a one-dimensional vector without changing the time dimension, and captures the time frames with drastic pixel changes through the attention mechanism, thereby locking the important time frames in the Fast channel features. Its overall process is similar to the Spatio Transformer, but there are some changes in the RS operation. The details are shown in the following equation (13):

[0057] (B,C,T,W,H)——→(W×H,B×T,C); (13)

[0058] The Spatio-temporal Transformer structure builds an attention-based bridge between dual-channel features, effectively solving the computational resource waste caused by the simple aggregation method of the SlowFast dual-channel mechanism. At the same time, it leverages the advantages of the CNN+Transformer structure, retaining the inductive bias characteristics of CNN while increasing the global attention weight allocation, making the model more robust.

[0059] S3: Use the trained model to detect violations in the power production scene images to be tested.

[0060] By inputting the image of the power production scene to be detected into the trained model, the result of whether there is any violation in the image can be obtained.

[0061] This invention, based on the SlowFast network, proposes ST-SlowFast to address the shortcomings of dual-channel information fusion. It utilizes a specially designed spatiotemporal attention channel to perform fine-grained fusion of dual-channel features at different scales. Simultaneously, the application of an attention mechanism allows the model to adaptively assign weights to the fused features. This information gain path can capture more detailed information between low-frame-rate spatial semantics and high-frame-rate temporal semantics. From an economic perspective, this method can significantly improve the safety awareness of construction workers and reduce the human resource supervision costs for enterprises.

[0062] Example 2:

[0063] The present invention also provides a terminal device for detecting violations in power production, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the method embodiment described in Embodiment 1 of the present invention.

[0064] Furthermore, as an executable solution, the power production violation detection terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The power production violation detection terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described composition of the power production violation detection terminal device is merely an example and does not constitute a limitation on the power production violation detection terminal device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the power production violation detection terminal device may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0065] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the power production violation detection terminal equipment, connecting all parts of the equipment via various interfaces and lines.

[0066] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, realizes various functions of the power production violation detection terminal equipment. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0067] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0068] If the modules / units integrated in the power production violation detection terminal equipment are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0069] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for detecting violations in power production, characterized in that, Includes the following steps: S1: Collect images of power production scenarios to construct a training set; S2: Build a violation detection model and train the model using a training set; The main network of the model adopts the SlowFast network. A spatiotemporal channel is added between the slow and fast channels of the SlowFast network. The output of each layer of the slow and fast channels is input into the spatiotemporal channel for spatiotemporal attention fusion. The first spatiotemporal channel uses the spatiotemporal attention fusion result as the output of this layer; The spatiotemporal channels of all layers other than the first layer add the spatiotemporal attention fusion result to the output of the previous spatiotemporal channel to obtain the output of that spatiotemporal channel; the output of the last spatiotemporal channel, together with the outputs of the last slow channel and the fast channel, is input into the global average pooling layer; the spatiotemporal attention fusion method is as follows: the outputs of the slow channel and the fast channel are input together into the Spatio Transformer module, the outputs of the slow channel and the fast channel are input together into the Temporal Transformer module, and the output of the Spatio Transformer module is added to the output of the Temporal Transformer module to obtain the spatiotemporal attention fusion result; S3: Use the trained model to detect violations in the power production scene images to be tested.

2. The method for detecting violations in power production according to claim 1, characterized in that: The power production scene images collected in step S1 are obtained from video clips, which are preprocessed from the video clips. The preprocessing process includes: segmenting the video into segments of fixed time length; selecting one frame from each segmented video frame image as a keyframe for annotation; converting the format of the annotated keyframes and adding identity tags according to the target tracking method.

3. The method for detecting violations in power production according to claim 1, characterized in that: The network architecture used in the slow channel, fast channel and spatiotemporal channel is ResNet-101, with an input layer, an output layer and 33 ResNet modules.

4. The method for detecting violations in power production according to claim 1, characterized in that: The SpatioTransformer module includes a multi-head cross-attention layer, a normalization layer, a multi-layer perceptron layer, and residual connections. Dropout is added during training to prevent overfitting. For input features of different dimensions, the channel dimension and time dimension are first transformed by 3D convolution to unify them. Then, the time dimension is integrated into the channel dimension by dimensional transformation, and the pixels are integrated into a one-dimensional vector.

5. The method for detecting violations in power production according to claim 1, characterized in that: Compared to the Spatio Transformer module, the TemporalTransformer module re-transforms the dimensions before the multi-head cross-attention layer. It transforms image pixels into one-dimensional vectors without changing the temporal dimension, and captures time frames with drastic pixel changes through an attention mechanism, thereby locking in important time frames in the fast channel features.

6. A terminal device for detecting violations in power production, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images