A power inspection human behavior recognition method, system, device and storage medium

Abstract

The application provides a power inspection human behavior recognition method, system, device and storage medium, the method comprises generating image data to be analyzed and skeleton data to be analyzed according to the video of the inspection personnel to be analyzed obtained by the power inspection unmanned aerial vehicle;The image data to be analyzed and the skeleton data to be analyzed are input into the double-flow human behavior recognition model including the multi-modal feature extraction module, the feature fusion module and the feature classification module connected in turn to carry out behavior recognition, and the corresponding human behavior recognition result is obtained.The application improves more accurate capture of key node information by introducing the multi-head attention ST-GCN++ model, solves the problem of node feature oversmoothing, and combines the multi-modal fusion technology of picture features and skeleton features to extract more comprehensive and accurate behavior features, effectively avoids the interference of the power inspection background on human action recognition, and further improves the accuracy of human behavior recognition in the power inspection scene.

Description

Technical Field

[0001] This invention relates to the field of power line inspection and safety monitoring technology, and in particular to a method, system, computer equipment, and storage medium for identifying human behavior during power line inspections. Background Technology

[0002] The safe and stable operation of a power system has a significant impact on people's production and lives, and the safety inspection of power equipment and lines is an essential task to ensure the normal operation of the power system. Therefore, whether power inspection personnel, who are the main force of power inspection, can perform safe and standardized inspection operations becomes a key factor directly affecting the stable operation of the power system.

[0003] The remote and complex working environment of power line inspection makes it difficult to supervise the operation of inspection personnel on-site. As a result, many abnormal situations of inspection personnel are only discovered after the fact, making it difficult to achieve the effect of prevention. Although video surveillance equipment can meet the monitoring needs to a certain extent, the human behaviors involved in power line inspection are quite complex. Using only RGB video data for behavior analysis is easily affected by the background of the power line inspection. Different weather conditions, lighting levels, and natural environments will have a significant impact on the collected video data, resulting in large deviations in the recognition of human behavior of inspection personnel. Therefore, it cannot truly meet the application needs of power line inspection for accurate recognition of human behavior in various inspection environments and scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method for recognizing human behavior during power line inspections. By introducing the ST-GCN++ model with multi-head attention, it improves the accuracy of capturing key node information. It also combines adaptive multimodal fusion technology of image features and skeletal features to recognize human behavior in power line inspection scenarios. This solves the application defects of existing human behavior recognition methods, which are easily affected by environmental background factors, thus limiting the accuracy of recognition. It effectively avoids the interference of power line inspection background on human action recognition and improves the accuracy of human behavior recognition in power line inspection scenarios.

[0005] To achieve the above objectives, it is necessary to provide a method, system, computer equipment, and storage medium for human behavior recognition in power line inspections, addressing the aforementioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a method for recognizing human behavior during power line inspections, the method comprising the following steps:

[0007] Power line inspection drones were used to acquire videos of inspection personnel for analysis.

[0008] Based on the video of the inspection personnel to be analyzed, generate image data to be analyzed and corresponding skeletal data to be analyzed.

[0009] The image data and skeletal data to be analyzed are input into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and the corresponding human behavior recognition results are obtained. The dual-stream human behavior recognition model includes a multimodal feature extraction module, a feature fusion module and a feature classification module connected in sequence. The multimodal feature extraction module includes a parallel image feature extraction module and a skeletal feature extraction module.

[0010] Furthermore, the step of generating image data to be analyzed and corresponding skeletal data to be analyzed based on the video of the inspector to be analyzed includes:

[0011] The video of the inspection personnel to be analyzed is divided into several non-overlapping sub-videos. One frame is randomly extracted from each sub-video to obtain the initial image data.

[0012] The initial image data is subjected to data augmentation processing to obtain the image data to be analyzed;

[0013] The image data to be analyzed is normalized to obtain normalized image data, and joint point analysis is performed on the normalized image data according to a preset human pose estimation model to obtain the skeleton data to be analyzed.

[0014] Further, the step of inputting the image data to be analyzed and the skeletal data to be analyzed into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and obtaining the corresponding human behavior recognition result, includes:

[0015] The image data to be analyzed is input into the corresponding image feature extraction module for feature extraction to obtain the corresponding image features;

[0016] The bone data to be analyzed is input into the corresponding bone feature extraction module for feature extraction to obtain the corresponding bone features;

[0017] The image features and the skeletal features are input into the feature fusion module for lightweight attention fusion to obtain the corresponding fused features to be classified.

[0018] The features to be fused are input into the feature classification module for feature classification to obtain the human behavior recognition result.

[0019] Furthermore, the image feature extraction module includes a ResNet50 base network and a temporal feature enhancement module connected in sequence; the ResNet50 base network is obtained by adding a time offset module to the bottleneck layer of the ResNet50 network;

[0020] The step of inputting the image data to be analyzed into the corresponding image feature extraction module for feature extraction to obtain the corresponding image features includes:

[0021] The image data to be analyzed is input into the ResNet50 basic network for feature extraction to obtain the corresponding temporal and spatial features.

[0022] The temporal features and spatial features are input into the temporal feature enhancement module for average pooling and flattening operations to obtain the image features.

[0023] Furthermore, the skeletal feature extraction module includes a preset number of consecutively stacked attention-based MA-GCN modules and an average pooling processing module; the attention-based MA-GCN module includes a multi-head attention-based feature extraction module and a residual connection module; the multi-head attention-based feature extraction module is obtained by embedding a multi-head attention module after the ST-GCN++ model; the multi-head attention module includes a joint attention module and a channel attention module;

[0024] The step of inputting the bone data to be analyzed into the corresponding bone feature extraction module for feature extraction to obtain the corresponding bone features includes:

[0025] The skeletal data to be analyzed is input into a preset number of consecutively stacked attention-based MA-GCN modules for spatiotemporal feature extraction to obtain initial skeletal features;

[0026] The initial skeletal features are input into the average pooling processing module for averaging to obtain the skeletal features.

[0027] Furthermore, the feature fusion module includes a feature transformation layer, a feature concatenation layer, a first fully connected layer, a Softmax activation layer, and a matrix multiplication operation layer connected in sequence; the feature transformation layer includes an image feature transformation layer and a skeletal feature transformation layer in parallel; both the image feature transformation layer and the skeletal feature transformation layer include a second fully connected layer and a Tanh activation layer connected in sequence;

[0028] The step of inputting the image features and the skeletal features into the feature fusion module for lightweight attention fusion to obtain the corresponding fused features to be classified includes:

[0029] The image features and the skeletal features are respectively input into the corresponding image feature conversion layer and the skeletal feature conversion layer for feature conversion to obtain the corresponding image conversion features and skeletal conversion features.

[0030] The image transformation features and the skeleton transformation features are input into the feature splicing layer for splicing processing to obtain the corresponding spliced ​​features;

[0031] The splicing features are input into the first fully connected layer for interactive fusion to obtain the corresponding fused features;

[0032] The fused features are input into the Softmax activation layer for activation processing to obtain the corresponding activation features. The activation features and the concatenated features are then input into the matrix multiplication operation layer for further fusion processing to obtain the fused features to be classified.

[0033] Furthermore, the feature classification module includes a random deactivation layer and a fully connected layer;

[0034] The step of inputting the fused features to be classified into the feature classification module for feature classification to obtain the human behavior recognition result includes:

[0035] The fused features to be classified are input into the random deactivation layer for sampling processing to obtain the corresponding features to be classified.

[0036] The features to be classified are input into the fully connected layer for feature mapping to obtain the human behavior recognition result.

[0037] Secondly, embodiments of the present invention provide a human behavior recognition system for power line inspection, the system comprising:

[0038] The data acquisition module is used to acquire videos of the inspection personnel to be analyzed using power line inspection drones.

[0039] The data preprocessing module is used to generate image data to be analyzed and corresponding skeletal data to be analyzed based on the video of the inspection personnel to be analyzed.

[0040] The classification and prediction module is used to input the image data to be analyzed and the skeletal data to be analyzed into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and obtain the corresponding human behavior recognition result; the dual-stream human behavior recognition model includes a multimodal feature extraction module, a feature fusion module and a feature classification module connected in sequence; the multimodal feature extraction module includes a parallel image feature extraction module and a skeletal feature extraction module.

[0041] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0042] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0043] This application provides a method, system, computer equipment, and storage medium for human behavior recognition during power line inspections. The method acquires videos of inspectors using a power line inspection drone, generates image data and corresponding skeletal data based on the videos, and then inputs these data into a pre-constructed dual-stream human behavior recognition model, which includes a multimodal feature extraction module, a feature fusion module, and a feature classification module connected in sequence, to perform behavior recognition and obtain the corresponding human behavior recognition results. Compared with existing technologies, this power line inspection human behavior recognition method improves the accuracy of capturing key node information by introducing a multi-head attention ST-GCN++ model, solving the problem of overly smooth node features. Furthermore, it combines adaptive multimodal fusion technology based on image and skeletal features to fully extract the temporal, background, object, and spatiotemporal information features of the video data, effectively avoiding interference from the power line inspection background on human action recognition and improving the accuracy of human behavior recognition in power line inspection scenarios. Attached Figure Description

[0044] Figure 1 This is a schematic diagram illustrating the application scenario of the human behavior recognition method for power line inspection in this embodiment of the invention;

[0045] Figure 2 This is a schematic diagram of the process framework for human behavior recognition during power line inspections in an embodiment of the present invention;

[0046] Figure 3 This is a flowchart illustrating the human behavior recognition method for power line inspection in an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram of the skeletal feature extraction module in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of the multi-head attention module in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of the joint attention module in an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the channel attention module in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the feature fusion module in an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the structure of the human behavior recognition system for power line inspection in an embodiment of the present invention;

[0053] Figure 10 This is an internal structural diagram of the computer device in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and beneficial effects of this application clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the embodiments described below are only part of the embodiments of the present invention and are used to illustrate the present invention, but are not intended to limit the scope of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0055] The power line inspection human behavior recognition method provided by this invention can be understood as a dual-stream human behavior recognition method that extracts image features and skeletal features from the video of the inspection personnel to be analyzed, and then performs multimodal complementary fusion of image features containing rich temporal, background, and object information with skeletal features containing rich spatiotemporal information for human behavior recognition and classification. This method can be applied to applications such as... Figure 1 The terminal or server shown. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The server can be implemented using the features provided by this invention, depending on the actual application requirements. Figure 2 The power line inspection human behavior recognition architecture shown performs efficient and accurate human behavior recognition and classification, and uses the obtained human behavior recognition results for subsequent research on the server, or transmits them to the terminal for terminal users to view and analyze; the following embodiments will describe the power line inspection human behavior recognition method of the present invention in detail.

[0056] In one embodiment, such as Figure 3 As shown, a method for recognizing human behavior during power line inspections is provided, including the following steps:

[0057] S11. Obtain video of the inspectors to be analyzed through power line inspection drones; the video of the inspectors to be analyzed can be understood as video data collected in real time by drones deployed in the power line inspection scene. The specific method of collecting video data by deploying drones can be set according to actual application needs, and no specific limitation is made here.

[0058] S12. Based on the video of the inspector to be analyzed, generate image data to be analyzed and corresponding skeletal data to be analyzed; wherein, the image data to be analyzed can be understood as RGB data generated by first extracting frames from the RGB video and then performing data augmentation to obtain image data that can meet the needs of subsequent human behavior recognition; the corresponding skeletal data to be analyzed can be understood as skeletal depth data that can accurately describe human movements and is not easily affected by environmental factors such as lighting, obtained by analyzing human joint points based on the image data to be analyzed;

[0059] Specifically, the step of generating image data to be analyzed and corresponding skeletal data to be analyzed based on the video of the inspector to be analyzed includes:

[0060] The video of the inspectors to be analyzed is divided into several non-overlapping sub-videos. One frame is randomly extracted from each sub-video to obtain the initial image data. The number of sub-videos can be determined according to the actual application requirements, and is usually consistent with the frame sampling strategy used by the image feature extraction module used later. For example, the input video of the inspectors to be analyzed is divided into 8 non-overlapping sub-videos, and one frame is randomly extracted from each sub-video to obtain RGB input data (initial image data) with a size of 8×3×320×240. At the same time, in order to mitigate the influence of background, lighting and other factors on the human movement trajectory in the video of the inspectors to be analyzed, this embodiment preferably performs data augmentation processing on each frame extracted from the video using the following method to improve the processing performance of the subsequent application model.

[0061] The initial image data is subjected to data augmentation processing to obtain the image data to be analyzed; wherein, the data augmentation processing includes: first, resizing each initial image data to adjust its resolution to 224×224, and then applying a random horizontal flip operation with a flip probability of 0.5 to the image after resolution adjustment to obtain the image data to be analyzed;

[0062] The image data to be analyzed is normalized to obtain normalized image data, and joint point analysis is performed on the normalized image data according to a preset human pose estimation model to obtain the skeleton data to be analyzed; wherein, the normalized image data can be understood as the image obtained by normalizing the enhanced image and the corresponding initial image data in each image data to be analyzed according to the corresponding mean and variance.

[0063] The aforementioned preset human pose estimation model can, in principle, employ any human pose estimation model. However, to address the detail issues in high-resolution images and ensure the effectiveness of skeletal data extraction, this embodiment preferably uses the HR-Net model, which consists of a four-layer cascaded structure, as the human pose estimation model. In each cascaded stage, the basic sub-modules of HR-Net are composed of multiple bottleneck and basic units stacked together. Furthermore, each basic unit contains multiple convolutional layers that employ channel-separated convolutions to reduce the number of parameters and computational load. This model can perform layer-by-layer feature extraction and fusion from low to high resolution, increasing resolution through upsampling. While preserving high-resolution information, it can also consider both spatial and channel information through a special bilateral convolutional structure, improving the expressive power of skeletal data and ultimately obtaining high-precision pose estimation results. It should be noted that when using the HR-Net model to extract skeletal data, if there are multiple people in the video, the joint data of the two people with the highest confidence are selected as the skeletal data.

[0064] S13. Input the image data to be analyzed and the skeletal data to be analyzed into a pre-constructed dual-stream human action recognition model for action recognition to obtain the corresponding human action recognition results; wherein, the dual-stream human action recognition model (Skeleton-RGB Based Human Action Recognition, SRHAR model) can be understood as a pre-constructed network model that can extract image features and skeletal features from the video of the inspector to be analyzed, and then perform multimodal complementary fusion of image features containing rich temporal, background and object information and skeletal features containing rich spatiotemporal information to obtain fused features for action classification and recognition; specifically, the dual-stream human action recognition model includes a multimodal feature extraction module, a feature fusion module and a feature classification module connected in sequence; the multimodal feature extraction module includes a parallel image feature extraction module and a skeletal feature extraction module;

[0065] Specifically, the step of inputting the image data to be analyzed and the skeletal data to be analyzed into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and obtaining the corresponding human behavior recognition result, includes:

[0066] It should be noted that, in order to better apply the dual-stream human behavior recognition model (two-dimensional convolutional network) selected in this embodiment, it is necessary to perform stacking processing in the time dimension before inputting the image data and skeleton data to be analyzed into the dual-stream human behavior recognition model, and then input the corresponding data obtained after stacking processing into the dual-stream human behavior recognition model for analysis. For example, if the batch-size is set to 4, the input shape of the image data to be analyzed is 32×3×224×224, and the input shape of the skeleton data to be analyzed is 4×3×50×17×2.

[0067] The image data to be analyzed is input into the corresponding image feature extraction module for feature extraction to obtain the corresponding image features. The image feature extraction module can be arbitrarily selected according to requirements, but in order to effectively balance recognition accuracy and running speed, this embodiment preferably uses ResNet50 as the base model for construction and adjustment to obtain a model for efficient and reliable feature extraction of RGB data. In practical applications, the specific structure and parameters can be constructed and adjusted according to different tasks. Specifically, the image feature extraction module includes a ResNet50 base network and a temporal feature enhancement module connected in sequence. The ResNet50 base network is obtained by adding a time offset module to the bottleneck layer of the ResNet50 network.

[0068] The ResNet50 consists of a 50-layer deep convolutional neural network, enabling the convolutional layers to extract low-level, mid-level, and high-level features from the input data. It also employs a residual structure to allow direct cross-layer information transfer within the network, effectively transmitting gradients and preventing gradient explosion or vanishing problems. Specifically, the step of inputting the image data to be analyzed into the corresponding image feature extraction module for feature extraction to obtain the corresponding image features includes:

[0069] The image data to be analyzed is input into the ResNet50 basic network for feature extraction, yielding corresponding temporal and spatial features. These temporal and spatial features can be understood as the short-term temporal and spatial features of the video data extracted by the Time-Shifted Model (TSM) module, embedded in the bottleneck layer (Bottleneck module) of the ResNet50 backbone network, through two parallel branches. Specifically, the TSM module separates the input features X∈R using a channel separation method. NT ×C×H×W Separate into X according to channel dimension spatial and X temporal Two parts, of which X temporal Used for time modeling, X spatial Used for spatial modeling;

[0070] The temporal and spatial features are input into the temporal feature enhancement module for average pooling and flattening operations to obtain the image features. The temporal feature enhancement module, added to the tail of the ResNet50 base network, aims to extract long-term temporal features from the input data and highlight more important frames. It does this by optimizing the feature vector output by the time offset module (TSM module). MBSTFE ∈R NT×C×H×W Perform average pooling and flattening operations to obtain the feature vector X. LSTM ∈R NT×C Then use LSTM layers to process X LSTM Long-term time-series modeling is performed to extract more abstract RGB feature vectors, which are the image features required for subsequent recognition and prediction.

[0071] The skeletal data to be analyzed is input into the corresponding skeletal feature extraction module for feature extraction to obtain the corresponding skeletal features. The skeletal feature extraction module can be understood as a skeletal data extraction model designed to address the problem of existing GCN models easily confusing actions with similar motion trajectories. The skeletal feature extraction module includes a preset number of consecutively stacked attention-based MA-GCN (Multi Attention Graph Convolutional Networks) modules and an average pooling processing module. Figure 4 As shown, the attention-based MA-GCN module includes a feature extraction module based on multi-head attention and a residual connection module, and the feature extraction module based on multi-head attention is obtained by embedding the multi-head attention module after the ST-GCN++ model;

[0072] The ST-GCN++ model described above can be understood as a spatiotemporal graph convolutional network model obtained by combining graph convolutional networks (GCN) and temporal convolutional networks (TCN). It adopts a multi-branch structure to construct the TCN temporal convolutional network. In this embodiment, the TCN temporal convolutional network preferably adopts a multi-stage TCN network (MS-TCN), which includes five processing branches, such as four 3×1 convolutions with increasing dilatation rates and one 3×1 max pooling. The corresponding number of parameters and computational cost are as follows:

[0073]

[0074] In the formula, FLOPs mstcn and Params mstcn These represent the computational cost of the convolution operation; C in Input the number of channels;

[0075] At the same time, for input features In contrast, the computational cost and parameter count of a traditional TCN module are as follows:

[0076]

[0077] In the formula, FLOPs tcn and Params tcn These represent the computational cost of the convolution operation; C in Input the number of channels;

[0078] Comparative analysis clearly shows that this embodiment, employing the MS-TCN model, effectively reduces the number of model parameters and computational load. The computational load is reduced to 38.89% of that of the traditional TCN module, and the number of parameters is reduced to approximately 24.07% of that of the traditional TCN module. Moreover, as shown below, it also improves the model's temporal modeling capabilities:

[0079]

[0080] In the formula, γ f γ represents the rate of reduction in the computational cost of the model. p This indicates the rate at which the number of model parameters has decreased;

[0081] The multi-attention (MA) module used in this embodiment can be understood as an adaptive attention module that integrates the calculation of keypoint attention scores and channel attention scores. Figure 5 As shown, the multi-head attention module includes a joint attention module (JA module) and a channel attention module (CA module); wherein, the joint attention JA module, as shown... Figure 6 As shown, for an input feature containing C channels, T frames, and V keypoints, F∈R C×T×V Average pooling is performed on both the time and keypoint dimensions to obtain the keypoint feature vector F. j ∈R C×V With time eigenvector F t ∈R C×T Then, these two weighted attention vectors are concatenated into a mixed feature vector F. f ∈R C×(V+T) After integrating temporal and keypoint information through a 1×1 convolution, two independent 1×1 convolutions are used to obtain joint attention feature maps and temporal attention feature maps, respectively. These two maps are then multiplied element-wise to obtain the final attention feature map, `out`. Joint ∈R C×T×V Correspondingly, the specific processing procedure can be represented as follows:

[0082]

[0083] In the formula, F f ∈R C×(V+T) For the mixed feature vector, out Joint ∈R C×T×V This is the final attention feature map; Concate() is the concatenation operation; TemporalPool() is the average pooling operation in the time dimension; JointPool() is the average pooling operation in the key dimension; Split() is the splitting operation; * is the element-wise multiplication operation; The superscript x represents the kernel size, and the subscript y represents the dimension of the convolution; σ(·) represents the Sigmoid activation function.

[0084] The channel attention CA module in this embodiment, such as Figure 7 As shown, max pooling and average pooling operations are performed on the input feature F along the channel dimension to obtain two channel masks of shape C×1×1. After concatenation, the masks are then processed by 1×1 convolution and sigmoid activation to obtain the channel attention vector. The specific processing can be represented as follows:

[0085]

[0086] In the formula, out channel This is the channel attention score vector obtained after processing by the channel attention CA module; MAXPool() is the maximum pooling operation; GAPool() represents the global average pooling operation.

[0087] In practical applications, such as Figure 5 As shown, for input feature F, the multi-head attention module (MA module) first calculates the joint attention score through the joint attention JA module, and then multiplies it element-wise with the input feature F to obtain the intermediate feature F. middle Then, the intermediate feature F middle The input is fed into the channel attention CA module for calculation and then compared with the intermediate feature F. middle Element-wise multiplication yields an output feature of the same size as the input feature; the specific computational steps within the corresponding multi-head attention (MA) module are as follows:

[0088]

[0089] In the formula, JA() represents the operation of the JA module for joint attention; CA() represents the operation of the CA module for channel attention; out MA This represents the final output of the multi-head attention (MA) module;

[0090] Based on the functional descriptions of the above modules, it can be seen that, Figure 4As shown, the MA-GCN module divides the input features into two paths: one path consists of a single ST-GCN module and a multi-head attention module (MA) based on an attention mechanism; the other path is a residual connection, which directly connects the input features to the end of the module, improving the model's spatial modeling ability. Finally, the two paths are element-wise summed to obtain the output features. The specific processing of the MA-GCN module can be represented as follows:

[0091]

[0092] In the formula, F input and out MA-Gcn These represent the input and output features of the MA-GCN module, respectively; F stgcn++ This indicates the output characteristics after passing through the ST-Gcn++ module; F MA The output features of the multi-head attention (MA) module are represented; STGcnPlus() represents the ST-Gcn++ model operation; MA() represents the multi-head attention (MA) module operation.

[0093] Specifically, the step of inputting the skeletal data to be analyzed into the corresponding skeletal feature extraction module for feature extraction to obtain the corresponding skeletal features includes:

[0094] The skeletal data to be analyzed is input into a preset number of consecutively stacked attention-based MA-GCN modules for spatiotemporal feature extraction to obtain initial skeletal features. The preset number can be selected according to actual application needs and is not specifically limited here. For example, MA-GCN modules with 10 consecutively stacked layers can be used to extract the spatiotemporal features of the skeletal data, and the number of output channels of each layer is the same as the number of output channels of the multi-head attention MA module.

[0095] The initial skeletal features are input into the average pooling processing module for averaging to obtain the skeletal features. The average pooling processing module can be understood as averaging the initial skeletal features extracted by a preset number of consecutively stacked attention-based MA-GCN modules, obtaining the output feature shape of the MA-GCN model according to the output requirements, and finally obtaining the required skeletal features.

[0096] Furthermore, to verify the effectiveness of the skeletal feature extraction module selected in this embodiment, an effectiveness ablation experiment was conducted on the MA-GCN module, using the ST-GCN++ model as the baseline. Experimental results show that deleting any sub-module from the MA module degrades the model's performance. When the JA and CA sub-modules were removed from the MA module, the accuracy decreased by 0.74% and 0.35% respectively under the Cross-View standard, and by 0.48% and 0.30% respectively under the Cross-Subject standard. When the two sub-modules were fused together, the model achieved performance 1.09% and 0.92% higher than the baseline, indicating that the two sub-modules are complementary. This demonstrates that the MA module can perceive changes in motion trajectory from different dimensions of skeletal data, thereby improving the model's recognition accuracy. Simultaneously, the selected... Using the ST-GCN++ model as a baseline, the impact of the number of layers in the MA-GCN module on the model's recognition accuracy was investigated. Experimental results show that when the number of layers in the MA-GCN module of this invention is greater than 7, the corresponding MA-GCN model shows varying degrees of improvement in Top-1 accuracy on both Cross-View and Cross-Subject standards as the number of module layers increases. Considering that stacking the ms-tcn module to a certain depth may lead to problems such as oversmoothing of node features, model overfitting, and increased training difficulty due to the increased number of parameters, continuously stacking multiple layers of MA-GCN modules to extract the spatiotemporal features of skeletal data can alleviate this problem to some extent, achieving relatively ideal feature extraction results. Furthermore, the number of layers of the MA-GCN module can be adjusted according to the needs of the actual task to improve the accuracy of behavior recognition.

[0097] This embodiment extracts skeletal features using the ST-GCN++ algorithm with an embedded multi-path attention (MA) module. The introduction of the MA module can capture information of key nodes more accurately, enabling downstream tasks to learn better feature representations and making feature learning more valuable.

[0098] After obtaining image features and skeletal features through the above steps, a multimodal feature fusion approach can be adopted. This involves inputting the image features and skeletal features into the feature fusion module for lightweight attention fusion to obtain the corresponding fused features to be classified. Lightweight attention fusion (LAF) is a preferred multimodal feature fusion method in this embodiment to address the application deficiency of existing feature fusion methods in achieving interaction between different modal features. It can achieve interaction between different modal features with only a few additional parameters, and through the attention mechanism, the model can adaptively learn the score weights of different modal features to better fuse the two modal features, enhancing the model's feature representation ability and thus improving the model's recognition accuracy.

[0099] The aforementioned feature fusion module can be understood as an adaptive learning module that learns the score weights of different modal features through independent fully connected layers and attention mechanisms, completing the interaction and fusion of image RGB and skeletal data, and using the Softmax function to activate and generate fused features for classification; specifically, such as Figure 8 As shown, the feature fusion module includes a feature transformation layer, a feature concatenation layer, a first fully connected layer, a Softmax activation layer, and a matrix multiplication operation layer connected in sequence; the feature transformation layer includes a parallel image feature transformation layer and a skeletal feature transformation layer; both the image feature transformation layer and the skeletal feature transformation layer include a second fully connected layer and a Tanh activation layer connected in sequence; correspondingly, the step of inputting the image features and the skeletal features into the feature fusion module for lightweight attention fusion to obtain the corresponding fused features to be classified includes:

[0100] The image features and the skeletal features are respectively input into the corresponding image feature conversion layer and skeletal feature conversion layer for feature conversion to obtain the corresponding image conversion features and skeletal conversion features. The image conversion features and skeletal conversion features can be understood as features obtained by dimensionality conversion through corresponding independent fully connected layers and Tanh activation layers, and the image conversion features and skeletal conversion features have the same dimension, which facilitates subsequent stitching and fusion.

[0101] The image transformation features and the skeleton transformation features are input into the feature concatenation layer for concatenation processing to obtain the corresponding concatenated features; wherein, the concatenated features can be understood as intermediate features F obtained by combining the image transformation features and skeleton transformation features obtained above. concate ∈R 2×d ;

[0102] The splicing features are input into the first fully connected layer for interactive fusion to obtain the corresponding fused features;

[0103] The fused features are input into the Softmax activation layer for activation processing to obtain the corresponding activation features. The activation features and the concatenated features are then input into the matrix multiplication operation layer for further fusion processing to obtain the fused features to be classified.

[0104] In practical applications, the process of obtaining the fused features to be classified through the feature fusion LAF module can be represented as:

[0105]

[0106] In the formula: F rgb RGB features; F skeleton For skeletal features; Linear() is a fully connected layer; δ() is the Tanh activation function; F concate This represents the activation features obtained after processing by the Softmax activation layer; out LAF This represents the final fused features to be classified, which are then used for classification.

[0107] It should be noted that, in order to ensure that the model is lightweight enough, the output dimension of the two transformation layers in the head of the feature fusion LAF module, namely the image feature transformation layer and the skeleton feature transformation layer, is set to 128. After transformation, the size of the image RGB features and the skeleton features are both 1×128. At the same time, the first fully connected layer in the feature fusion LAF module, which plays an interactive role, has its output dimension set to 1, and the shape of the output feature is 1×2. After subsequent softmax activation, it can complete the subsequent matrix multiplication operation. The final obtained dual-stream fusion feature based on spatiotemporal information has a shape of 1×128.

[0108] This embodiment achieves feature fusion of the acquired RGB image features and skeletal features using the lightweight attention fusion module LAF, enabling interaction of different modal features with only a few additional parameters, thus better fulfilling the human behavior recognition task.

[0109] The fused features to be classified are input into the feature classification module for feature classification to obtain the human behavior recognition result. The feature classification module can be understood as a processing module that maps the fused features to be classified to specific action categories and outputs a dual-stream human behavior recognition result based on spatiotemporal information. Specifically, the feature classification module includes a random deactivation Dropout layer and a fully connected layer (FC). The random deactivation layer can be selected based on actual application requirements. Correspondingly, the step of inputting the fused features to be classified into the feature classification module for feature classification to obtain the human behavior recognition result includes:

[0110] The fused features to be classified are input into the random deactivation layer for sampling processing to obtain the corresponding features to be classified.

[0111] The features to be classified are input into the fully connected layer for feature mapping to obtain the human behavior recognition result.

[0112] It should be noted that the dual-stream human behavior recognition model used in this embodiment is obtained by training an initial network model with the aforementioned modules using the cross-entropy loss function commonly used in multi-class classification problems (the loss function can be adjusted according to different task requirements):

[0113]

[0114] In the formula, N is the total number of samples; M is the number of categories; y ic Let y represent the predicted label of the i-th sample. If the predicted category c is the same as the true label of the sample, then y ic If y is 1, otherwise y ic p is 0; ic This is the probability that the i-th sample in the probability vector belongs to class c.

[0115] Furthermore, to ensure the effectiveness of the power line inspection human behavior recognition method provided in this application, the following comprehensive evaluation methods can be employed: Top-1 accuracy as the model performance evaluation metric, memory access volume as the model's space complexity evaluation metric, and efficiency metrics as the model's lightweight level evaluation metric to assess the model's reliability. Based on these evaluation results, the model structure and / or parameters can be optimized and adjusted accordingly.

[0116] Top-1 accuracy is an evaluation metric obtained by matching the human behavior recognition results (the maximum probability of belonging to each category) generated by the model for each input test sample with the real behavior label classification (if the human behavior recognition result is the same as the real category corresponding to the sample data, the preset is considered correct).

[0117] The memory access volume of a model can be understood as the number of parameters in the model. The dual-stream human behavior recognition model (SRHAR model) proposed in this invention mainly consists of three parts: convolutional layers, fully connected layers, and the MA-GCN model. The calculation method for the number of parameters in the convolutional layers and fully connected layers is as follows:

[0118]

[0119] In the formula, k 2 ×C in C represents the number of parameters in a convolution kernel; +1 represents the number of parameters in the biased kernel; C out I represents the number of output channels; I represents the input dimension of the fully connected layer; O represents the output dimension of the fully connected layer.

[0120] The parameters of the MA-GCN model mainly consist of two parts: the parameters of the convolutional layers and the variables that need to be stored for the graph convolution operation. Its space complexity is:

[0121]

[0122] In the formula, the first term is the space complexity of the variables that need to be stored for the graph convolution operation, and the second term is the space complexity of all convolutional layer parameters.

[0123] The computational complexity of the MA-GCN model is concentrated in four parts: the gcn module, the MS-TCN module, the MA module, and the weight matrix. Its time complexity is:

[0124]

[0125] In the formula, the first term is the time complexity of the weight matrix multiplication, the second term is the time complexity of all convolutional layers, and the third term is the time complexity of the feature fusion operation between the adjacency matrix and the input features in the gcn module.

[0126] The lightweight efficiency index η represents the number of parameters required to improve the Top-1 accuracy by one unit. A lower η value indicates higher efficiency. The corresponding calculation formula is as follows:

[0127]

[0128] In the formula, ΔParams represents the change in the number of parameters, and ΔTop-1 represents the change in Top-1 accuracy.

[0129] This application embodiment realizes a drone-based dual-stream human behavior recognition scheme that acquires videos of inspectors to be analyzed using power line inspection drones, extracts modal data from the videos to generate image data and corresponding skeletal data to be analyzed, and then inputs the image data and skeletal data to be analyzed into a pre-constructed dual-stream human behavior recognition model, which includes a multimodal feature extraction module, a feature fusion module, and a feature classification module connected in sequence, to perform behavior recognition and obtain the corresponding human behavior recognition results. This scheme not only introduces skeletal modal data, enabling complementary information between RGB modalities, but also provides the model with rich human behavior information and background information from the input video through the image feature extraction branch. Combined with the skeletal feature extraction branch improved by incorporating multi-head attention, which provides the model with accurate human joint data, the feature fusion module completes feature interaction between different modalities, effectively combining the advantages of two data modalities to more comprehensively describe human actions. Furthermore, the attention mechanism adaptively learns the fusion strategy, effectively improving the model's recognition accuracy. This effectively avoids interference from the power line inspection background on human action recognition and improves the accuracy of human behavior recognition in power line inspection scenarios.

[0130] It should be noted that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0131] In one embodiment, such as Figure 9 As shown, a human behavior recognition system for power line inspection is provided, the system comprising:

[0132] Data acquisition module 1 is used to acquire videos of the inspection personnel to be analyzed through power line inspection drones;

[0133] Data preprocessing module 2 is used to generate image data to be analyzed and corresponding skeletal data to be analyzed based on the video of the inspection personnel to be analyzed;

[0134] The classification prediction module 3 is used to input the image data to be analyzed and the skeletal data to be analyzed into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and obtain the corresponding human behavior recognition result; the dual-stream human behavior recognition model includes a multimodal feature extraction module, a feature fusion module and a feature classification module connected in sequence; the multimodal feature extraction module includes a parallel image feature extraction module and a skeletal feature extraction module.

[0135] Specific limitations regarding the human behavior recognition system for power line inspections can be found in the limitations of the human behavior recognition method for power line inspections described above; the corresponding technical effects are equivalent and will not be repeated here. Each module in the aforementioned human behavior recognition system for power line inspections can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0136] Figure 10 An internal structural diagram of a computer device is shown in one embodiment. This computer device may specifically be a terminal or a server. Figure 10 As shown, the computer device includes a processor, memory, network interface, display, camera, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a human behavior recognition method for power line inspection. The display screen can be an LCD screen or an e-ink display screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0137] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computing devices may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0138] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0140] In summary, the present invention provides a method and system for human behavior recognition during power line inspections. The method acquires videos of inspectors to be analyzed using a power line inspection drone, generates image data and corresponding skeletal data based on the video, and then inputs these data into a pre-constructed dual-stream human behavior recognition model comprising a multimodal feature extraction module, a feature fusion module, and a feature classification module connected sequentially. This results in the corresponding human behavior recognition outcome. This method not only introduces skeletal modal data, enabling complementary information between the RGB modalities, but also provides the model with rich human behavior and background information from the input video through image feature extraction branches. Based on the improved skeletal feature extraction branch obtained by incorporating multi-head attention into the ST-GCN++ model, which provides accurate human joint data for the model, the feature fusion module completes the feature interaction between different modalities, effectively combining the advantages of the two data modalities to more comprehensively describe human actions. Furthermore, the attention mechanism is used to adaptively learn the fusion strategy, effectively improving the model's recognition accuracy. Specifically, the ST-GCN++ model improvement by incorporating multi-head attention more accurately captures key node information, and the adaptive multimodal fusion technology based on image features and skeletal features fully extracts the temporal, background, object, and spatiotemporal information features of video data, effectively avoiding interference from the background of power inspection on human action recognition and improving the accuracy of human behavior recognition in power inspection scenarios.

[0141] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0142] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A method for recognizing human behavior during power line inspections, characterized in that, The method includes the following steps: Power line inspection drones were used to acquire videos of inspection personnel for analysis. Based on the video of the inspection personnel to be analyzed, generate image data to be analyzed and corresponding skeletal data to be analyzed. The image data and skeleton data to be analyzed are input into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and the corresponding human behavior recognition results are obtained. The dual-stream human behavior recognition model includes a multimodal feature extraction module, a feature fusion module, and a feature classification module connected in sequence. The multimodal feature extraction module includes a parallel image feature extraction module and a skeleton feature extraction module. The step of inputting the image data to be analyzed and the skeletal data to be analyzed into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and obtaining the corresponding human behavior recognition result, includes: The image data to be analyzed is input into the corresponding image feature extraction module for feature extraction to obtain the corresponding image features; The skeletal data to be analyzed is input into the corresponding skeletal feature extraction module for feature extraction to obtain the corresponding skeletal features. The skeletal feature extraction module includes a preset number of consecutively stacked attention-based MA-GCN modules and an average pooling processing module. The attention-based MA-GCN module includes a feature extraction module based on multi-head attention and a residual connection module. The feature extraction module based on multi-head attention is obtained by embedding a multi-head attention module after the ST-GCN++ model. The multi-head attention module includes a joint attention module and a channel attention module. The ST-GCN++ model is a spatiotemporal graph convolutional network model obtained by combining graph convolutional networks and temporal convolutional networks. The image features and the skeletal features are input into the feature fusion module for lightweight attention fusion to obtain the corresponding fused features to be classified. The features to be classified and fused are input into the feature classification module for feature classification to obtain the human behavior recognition result; The step of inputting the bone data to be analyzed into the corresponding bone feature extraction module for feature extraction to obtain the corresponding bone features includes: The skeletal data to be analyzed is input into a preset number of consecutively stacked attention-based MA-GCN modules for spatiotemporal feature extraction to obtain initial skeletal features; The initial skeletal features are input into the average pooling processing module for averaging to obtain the skeletal features.

2. The power line inspection human behavior recognition method as described in claim 1, characterized in that, The step of generating image data to be analyzed and corresponding skeletal data to be analyzed based on the video of the inspection personnel to be analyzed includes: The video of the inspection personnel to be analyzed is divided into several non-overlapping sub-videos. One frame is randomly extracted from each sub-video to obtain the initial image data. The initial image data is subjected to data augmentation processing to obtain the image data to be analyzed; The image data to be analyzed is normalized to obtain normalized image data, and joint point analysis is performed on the normalized image data according to a preset human pose estimation model to obtain the skeleton data to be analyzed.

3. The power line inspection human behavior recognition method as described in claim 1, characterized in that, The image feature extraction module includes a ResNet50 base network and a temporal feature enhancement module connected in sequence; the ResNet50 base network is obtained by adding a time offset module to the bottleneck layer of the ResNet50 network; The step of inputting the image data to be analyzed into the corresponding image feature extraction module for feature extraction to obtain the corresponding image features includes: The image data to be analyzed is input into the ResNet50 basic network for feature extraction to obtain the corresponding temporal and spatial features. The temporal features and spatial features are input into the temporal feature enhancement module for average pooling and flattening operations to obtain the image features.

4. The power line inspection human behavior recognition method as described in claim 1, characterized in that, The feature fusion module includes a feature transformation layer, a feature splicing layer, a first fully connected layer, a Softmax activation layer, and a matrix multiplication operation layer connected in sequence; the feature transformation layer includes an image feature transformation layer and a skeleton feature transformation layer in parallel; both the image feature transformation layer and the skeleton feature transformation layer include a second fully connected layer and a Tanh activation layer connected in sequence; The step of inputting the image features and the skeletal features into the feature fusion module for lightweight attention fusion to obtain the corresponding fused features to be classified includes: The image features and the skeletal features are respectively input into the corresponding image feature conversion layer and the skeletal feature conversion layer for feature conversion to obtain the corresponding image conversion features and skeletal conversion features. The image transformation features and the skeleton transformation features are input into the feature splicing layer for splicing processing to obtain the corresponding spliced ​​features; The splicing features are input into the first fully connected layer for interactive fusion to obtain the corresponding fused features; The fused features are input into the Softmax activation layer for activation processing to obtain the corresponding activation features. The activation features and the concatenated features are then input into the matrix multiplication operation layer for further fusion processing to obtain the fused features to be classified.

5. The power line inspection human behavior recognition method as described in claim 1, characterized in that, The feature classification module includes a random deactivation layer and a fully connected layer; The step of inputting the fused features to be classified into the feature classification module for feature classification to obtain the human behavior recognition result includes: The fused features to be classified are input into the random deactivation layer for sampling processing to obtain the corresponding features to be classified. The features to be classified are input into the fully connected layer for feature mapping to obtain the human behavior recognition result.

6. A human behavior recognition system for power line inspection, characterized in that, The power line inspection human behavior recognition system as described in claim 1, wherein the system comprises: The data acquisition module is used to acquire videos of the inspection personnel to be analyzed using power line inspection drones. The data preprocessing module is used to generate image data to be analyzed and corresponding skeletal data to be analyzed based on the video of the inspection personnel to be analyzed. The classification and prediction module is used to input the image data to be analyzed and the skeletal data to be analyzed into a pre-constructed dual-stream human behavior recognition model for behavior recognition, and obtain the corresponding human behavior recognition result; the dual-stream human behavior recognition model includes a multimodal feature extraction module, a feature fusion module and a feature classification module connected in sequence; the multimodal feature extraction module includes a parallel image feature extraction module and a skeletal feature extraction module.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images