Telegraph pole tower climbing behavior monitoring method, system, equipment and medium

Abstract

The invention discloses a telegraph pole tower climbing behavior monitoring method, system and device and a medium. The method comprises the steps that an image sequence of a telegraph pole tower area is obtained through a camera; preprocessing each frame of area image in the image sequence to generate a preprocessed image; sequentially calling a space-time diagram convolution model, a self-adaptive optimization model and a spatial configuration model, and respectively extracting a joint point feature matrix from each preprocessed image; calling a convolutional neural network to classify each joint point feature matrix, and judging whether a target feature conforming to a climbing behavior exists or not; calling a long short-term memory network to predict each target feature, and determining climbing trend information; and if the target characteristics exist or the climbing trend information reflects that the climbing behavior is continuously carried out, outputting an early warning signal and feeding back a response strategy so as to stably and reliably reflect the real state of the telegraph pole tower and effectively realize real-time monitoring of the transmission tower.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior monitoring, and in particular to a method, system, equipment and medium for monitoring electric pole tower climbing behavior. Background Art

[0002] In today's society, electricity, as a vital energy source, is widely used in various fields and is the cornerstone for maintaining normal social operation and economic development. From bustling urban commercial areas to remote rural areas, from daily lighting and household appliance use to the operation of various industrial production processes, the stability and reliability of power supply play a decisive role. Once a power supply failure occurs, it not only causes numerous inconveniences to people's lives, but can also cause huge economic losses to industrial production and even affect the normal operation of key areas such as healthcare and transportation. Therefore, ensuring the safe and stable operation of the power system is a paramount task.

[0003] Throughout the power system, utility towers, as key support structures for transmission lines, carry the crucial mission of transmitting electricity from power plants to various power-consuming areas. With the continuous growth of electricity demand and the continued expansion of the power grid, the number of utility towers is also increasing, making effective monitoring of utility towers increasingly critical. Promptly detecting abnormal conditions on utility towers is crucial for preventing power accidents and ensuring the safe and stable operation of the power system.

[0004] Currently, power pole monitoring technologies primarily fall into two categories: video surveillance and sensor monitoring. Video surveillance captures images of the tower area and analyzes them using manual or automated algorithms to detect abnormal behavior. Sensor monitoring, on the other hand, installs various sensors on the tower to monitor its status and surrounding environmental changes in real time. However, existing solutions suffer from poor image quality due to the effects of outdoor lighting, wind, rain, and obstructions. Sensor monitoring is also susceptible to environmental noise and electromagnetic interference, resulting in inaccurate data and an inability to reliably and stably reflect the true status of the power pole. Summary of the Invention

[0005] The present invention provides a method, system, device and medium for monitoring climbing behavior on utility poles and towers, which solves the technical problems in existing solutions, in which video monitoring is affected by factors such as outdoor lighting, wind and rain, and obstructions, resulting in poor image acquisition quality, and sensor monitoring is also easily affected by environmental noise and electromagnetic interference, resulting in inaccurate collected data and an inability to stably and reliably reflect the actual status of utility poles and towers.

[0006] A first aspect of the present invention provides a method for monitoring climbing behavior of a power pole tower, comprising:

[0007] Acquire an image sequence of the power pole tower area through a camera;

[0008] Preprocessing each frame of the regional image in the image sequence to generate a preprocessed image;

[0009] Sequentially calling the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract joint feature matrices from each of the preprocessed images;

[0010] Calling a convolutional neural network to classify the feature matrix of each joint point to determine whether there is a target feature that meets the climbing behavior;

[0011] Calling a long short-term memory network to predict each of the target features and determine climbing trend information;

[0012] If the target feature exists, or the climbing trend information reflects that the climbing behavior is continuing, a warning signal is output and a response strategy is fed back.

[0013] Optionally, preprocessing each frame of the regional image in the image sequence to generate a preprocessed image includes:

[0014] Denoising each frame of the regional image in the image sequence to obtain a denoised image;

[0015] Converting the denoised image into a binary space to obtain a binary image;

[0016] Performing morphological operations on the binary image to obtain an intermediate image;

[0017] Motion detection is performed on the intermediate images of two adjacent frames, and a changed area image is determined as a preprocessed image.

[0018] Optionally, the sequentially calling of the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract joint feature matrices from each of the preprocessed images includes:

[0019] Calling the spatiotemporal graph convolution model to extract the initial spatiotemporal feature matrix from each of the preprocessed images;

[0020] Calling the adaptive optimization model to perform feature optimization on each of the initial spatiotemporal feature matrices to generate a target spatiotemporal feature matrix;

[0021] The spatial configuration model is called to decode the joint point spatial coordinates from each of the target spatiotemporal feature matrices, and the joint point spatial coordinates are adaptively weighted updated to generate a joint point feature matrix.

[0022] Optionally, the calling of the spatiotemporal graph convolution model to extract an initial spatiotemporal feature matrix from each of the preprocessed images includes:

[0023] Each of the preprocessed images is converted into skeletal joint point information through the spatiotemporal graph convolution model, and a corresponding joint point matrix is ​​constructed;

[0024] Extracting initial spatiotemporal feature matrices from each joint point matrix using the spatiotemporal graph convolution model;

[0025] Among them, the spatiotemporal graph convolution model is:

[0026]

[0027] in, For the The initial spatiotemporal feature matrix of the layer, is the adjacency matrix with self-loops added, is the original adjacency matrix, is the identity matrix, is the degree matrix, It is The feature representation of the layer, , is the joint point matrix; It is The weight matrix of the layer, is a non-linear activation function.

[0028] Optionally, the adaptive optimization model includes an adaptive graph convolutional network and a spatiotemporal attention module; calling the adaptive optimization model to perform feature optimization on each of the initial spatiotemporal feature matrices to generate a target spatiotemporal feature matrix includes:

[0029] Calling the adaptive graph convolutional network to optimize each of the initial spatiotemporal feature matrices to obtain adaptive spatiotemporal feature matrices respectively;

[0030] The spatial attention score matrix of each of the skeletal joint information is calculated by the spatiotemporal attention module, and the adaptive spatiotemporal feature matrix is ​​weighted to obtain a spatial weighted feature matrix;

[0031] Calculating the time attention score matrix of each time step through the spatiotemporal attention module and weighting the spatial weighted feature matrix to obtain a time weighted feature matrix;

[0032] The temporal weighted feature matrix and the spatial weighted feature matrix are residually connected through the spatiotemporal attention module to generate a target spatiotemporal feature matrix.

[0033] Optionally, the calling of the spatial configuration model to decode the joint point spatial coordinates from each of the target spatiotemporal feature matrices and adaptively weighting the joint point spatial coordinates to generate a joint point feature matrix includes:

[0034] Calling a spatial configuration model to decode multiple joint point spatial coordinates from each of the target spatiotemporal feature matrices;

[0035] Calculating the dynamic importance weight corresponding to each pair of spatial coordinates of the joint points;

[0036] Weighting the spatial distance between each pair of the joint point spatial coordinates according to the dynamic importance weight to obtain a weighted spatial distance;

[0037] The weighted spatial distance is hierarchically traversed, and the target spatiotemporal feature matrix is ​​fused to generate a joint feature matrix.

[0038] Optionally, the convolutional neural network is composed of a plurality of convolutional layers, pooling layers, fully connected layers, and classification layers connected in sequence; calling the convolutional neural network to classify the feature matrix of each joint point to determine whether there is a target feature that meets the climbing behavior, including:

[0039] Calling each of the convolutional layers in sequence to perform convolution on each of the joint feature matrices to generate a local feature map of the human body posture;

[0040] Downsampling the human body posture local feature map through the pooling layer to obtain an intermediate feature map;

[0041] Converting the intermediate feature map into a one-dimensional vector;

[0042] Weighting the one-dimensional vector through the fully connected layer and superimposing a preset bias term to obtain a vector to be classified;

[0043] Calculating the probability distribution of the vector to be classified by the classification layer, and determining whether the vector to be classified conforms to the climbing behavior;

[0044] If yes, the vector to be classified is determined as the target feature;

[0045] If not, it is determined that there is no target feature that meets the climbing behavior.

[0046] A second aspect of the present invention provides a system for monitoring electric pole tower climbing behavior, comprising:

[0047] A data acquisition module is used to obtain an image sequence of the power pole tower area through a camera;

[0048] A preprocessing module, configured to preprocess each frame of the regional image in the image sequence to generate a preprocessed image;

[0049] A feature extraction module is used to sequentially call the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract joint feature matrices from each of the preprocessed images;

[0050] A behavior recognition module is used to call a convolutional neural network to classify the feature matrix of each joint point to determine whether there is a target feature that meets the climbing behavior;

[0051] A sequence prediction module is used to call a long short-term memory network to predict each of the target features and determine climbing trend information;

[0052] The feedback response module is used to output a warning signal and feedback a response strategy if the target feature exists or the climbing trend information reflects that the climbing behavior is continuing.

[0053] A third aspect of the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method for monitoring climbing behavior of a utility pole tower as described in any one of the first aspects of the present invention.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method for monitoring climbing behavior of a utility pole tower as described in any one of the first aspects of the present invention.

[0055] It can be seen from the above technical solutions that the present invention has the following advantages:

[0056] The present invention obtains an image sequence of a power pole tower area through a camera; preprocesses each frame of the regional image in the image sequence to generate a preprocessed image; sequentially calls a spatiotemporal graph convolution model, an adaptive optimization model, and a spatial configuration model to extract a joint feature matrix from each of the preprocessed images; calls a convolutional neural network to classify each of the joint feature matrices to determine whether there is a target feature that conforms to climbing behavior; calls a long short-term memory network to predict each of the target features to determine climbing trend information; if the target feature exists, or the climbing trend information reflects that the climbing behavior is continuing, outputs a warning signal and feeds back a response strategy, thereby stably and reliably reflecting the true state of the power pole tower and effectively realizing real-time monitoring of the transmission tower. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A flowchart of a method for monitoring climbing behavior of a power pole tower provided by an embodiment of the present invention;

[0059] Figure 2 This is a structural block diagram of a power pole tower climbing behavior monitoring system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] Embodiments of the present invention provide a method, system, device, and medium for monitoring climbing behavior on utility poles and towers, which are used to address the technical problem in existing solutions that video surveillance is affected by factors such as outdoor lighting, wind and rain, and obstructions, resulting in poor image acquisition quality. Sensor monitoring is also susceptible to environmental noise and electromagnetic interference, resulting in inaccurate collected data and an inability to stably and reliably reflect the actual status of utility poles and towers.

[0061] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] See also Figure 1 , Figure 1 A flowchart of a method for monitoring climbing behavior of a power pole tower provided in an embodiment of the present invention.

[0063] The present invention provides a method for monitoring climbing behavior of a power pole tower, comprising:

[0064] Step 101, obtaining an image sequence of a power pole tower area through a camera;

[0065] In this embodiment, a fixed or movable camera is used to capture a frame-by-frame image sequence within the electric pole tower area according to the set array and parameters. The image content includes at least the electric pole tower that needs to be monitored, and key parts such as the top, middle and bottom of the electric pole tower can be clearly captured, serving as the data basis for subsequent climbing behavior monitoring.

[0066] When selecting a camera, consider models with high resolution, wide dynamic range, and excellent night vision performance. They must also be dustproof, waterproof, and shockproof to withstand harsh outdoor environments. This ensures that the camera can capture high-quality images not only during the day but also at night or in adverse weather conditions, providing a solid data foundation for subsequent image processing.

[0067] Step 102, preprocessing each frame of the regional image in the image sequence to generate a preprocessed image;

[0068] After acquiring the image sequence, each frame region image in the image sequence is preprocessed separately through the OpenCV library. The preprocessing methods include but are not limited to denoising, grayscale, morphological operations, and frame difference method to generate preprocessed images.

[0069] In one example of the present invention, step 102 may include the following sub-steps:

[0070] Denoise each frame of the image sequence to obtain a denoised image;

[0071] Convert the denoised image into a binary space to obtain a binary image;

[0072] Perform morphological operations on the binary image to obtain an intermediate image;

[0073] Perform motion detection on the intermediate images of two adjacent frames and determine the image of the changed area as the preprocessing image.

[0074] In this embodiment, since the image is often interfered with by various noises during the acquisition process, such as equipment noise, environmental noise, etc., these noises will seriously affect the quality of the image, and thus affect the subsequent feature extraction and behavior recognition effects. The video image is smoothed to effectively filter out the noise in the video image. To this end, the image is denoised by using Gaussian filtering. Gaussian filtering is a linear smoothing filter that removes high-frequency components and retains low-frequency components by weighted averaging, thereby achieving the purpose of noise reduction. The specific denoising operation performed on each frame of the regional image can be achieved by the following methods:

[0075]

[0076] in, is the denoised image after Gaussian function processing, are the pixel coordinates within the region image, is the standard deviation of the Gaussian filter. The denoised image reduces the interference of irrelevant information, allowing subsequent processing to focus more on important visual features. This not only improves the grayscale effect but also lays a better foundation for morphological operations.

[0077] After obtaining the denoised image, the color image is converted to a grayscale image, retaining only the brightness information of the image. This reduces the amount of data processing, simplifies the subsequent processing flow, effectively improves the image quality, and provides clearer and more accurate image data for subsequent feature extraction and behavior recognition. The specific algorithm implementation process is as follows:

[0078]

[0079] Gray represents the grayscale value, while R, G, and B represent the pixel values ​​of the red, green, and blue channels, respectively. This formula reflects the varying sensitivity of the human eye to different colors, with the green channel having the highest weight, followed by red, and the blue least. After grayscaling, the image retains only luminance information, eliminating the complexity introduced by color variations. This is particularly advantageous for subsequent morphological operations, which rely on image binarization, a process that is more easily performed on grayscale images.

[0080] To further improve image quality, after generating a binary image, morphological operations such as dilation and erosion are required to remove small noise and fill holes in the image to obtain an intermediate image. The dilation operation expands the white area in the binary image; the erosion operation shrinks the white area. These two operations can be used to clean up irregular shapes on the image boundary. Specifically, for the dilation operation, the structural element is , then the expanded image It can be expressed by the following formula: . For the corrosion operation, we have: here, represents the original binary image, is a structural element, and denote the dilation and erosion operators respectively.

[0081] Frame difference method is a commonly used motion detection method that detects object movement by comparing the differences between adjacent frames to achieve foreground segmentation and quickly lock the position of the moving object. First, the current frame is subtracted from the previous frame to obtain a differential image. Then, threshold processing is applied to highlight the changed area. Specifically, for the middle image, the current frame is , the previous frame is , then the difference image for:

[0082]

[0083] Next, set a threshold , if the pixel value in the difference image is greater than , the position is considered to have changed:

[0084]

[0085] The frame difference method is used to perform motion detection on the intermediate images of two adjacent frames, and the image of the changed area is determined as the preprocessed image. , thus combining with the grayscale image, the human body posture can be located more accurately, providing key clues for subsequent behavior recognition.

[0086] Step 103: sequentially call the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract joint feature matrices from each preprocessed image.

[0087] In this embodiment, since the above steps have filtered out the changed area images, after obtaining multiple frames of pre-processed images, the spatiotemporal graph convolution model, the adaptive optimization model and the spatial configuration model can be called in sequence to process each frame of the pre-processed image separately to extract the joint point feature matrix therefrom.

[0088] In one example of the present invention, step 103 may include the following sub-steps S11-S13:

[0089] S11, calling the spatiotemporal graph convolution model to extract the initial spatiotemporal feature matrix from each preprocessed image;

[0090] Optionally, S11 may include the following sub-steps:

[0091] Each pre-processed image is converted into skeletal joint information through the spatiotemporal graph convolution model, and the corresponding joint point matrix is ​​constructed;

[0092] The initial spatiotemporal feature matrix is ​​extracted from each joint point matrix through the spatiotemporal graph convolution model;

[0093] Among them, the spatiotemporal graph convolution model is:

[0094] Among them, the spatiotemporal graph convolution model is:

[0095]

[0096] in, For the The initial spatiotemporal feature matrix of the layer, is the adjacency matrix with self-loops added, is the original adjacency matrix, which represents the connection relationship between joint points; is the unit matrix, used to introduce self-loop connections; is the degree matrix, which represents the degree of the nodes in the adjacency matrix; It is The feature representation of the layer, , is the joint point matrix; It is The weight matrix of the layer, is a non-linear activation function.

[0097] The spatial-temporal graph convolutional model refers to ST-GCN (Spatial Temporal Graph Convolutional Network), which is used to extend the graph convolutional network (GCN) to the spatiotemporal domain, while modeling and extracting features from data in both spatial and temporal dimensions, thereby effectively modeling the temporal dependency and spatial structure of actions.

[0098] In an embodiment of the present invention, the preprocessed image is converted into skeletal joint point information to construct a spatiotemporal graph. This skeletal information includes the positions of various key points of the human body, forming a skeleton graph. The nodes of this graph represent the different joints of the human body, and the edges represent the spatial relationships between the joints. This skeleton information is then used as the joint point matrix "X" to construct a spatiotemporal graph. The nodes and edges in the graph change as the time series changes. Then, a multi-layer graph convolution layer is used to extract the initial spatiotemporal feature matrix, which contains information about the changes in human posture over time and space.

[0099] ST-GCN extracts spatiotemporal features through graph convolution operations. Each layer of graph convolution transfers information between nodes through the adjacency matrix. This information transfer mechanism enables the network to effectively capture motion patterns and process the combination of temporal data and spatial structure to extract the initial spatiotemporal feature matrix. Specifically, the joint point matrix is the initial input of the graph convolution operation (i.e. , gradually updated to the initial spatiotemporal feature matrix through multi-layer graph convolution .

[0100] S12, calling the adaptive optimization model to optimize the features of each initial spatiotemporal feature matrix to generate a target spatiotemporal feature matrix;

[0101] In this embodiment, the spatiotemporal graph convolutional model is primarily used to extract features from spatiotemporal sequence data for skeleton-based dynamic posture analysis. It combines temporal data with spatial structural information, helping the model understand dynamic motion patterns derived from time-series images captured by drones. However, the spatiotemporal graph convolutional model has a fixed topological structure, limiting its flexibility. Further feature optimization can be achieved by adapting the newly optimized model to an adaptive graph convolutional network. This model introduces additional learning parameters to adjust the connection strengths between different nodes, processing irregular data in complex environments (such as obstacles of varying heights and angles) with adaptive weights. Furthermore, a spatiotemporal attention module is used to highlight important spatiotemporal regions and suppress irrelevant information, thereby increasing the model's sensitivity to climbing behavior.

[0102] Furthermore, the adaptive optimization model includes an adaptive graph convolutional network and a spatiotemporal attention module; S12 may include the following sub-steps:

[0103] Call the adaptive graph convolutional network to optimize each initial spatiotemporal feature matrix to obtain the adaptive spatiotemporal feature matrix respectively;

[0104] The spatial attention score matrix of each skeletal joint information is calculated through the spatiotemporal attention module, and the adaptive spatiotemporal feature matrix is ​​weighted to obtain the spatial weighted feature matrix;

[0105] The spatiotemporal attention module calculates the time attention score matrix of each time step and weights the spatial weighted feature matrix to obtain the time weighted feature matrix;

[0106] The temporal weighted feature matrix and the spatial weighted feature matrix are residually connected through the spatiotemporal attention module to generate the target spatiotemporal feature matrix.

[0107] Adaptive Graph Convolutional Network (AGCN) introduces an adaptive mechanism that can dynamically adjust the graph's adjacency matrix, that is, the connection relationships and weights between nodes, according to the input data. This allows the model to better capture complex patterns and dependencies in the data, allowing it to adapt more flexibly and effectively to different data features and task requirements.

[0108] In this embodiment, an adaptive adjacency matrix is ​​introduced. This enables the graph convolutional network to dynamically adjust edge weights based on the relationships in the actual data. In this way, AGCN can handle more complex scenarios and irregular data, thereby enhancing the model's ability to adapt to environmental changes. Specifically, the optimization update method for the adaptive spatiotemporal feature matrix is ​​as follows:

[0109]

[0110] in, is the updated adaptive spatiotemporal feature matrix of the lth layer, is the new degree matrix calculated based on the adaptive adjacency matrix, is the adaptive adjacency matrix.

[0111] By generating an adaptive spatiotemporal feature matrix in the above manner, the matrix can better reflect the real changes in human posture and improve the accuracy of behavior recognition.

[0112] After obtaining the adaptive spatiotemporal feature matrix, the model is able to better focus on important action features by placing greater emphasis on key locations. This allows the spatiotemporal graph convolutional network to not only adapt to environmental changes and adjust the graph structure when extracting spatiotemporal features, but also further enhances the model's adaptability through the attention mechanism. The spatiotemporal attention mechanism (AST) helps AGCN prioritize the most critical information in complex environments by weighting the importance of different temporal and spatial regions.

[0113] Specifically, the adaptive spatiotemporal feature matrix contains rich spatiotemporal information. By weighting the importance of different temporal and spatial regions in these adaptive spatiotemporal feature matrices, we can focus on key information and improve the recognition accuracy of climbing behavior. Among them, the spatiotemporal attention module includes a spatial attention mechanism and a temporal attention mechanism. The spatial attention mechanism is used to calculate the importance of different joints and highlight the impact of key joints on behavior recognition. The spatial attention score is calculated as follows:

[0114]

[0115] in,

[0116] is a node The unnormalized score of , usually calculated by a lightweight neural network;

[0117] is a node The spatial attention score of a feature reflects its importance in the current frame.

[0118] Using spatial attention score For the feature matrix Perform weighting to generate a spatial weighted feature matrix :

[0119]

[0120] in, is the spatial attention score matrix; Represents element-wise multiplication.

[0121] The temporal attention mechanism is used to calculate the correlation between different moments and highlight frames with important changes in time. , calculate its unnormalized score And normalized to the time attention score through the Softmax function :

[0122]

[0123] in, is the time step The unnormalized score of is calculated by a lightweight neural network; is the time step The temporal attention score of a feature reflects its importance in the time series.

[0124] Using temporal attention scores Spatially weighted feature matrix Perform weighting to generate a time-weighted feature matrix :

[0125]

[0126] in, is the temporal attention score matrix; Represents element-wise multiplication.

[0127] The time-weighted feature matrix after spatial and temporal attention is weighted by the spatiotemporal attention module and adaptive spatiotemporal feature matrix After fusion, the AST module finally outputs the target spatiotemporal feature matrix :

[0128]

[0129] Among them, MLP is a multi-layer perceptron, which is used to fuse attention-weighted features; Represents residual connection, which retains the original feature information.

[0130] The above AST module enables the model to more keenly capture the key moments and key parts of human posture changes, thereby improving the recognition accuracy of climbing behavior. The AST module generates a more accurate target spatiotemporal feature matrix. These attention-weighted features will provide more sensitive input for subsequent behavior recognition and improve the model's ability to detect climbing behavior.

[0131] S13. Call the spatial configuration model to decode the joint point spatial coordinates from each target spatiotemporal feature matrix, and perform adaptive weighted update on the joint point spatial coordinates to generate a joint point feature matrix.

[0132] Furthermore, S13 may include the following sub-steps:

[0133] Call the spatial configuration model to decode the spatial coordinates of multiple joint points from each target spatiotemporal feature matrix;

[0134] Calculate the dynamic importance weight corresponding to each pair of joint space coordinates;

[0135] The spatial distance between each pair of joint point spatial coordinates is weighted according to the dynamic importance weight to obtain the weighted spatial distance;

[0136] The weighted spatial distance is hierarchically traversed and the target spatiotemporal feature matrix is ​​fused to generate the joint feature matrix.

[0137] In a tower climbing monitoring system, climbing behavior recognition relies on accurate analysis of human posture. However, complex scenarios (such as perspective changes, occlusions, and lighting interference) often lead to inaccurate joint positioning, which in turn affects behavior recognition. Image convolutional networks rely on a fixed adjacency matrix and struggle to adapt to the dynamic relationships between joints in complex movements. To address this, this strategy optimizes the spatial configuration of joints by introducing dynamic weight allocation, semantic distance metrics, and a hierarchical traversal mechanism, enhancing the model's ability to analyze climbing behavior.

[0138] In this embodiment, after obtaining the target spatiotemporal feature matrix After that, its dimension is ,in is the number of joints, is the feature dimension. The spatial coordinates of each joint point can be decoded from the target spatiotemporal feature matrix First, a lightweight network is used to perform feature splicing and nonlinear transformation to generate dynamic importance weights that reflect the semantic relationship between joint points. :

[0139]

[0140] in, Represents feature splicing; and is a learnable parameter; and Joint points and The eigenvector of .

[0141] In getting dynamic importance weight Finally, the traditional spatial distance formula only considers the Euclidean distance (i.e., spatial distance) between joints. However, different joints have different importance in human posture and movement. To enhance the influence of key joints, a weight factor can be introduced for each pair of joints, making the connection between joints more flexible and accurate. The spatial distance formula between joints is set as:

[0142]

[0143] in, and For joint points and The spatial coordinates of For joint points and This weight factor can be set based on the importance of the joint. For example, the weight of the limb joints may be higher than that of the torso joints, or some joints may have a greater influence in a specific action.

[0144] The weighted spatial distance is traversed hierarchically, which is divided into local traversal and global traversal. The local traversal prioritizes the connections between the limb joints to capture local motion details; the global traversal aggregates local features based on dynamic weights to generate a global motion pattern description. During the traversal process, dynamic weights are used to and weighted distance Optimize the spatial relationship between joints.

[0145] After the traversal is completed, the target spatiotemporal feature matrix is ​​fused according to the traversal results to generate the joint feature matrix The matrix combines spatially optimized joint point relationships with dynamic feature information. These optimized features more accurately reflect the spatial changes in the human body in different postures and movements, allowing the model to more carefully analyze changes in human posture, especially in complex movements such as climbing, and more accurately identify abnormal behaviors. Through this improved spatial configuration, the model can more carefully analyze the dynamic changes of the human body, especially in complex behavior patterns (such as climbing), thereby improving the recognition accuracy of these behaviors. The specific fusion method is as follows:

[0146]

[0147] in, is the dynamic weight matrix; is a weighted distance matrix; MLP is a multi-layer perceptron used to fuse spatial optimization information.

[0148] Step 104: Calling a convolutional neural network to classify the feature matrix of each joint point to determine whether there is a target feature that meets the climbing behavior;

[0149] In this embodiment, a convolutional neural network is used to perform an in-depth analysis of the joint feature matrix to determine whether climbing behavior occurs.

[0150] Among them, the convolutional layer of the convolutional neural network is as follows:

[0151]

[0152] in, For the The convolutional layer Output value of position; is the weight of the convolution kernel; For the Layer The input value of the position; is the bias term; is an activation function, such as the ReLU function.

[0153] In one example of the present invention, the convolutional neural network is composed of a plurality of convolutional layers, pooling layers, fully connected layers, and classification layers connected in sequence; step 104 may include the following sub-steps:

[0154] Call each convolution layer in turn to convolve the feature matrix of each joint point to generate local feature maps of the human body posture;

[0155] The local feature map of the human body posture is downsampled through the pooling layer to obtain the intermediate feature map;

[0156] Convert the intermediate feature map into a one-dimensional vector;

[0157] The one-dimensional vector is weighted by the fully connected layer and the preset bias term is superimposed to obtain the vector to be classified;

[0158] The probability distribution of the vector to be classified is calculated through the classification layer to determine whether the vector to be classified conforms to the climbing behavior;

[0159] If so, the vector to be classified is determined as the target feature;

[0160] If not, it is determined that there is no target feature that meets the climbing behavior.

[0161] In an embodiment of the present invention, a convolutional neural network is composed of multiple convolutional layers, pooling layers, fully connected layers and classification layers connected in sequence. The convolutional layer extracts local features of human posture through convolution operations, the pooling layer reduces the dimension and computational complexity of the feature map through downsampling, and the fully connected layer maps the features extracted by the convolutional layer to the classification space to determine whether there is climbing behavior.

[0162] Specifically, the joint feature matrix is ​​convolved through the above multi-layer convolution layer to generate a local feature map of the human body posture. The local feature map of the human body posture is downsampled through a pooling layer (such as a maximum pooling layer or an average pooling layer) to reduce the dimension and computational complexity of the feature map, and an intermediate feature map is obtained, as shown below:

[0163]

[0164] in, For the The pooling layer is at position The output value of is the pooling step size; Indicates taking the maximum value within the pooling window.

[0165] After obtaining the intermediate feature map, the intermediate feature map is flattened into a one-dimensional vector. The one-dimensional vector is weighted by the fully connected layer and the preset bias term is superimposed to obtain the vector to be classified. Specifically:

[0166]

[0167] in, is the vector to be classified output by the fully connected layer, is the weight matrix; is the flattened one-dimensional vector; is the bias term.

[0168] After obtaining the vector to be classified, it is mapped to the classification space. The softmax function is called through the classification layer to calculate the probability distribution of each category and determine which category the input belongs to (climbing behavior or non-climbing behavior), as shown below:

[0169]

[0170] in, For input Belong to category probability; For the The output value of each neuron (i.e., the result of weighted summation), C is the total number of categories (which can be 2, i.e., climbing behavior and non-climbing behavior).

[0171] Step 105: Using a long short-term memory network to predict each target feature and determine climbing trend information;

[0172] In this example, since climbing behavior is continuous, a long short-term memory (LSTM) network can be used to perform time series analysis to predict climbing trends in order to promptly determine climbing trends. LSTM is a special type of recurrent neural network (RNN) that can capture long-term dependencies in time series. By constructing an LSTM model, the duration and trend of climbing behavior can be predicted as follows:

[0173]

[0174] in, 、 、 , are the output values ​​of the forget gate, input gate and output gate respectively, is the target feature at time t; 、 are cell states and candidate cell states, respectively; Output value for the hidden layer; 、 、 、 are the weight matrices of the forget gate, input gate, cell state, and output gate respectively; 、 、 、 are the corresponding bias terms respectively; is the sigmoid activation function; tanh is the hyperbolic tangent activation function.

[0175] Step 106: If the target feature exists, or the climbing trend information reflects that the climbing behavior is continuing, a warning signal is output and a response strategy is fed back.

[0176] When target features consistent with climbing behavior are detected, or climbing trend information reflects that climbing behavior will continue, it indicates that there is a climbing risk in the current scenario. At this time, an early warning signal can be output to the monitoring end to inform technical personnel to respond.

[0177] Furthermore, different response strategies can be adapted for different situations, such as when climbing behavior is detected and predicted to continue. For example, if climbing behavior is detected but there is a possibility of continued climbing, a pre-recorded voice warning can be played when the warning signal is output, informing the climber of the danger of their behavior and requiring them to stop climbing and descend as soon as possible. If the climbing behavior is a false alarm caused by routine maintenance, the warning signal can be directly terminated.

[0178] In addition, the specific implementation of each of the above steps can be integrated through virtual modules, and then connected through integration and optimization modules to form a complete monitoring system. The processing speed and robustness can also be improved by optimizing the algorithm.

[0179] In this embodiment of the present invention, a camera captures an image sequence of a power pole area. Each frame of the image sequence is preprocessed to generate a preprocessed image. A spatiotemporal graph convolution model, an adaptive optimization model, and a spatial configuration model are sequentially invoked to extract joint feature matrices from each preprocessed image. A convolutional neural network is then used to classify each joint feature matrix to determine whether a target feature consistent with climbing behavior exists. A long-short-term memory network is then used to predict each target feature and determine climbing trend information. If the target feature exists, or if the climbing trend information indicates that climbing behavior is ongoing, a warning signal is output and a response strategy is provided. By integrating OpenCV's image preprocessing algorithms with advanced graph convolution techniques, as well as deep learning feature extraction and behavior recognition algorithms, the accuracy of climbing behavior recognition is significantly improved. This eliminates the need to install a large number of sensors on the poles and towers, utilizing existing camera equipment to achieve monitoring, reducing hardware costs. The neural network model is highly robust and can cope with various lighting conditions, occlusion, noise, and other interference factors. A warning and response mechanism has also been established. Once climbing behavior is detected or predicted to continue, a warning signal can be immediately triggered and a corresponding response strategy can be formulated. This can stably and reliably reflect the actual status of power poles and towers, effectively achieving real-time monitoring of transmission towers.

[0180] See also Figure 2 , Figure 2 4 is a structural block diagram of a power pole tower climbing behavior monitoring system in an embodiment of the present invention.

[0181] An embodiment of the present invention provides a system for monitoring electric pole tower climbing behavior, comprising:

[0182] The data acquisition module 201 is used to obtain an image sequence of the power pole tower area through a camera;

[0183] A preprocessing module 202 is used to preprocess each frame of the regional image in the image sequence to generate a preprocessed image;

[0184] The feature extraction module 203 is used to sequentially call the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract the joint feature matrix from each preprocessed image;

[0185] The behavior recognition module 204 is used to call the convolutional neural network to classify the feature matrix of each joint point and determine whether there is a target feature that meets the climbing behavior;

[0186] The sequence prediction module 205 is used to call the long short-term memory network to predict each target feature and determine the climbing trend information;

[0187] The feedback response module 206 is configured to output a warning signal and provide feedback on a response strategy if the target feature exists or the climbing trend information indicates that the climbing behavior is continuing.

[0188] Optionally, the pre-processing module 202 is specifically configured to:

[0189] Denoise each frame of the image sequence to obtain a denoised image;

[0190] Convert the denoised image into a binary space to obtain a binary image;

[0191] Perform morphological operations on the binary image to obtain an intermediate image;

[0192] Perform motion detection on the intermediate images of two adjacent frames and determine the image of the changed area as the preprocessing image.

[0193] Optionally, the feature extraction module 203 includes:

[0194] The spatiotemporal feature extraction submodule is used to call the spatiotemporal graph convolution model to extract the initial spatiotemporal feature matrix from each preprocessed image;

[0195] The spatiotemporal feature optimization submodule is used to call the adaptive optimization model to optimize the features of each initial spatiotemporal feature matrix and generate the target spatiotemporal feature matrix;

[0196] The feature update submodule is used to call the spatial configuration model to decode the joint point spatial coordinates from each target spatiotemporal feature matrix, and perform adaptive weighted update on the joint point spatial coordinates to generate the joint point feature matrix.

[0197] Optionally, the spatiotemporal feature extraction submodule is specifically used to:

[0198] Each pre-processed image is converted into skeletal joint information through the spatiotemporal graph convolution model, and the corresponding joint point matrix is ​​constructed;

[0199] The initial spatiotemporal feature matrix is ​​extracted from each joint point matrix through the spatiotemporal graph convolution model;

[0200] Among them, the spatiotemporal graph convolution model is:

[0201]

[0202] in, For the The initial spatiotemporal feature matrix of the layer, for, is the degree matrix, which represents the degree of the nodes in the adjacency matrix, It is The feature representation of the layer, , is the joint point matrix; It is The weight matrix of the layer, is a non-linear activation function.

[0203] Optionally, the adaptive optimization model includes an adaptive graph convolutional network and a spatiotemporal attention module; the spatiotemporal feature optimization submodule is specifically used to:

[0204] Call the adaptive graph convolutional network to optimize each initial spatiotemporal feature matrix to obtain the adaptive spatiotemporal feature matrix respectively;

[0205] The spatial attention score matrix of each skeletal joint information is calculated through the spatiotemporal attention module, and the adaptive spatiotemporal feature matrix is ​​weighted to obtain the spatial weighted feature matrix;

[0206] The spatiotemporal attention module calculates the time attention score matrix of each time step and weights the spatial weighted feature matrix to obtain the time weighted feature matrix;

[0207] The temporal weighted feature matrix and the spatial weighted feature matrix are residually connected through the spatiotemporal attention module to generate the target spatiotemporal feature matrix.

[0208] Optionally, the feature update submodule is specifically used to:

[0209] Call the spatial configuration model to decode the spatial coordinates of multiple joint points from each target spatiotemporal feature matrix;

[0210] Calculate the dynamic importance weight corresponding to each pair of joint space coordinates;

[0211] The spatial distance between each pair of joint point spatial coordinates is weighted according to the dynamic importance weight to obtain the weighted spatial distance;

[0212] The weighted spatial distance is hierarchically traversed and the target spatiotemporal feature matrix is ​​fused to generate the joint feature matrix.

[0213] Optionally, the convolutional neural network is composed of a plurality of convolutional layers, pooling layers, fully connected layers, and classification layers connected in sequence; the behavior recognition module 204 is specifically used to:

[0214] Call each convolution layer in turn to convolve the feature matrix of each joint point to generate local feature maps of the human body posture;

[0215] The local feature map of the human body posture is downsampled through the pooling layer to obtain the intermediate feature map;

[0216] Convert the intermediate feature map into a one-dimensional vector;

[0217] The one-dimensional vector is weighted by the fully connected layer and the preset bias term is superimposed to obtain the vector to be classified;

[0218] The probability distribution of the vector to be classified is calculated through the classification layer to determine whether the vector to be classified conforms to the climbing behavior;

[0219] If so, the vector to be classified is determined as the target feature;

[0220] If not, it is determined that there is no target feature that meets the climbing behavior.

[0221] An embodiment of the present invention provides an electronic device including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the method for monitoring electric pole climbing behavior in any embodiment of the present invention.

[0222] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, the method for monitoring electric pole tower climbing behavior according to any embodiment of the present invention is implemented.

[0223] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0224] In the several embodiments provided herein, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the module division is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the system or module, and can be electrical, mechanical or other forms.

[0225] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0226] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0227] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0228] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring electric pole tower climbing behavior, characterized in that: include: Acquire an image sequence of the power pole tower area through a camera; Preprocessing each frame of the regional image in the image sequence to generate a preprocessed image; Sequentially calling the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract joint feature matrices from each of the preprocessed images; Calling a convolutional neural network to classify the feature matrix of each joint point to determine whether there is a target feature that meets the climbing behavior; Calling a long short-term memory network to predict each of the target features and determine climbing trend information; If the target feature exists, or the climbing trend information reflects that the climbing behavior is continuing, a warning signal is output and a response strategy is fed back.

2. The method according to claim 1, characterized in that The preprocessing of each frame of the regional image in the image sequence to generate a preprocessed image includes: Denoising each frame of the regional image in the image sequence to obtain a denoised image; Converting the denoised image into a binary space to obtain a binary image; Performing morphological operations on the binary image to obtain an intermediate image; Motion detection is performed on the intermediate images of two adjacent frames, and a changed area image is determined as a preprocessed image.

3. The method according to claim 1, characterized in that The sequentially calling of the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract joint feature matrices from each of the preprocessed images includes: Calling the spatiotemporal graph convolution model to extract the initial spatiotemporal feature matrix from each of the preprocessed images; Calling the adaptive optimization model to perform feature optimization on each of the initial spatiotemporal feature matrices to generate a target spatiotemporal feature matrix; The spatial configuration model is called to decode the joint point spatial coordinates from each of the target spatiotemporal feature matrices, and the joint point spatial coordinates are adaptively weighted updated to generate a joint point feature matrix.

4. The method according to claim 3, characterized in that The calling of the spatiotemporal graph convolution model to extract an initial spatiotemporal feature matrix from each of the preprocessed images includes: Each of the preprocessed images is converted into skeletal joint point information through the spatiotemporal graph convolution model, and a corresponding joint point matrix is ​​constructed; Extracting initial spatiotemporal feature matrices from each joint point matrix using the spatiotemporal graph convolution model; Among them, the spatiotemporal graph convolution model is: in, For the The initial spatiotemporal feature matrix of the layer, is the adjacency matrix with self-loops added, is the original adjacency matrix, is the identity matrix, is the degree matrix, It is The feature representation of the layer, , is the joint point matrix; It is The weight matrix of the layer, is a non-linear activation function.

5. The method according to claim 3, characterized in that The adaptive optimization model includes an adaptive graph convolutional network and a spatiotemporal attention module; The calling of the adaptive optimization model to perform feature optimization on each of the initial spatiotemporal feature matrices to generate a target spatiotemporal feature matrix includes: Calling the adaptive graph convolutional network to optimize each of the initial spatiotemporal feature matrices to obtain adaptive spatiotemporal feature matrices respectively; The spatial attention score matrix of each of the skeletal joint information is calculated by the spatiotemporal attention module, and the adaptive spatiotemporal feature matrix is ​​weighted to obtain a spatial weighted feature matrix; Calculating the time attention score matrix of each time step through the spatiotemporal attention module and weighting the spatial weighted feature matrix to obtain a time weighted feature matrix; The temporal weighted feature matrix and the spatial weighted feature matrix are residually connected through the spatiotemporal attention module to generate a target spatiotemporal feature matrix.

6. The method according to claim 3, characterized in that The calling spatial configuration model decodes the joint point spatial coordinates from each of the target spatiotemporal feature matrices and performs adaptive weighted update on the joint point spatial coordinates to generate a joint point feature matrix, including: Calling a spatial configuration model to decode multiple joint point spatial coordinates from each of the target spatiotemporal feature matrices; Calculating the dynamic importance weight corresponding to each pair of the joint point spatial coordinates; Weighting the spatial distance between each pair of the joint point spatial coordinates according to the dynamic importance weight to obtain a weighted spatial distance; The weighted spatial distance is hierarchically traversed, and the target spatiotemporal feature matrix is ​​fused to generate a joint feature matrix.

7. The method according to claim 1, characterized in that The convolutional neural network is composed of a plurality of convolutional layers, pooling layers, fully connected layers, and classification layers connected in sequence. The calling of the convolutional neural network to classify the feature matrix of each joint point and determine whether there is a target feature that meets the climbing behavior includes: Calling each of the convolutional layers in sequence to perform convolution on each of the joint feature matrices to generate a local feature map of the human body posture; Downsampling the human body posture local feature map through the pooling layer to obtain an intermediate feature map; Converting the intermediate feature map into a one-dimensional vector; Weighting the one-dimensional vector through the fully connected layer and superimposing a preset bias term to obtain a vector to be classified; Calculating the probability distribution of the vector to be classified by the classification layer, and determining whether the vector to be classified conforms to the climbing behavior; If yes, the vector to be classified is determined as the target feature; If not, it is determined that there is no target feature that meets the climbing behavior.

8. A system for monitoring electric pole tower climbing behavior, characterized in that: include: A data acquisition module is used to obtain an image sequence of the power pole tower area through a camera; A preprocessing module, configured to preprocess each frame of the regional image in the image sequence to generate a preprocessed image; A feature extraction module is used to sequentially call the spatiotemporal graph convolution model, the adaptive optimization model, and the spatial configuration model to extract joint feature matrices from each of the preprocessed images; A behavior recognition module is used to call a convolutional neural network to classify the feature matrix of each joint point to determine whether there is a target feature that meets the climbing behavior; A sequence prediction module is used to call a long short-term memory network to predict each of the target features and determine climbing trend information; The feedback response module is used to output a warning signal and feedback a response strategy if the target feature exists or the climbing trend information reflects that the climbing behavior is continuing.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for monitoring climbing behavior of a power pole tower as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method for monitoring electric pole tower climbing behavior according to any one of claims 1 to 7 is implemented.

Images