Electric power high-altitude worker working state detection method based on image feature fusion

Abstract

The invention relates to the technical field of internet big data, in particular to an electric power high-altitude worker working state detection method based on image feature fusion, which comprises the following steps: S1, acquiring a to-be-detected original image; s2, inputting the original image into an OpenPose network model to perform human body posture feature extraction to obtain human body posture key point features of each person target in the original image; s3, inputting the original image into a YOLACT model to perform power equipment feature extraction to obtain power equipment features; s4, performing feature fusion on the human body posture key point features and the power equipment features of each personnel target to obtain fusion features of each personnel target; and S5, inputting the fusion features of each personnel target into a trained online sequential extreme learning machine for classification, judging whether each personnel target is performing high-altitude electric power operation or not, and outputting the personnel target performing high-altitude electric power operation in the original image. The detection precision of the working state of the electric power high-altitude worker can be improved.

Description

Technical Field

[0001] The present invention relates to the field of Internet big data technology, and in particular to a method for detecting the operating status of power high-altitude workers based on image feature fusion. Background Art

[0002] In the power industry, aerial work is common in critical areas such as transmission line maintenance and substation overhaul. These complex and high-risk working environments place extremely high demands on worker safety. Depending on the task and job responsibilities, workers must adhere to differentiated safety operating procedures. For example, frontline workers performing aerial work tasks must strictly wear protective equipment such as safety belts and helmets. While management personnel such as on-site supervisors and safety officers may be present in the work area, they do not need to directly participate in the work, and therefore, the safety regulations required are relatively low. Therefore, in the intelligent monitoring of aerial work in the power industry, accurately identifying the actual working status of workers is a prerequisite for achieving differentiated safety monitoring management.

[0003] Currently, domestic power construction safety supervision primarily relies on manual inspections and verifications, relying on on-site personnel to monitor safety equipment wear and behavioral compliance. This approach is not only inefficient and difficult to implement for long periods of time, but is also susceptible to subjective judgment, limited human resources, and other factors, making it difficult to ensure the accuracy and timeliness of supervision results.

[0004] In recent years, with the development of computer vision and deep learning technologies, intelligent monitoring methods based on image analysis have been gradually introduced into the power industry, primarily for applications such as violation detection and on-site safety identification. However, existing research is generally limited to detecting specific targets (such as helmets and seat belts), lacks a mechanism to distinguish between "workers" and "non-workers" in images, and fails to fully consider the relationship between workers' behavioral characteristics in images and the semantics of the environment.

[0005] The applicant discovered that workers actually working at heights have distinct differences in their posture, location, and interactions with equipment, making them high-priority targets for safety monitoring. Therefore, designing a method that can effectively detect the working status of workers at heights in power plants is a pressing technical challenge. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the existing technology, the technical problem to be solved by the present invention is: how to provide a method for detecting the working status of high-altitude power workers based on image feature fusion. By fusing the key point features of human posture and the features of power equipment, a joint analysis of "human behavior-equipment operation" is realized, which significantly improves the accuracy and effectiveness of the recognition of high-risk actions unique to the power industry, thereby improving the accuracy of the detection of the working status of high-altitude power workers.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The method for detecting the working status of power workers at high altitudes based on image feature fusion includes:

[0009] S1: Get the original image to be detected;

[0010] S2: Input the original image into the OpenPose network model to extract human posture features and obtain the key point features of the human posture of each person target in the original image;

[0011] S3: Input the original image into the YOLACT model to extract the power equipment features and obtain the power equipment features;

[0012] S4: performing feature fusion on the key point features of the human body posture of each person target and the features of the power equipment to obtain the fusion features of each person target;

[0013] S5: Input the fusion features of each personnel target into the trained online sequential extreme learning machine for classification, determine whether each personnel target is performing high-altitude power work, and output the personnel targets performing high-altitude power work in the original image.

[0014] Preferably, in step S2, the VGG-19 network of the backbone network in the OpenPose network model is replaced with a MobileNet V2 module with an attention mechanism;

[0015] The MobileNet V2 module with attention mechanism includes a 1×1 convolution module, a 3×3 depth convolution module, a spatial and channel attention mechanism module, and a 1×1 convolution module connected end to end. The spatial and channel attention mechanism module is composed of a channel attention unit and a spatial attention unit in series, which is used to enhance feature expression capabilities from both the channel dimension and the spatial dimension.

[0016] Preferably, in step S2, the key point features of the human body posture of the target person extracted by the OpenPose network model include the horizontal and vertical coordinates (x i ,y i ) and confidence ci .

[0017] Preferably, in step S3, when training the YOLACT model, the generalized intersection-over-union loss L is introduced into the loss function of the YOLACT model. GIoU ;

[0018] The total loss function of the YOLACT model is expressed as:

[0019] L total =L bbox +λ GIoU ·L GIoU ;

[0020] L GIoU =1-GIoU(A,B);

[0021]

[0022] Where: L total Indicates adding generalized intersection loss L GIoU The total loss of the post-YOLACT model; L bbox represents the bounding box regression loss; λ GIoU Represents a hyperparameter used to control the generalized intersection-over-union loss L GIoU Impact on the total loss; GIoU(A,B) represents the generalized intersection-over-union ratio between the predicted box A and the true box B; U represents the union area of the predicted box and the true box, that is, their total area minus the intersection part; A c Represents the minimum enclosed area of the predicted box A and the true box B.

[0023] Preferably, in step S3, the power equipment features include several power equipment categories, and detection box position information, detection box position confidence score and power equipment semantic segmentation mask corresponding to each power equipment category.

[0024] Preferably, in step S3, the NMS module in the YOLACT model is replaced with an adaptive non-maximum suppression module to reduce redundant candidate boxes generated during the detection process, and retain the most representative detection box when there are multiple overlapping high-confidence boxes.

[0025] Preferably, in step S4, feature fusion is performed through the following steps:

[0026] S401: Obtain the horizontal and vertical coordinates (x and y) of each person target, including 25 key points i ,y i ) and confidence c iThe human body posture key point features, as well as the power equipment features including several power equipment categories and the detection box position information corresponding to each power equipment category, the detection box position confidence score and the power equipment semantic segmentation mask;

[0027] S402: extracting features from the semantic segmentation mask in each power equipment category using a convolutional neural network to obtain corresponding mask extraction features;

[0028] S403: Perform tensor splicing on the key point features of the human body posture of each personnel target and the position information, confidence score and mask extraction features corresponding to each power equipment category to obtain the fusion features of each personnel target.

[0029] Preferably, in step S402, the convolutional neural network includes a first convolutional layer with a dimension of 8×700×700, a first pooling layer with a dimension of 8×350×350, a second convolutional layer with a dimension of 16×350×350, a second pooling layer with a dimension of 16×175×175, a third convolutional layer with a dimension of 32×175×175, a third pooling layer with a dimension of 32×87×87, a flattening operation layer, and a first fully connected layer and a second fully connected layer, which are connected end to end in sequence; the second fully connected layer outputs a mask to extract features.

[0030] Preferably, in step S5, the processing steps for training the online sequential extreme learning machine are as follows:

[0031] S501: Acquire several groups of samples to construct a training set and a test set, where each group of samples includes key point features of a person's body posture and features of power equipment in the same image;

[0032] S502: For samples in the training set and the test set: connect and sort the key points in the key point features of the human body posture of the target according to the human body structure, and label the working status label of the target, where the label 0 indicates that no high-altitude power work is being performed, and 1 indicates that high-altitude power work is being performed; perform feature fusion on the key point features of the human body posture of the target and the features of the power equipment to obtain training data, and associate the input data with the working status label of the target;

[0033] S503: Using the training data in the training set as input to the online sequential extreme learning machine to train and update the weight β; wherein the supervision label of the online sequential extreme learning machine is two-dimensionally one-hot encoded, where the bit with index 0 represents no high-altitude power work, and the bit with index 1 represents current high-altitude power work;

[0034] The formula for updating the weight β is expressed as:

[0035]

[0036] Where: H is the feature matrix of training data, T is the label matrix;

[0037] S504: Repeat step S503 to iteratively train and update the weight β until the online sequential extreme learning machine converges;

[0038] S505: Input the training data and trained weights β in the test set into the online sequential extreme learning machine and output the corresponding prediction result t pre =[t pre0 ,t pre1 ]:When t pre0 >t pre1 When t pre0 <t pre1 When the predicted label is 1, it means that electrical high-altitude work is in progress.

[0039] Preferably, in step S5, a learnable weight matrix is set for the 25 key points of the key point features of the human body posture in the input layer of the online sequential extreme learning machine, and each element of the weight matrix corresponds to a specific weight of a key point; the online sequential extreme learning machine dynamically updates the weight matrix according to the classification loss function during the training process to achieve feature weighting of different key points;

[0040] The processing steps for dynamically updating the weight matrix include:

[0041] S201: Initialize the weight matrix to the identity matrix, that is, each key point has the same weight;

[0042] S202: performing weighted calculation on the extracted human posture key point features X and the weight matrix W to obtain weighted human posture key point features;

[0043] The formula is:

[0044] X′=X·W;

[0045] S203: Fusing the weighted human posture key point features X′ and the power equipment features to obtain corresponding fused features; then inputting the fused features into an online sequential extreme learning machine for classification, and calculating the classification loss function of the online sequential extreme learning machine;

[0046] S204: Using the defined back propagation algorithm, the update formula of the weight matrix W is:

[0047]

[0048] Where: α represents the learning rate; L represents the classification loss of the online sequential extreme learning machine;

[0049] S205: Repeat steps S202 to S204 to iteratively train the online sequential extreme learning machine until convergence, and obtain an updated weight matrix.

[0050] Compared with the existing technology, the method for detecting the working status of power workers at high altitudes based on image feature fusion in the present invention has the following beneficial effects:

[0051] The present invention uses the human posture key point features extracted by the OpenPose network (the horizontal and vertical coordinates and confidence levels of 25 key points) to accurately characterize the dynamic motion characteristics of high-altitude workers, overcoming the misjudgment problem caused by human posture deformation or occlusion in traditional methods; at the same time, the power equipment features (such as cables and towers / poles) extracted by the YOLACT model can accurately capture the spatial correlation between equipment and personnel in the working scene. By integrating the human posture key point features and the power equipment features, the joint analysis of "human behavior-equipment operation" is realized, which significantly improves the recognition accuracy and effectiveness of high-risk actions unique to the power industry (such as touching wires when climbing towers and tool operation postures), thereby improving the accuracy of detection of the working status of power high-altitude workers.

[0052] The present invention adopts a multi-stage prediction architecture and heat map regression mechanism through the OpenPose network model, which can accurately identify the key points of the limbs under multi-angle shooting such as top view and side view. It can be well applied to the posture detection of personnel in the vertical structure scene of the power tower. Its branch processing capability can effectively distinguish overlapping and interlaced human targets in dense working scenes, reducing the posture false detection rate. At the same time, the present invention realizes high-precision instance segmentation of power equipment while maintaining real-time processing speed through the architecture design of the YOLACT model based on parallel mask generation and target detection. Its prototype generation network has a strong characterization capability for power equipment with complex textures (such as cables and towers / poles), and can clearly distinguish the boundary features of equipment and background, providing reliable spatial information for subsequent feature fusion. In addition, the present invention adopts the incremental learning characteristics of the online sequential extreme learning machine to support dynamic model updates for newly emerging working postures and equipment types, significantly enhancing the system's adaptability to changes in power working scenes (such as new safety equipment and seasonal differences in work clothes), and through the hidden layer node adaptive adjustment mechanism, it can effectively solve the classification boundary fuzzy problem caused by the expansion of the working posture library, ensuring long-term stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to make the purpose, technical solutions and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0054] Figure 1 Logic block diagram of the working status detection method for power high-altitude workers based on image feature fusion.

[0055] Figure 2 This is the network structure diagram of the OpenPose network.

[0056] Figure 3 This is the framework of the improved OpenPose network.

[0057] Figure 4 This is the network structure diagram of the MobileNet V2 module with attention mechanism.

[0058] Figure 5 Figure 2 is a network structure diagram of a convolutional neural network used to extract mask features.

[0059] Figure 6 Schematic diagram of the online sequential extreme learning machine.

[0060] Figure 7 Schematic diagram of implementing binary classification for an online sequential extreme learning machine.

[0061] Figure 8 This is a demonstration of the effect of the improved human posture estimation algorithm.

[0062] Figure 9 This figure shows the effect of the model on the high-altitude power operation dataset.

[0063] Figure 10 Classify and judge the effect for high-altitude power workers. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] It should be noted that similar reference numerals and letters denote similar items in the following figures. Therefore, once an item is defined in one figure, it does not require further definition or explanation in subsequent figures. In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" indicate positions or relationships based on the positions or relationships shown in the figures, or the positions or relationships in which the inventive product is typically placed when in use. These terms are intended solely to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation, and are therefore not to be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third," etc., are used solely to distinguish descriptions and are not to be construed as indicating or implying relative importance. Furthermore, terms such as "horizontal" and "vertical" do not imply that a component is absolutely horizontal or overhanging, but rather may be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but rather may be slightly tilted. In the description of the present invention, it should also be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0066] The following is a further detailed description through specific implementation methods:

[0067] Example:

[0068] This embodiment discloses a method for detecting the working status of power workers at high altitudes based on image feature fusion.

[0069] like Figure 1 As shown in FIG, a method for detecting the working status of electric power workers at high altitudes based on image feature fusion includes:

[0070] S1: Get the original image to be detected;

[0071] S2: Input the original image into the OpenPose network model to extract human posture features and obtain the human posture key point features of each person target (staff member) in the original image;

[0072] S3: Input the original image into the YOLACT model to extract the power equipment features and obtain the power equipment features;

[0073] S4: performing feature fusion on the key point features of the human body posture of each person target and the features of the power equipment to obtain the fusion features of each person target;

[0074] S5: Input the fusion features of each personnel target into the trained online sequential extreme learning machine for classification, determine whether each personnel target is performing high-altitude power work, and output the personnel targets performing high-altitude power work in the original image.

[0075] The present invention uses the human posture key point features (horizontal and vertical coordinates and confidence levels of 25 key points) extracted by the OpenPose network to accurately characterize the dynamic motion characteristics of high-altitude workers, overcoming the misjudgment problem caused by human posture deformation or occlusion in traditional methods. At the same time, the power equipment features (such as cables and towers / poles) extracted by the YOLACT model can accurately capture the spatial correlation between equipment and personnel in the work scene. By fusing the human posture key point features and power equipment features, a joint analysis of "human behavior-equipment operation" is achieved, significantly improving the recognition accuracy and effectiveness of high-risk actions unique to the power industry (such as touching wires when climbing towers and tool operation postures), thereby improving the accuracy of work status detection for high-altitude power workers. In addition, the present invention can also dynamically adjust the contribution of human posture features and equipment features through a spatial weight distribution mechanism during the feature fusion process, effectively alleviating interference problems such as dense equipment occlusion and lighting changes in high-altitude work scenes. For example, when part of a person's limbs are blocked by power equipment, the system can enhance the spatial correlation of equipment features and infer the complete action intention by combining visible limb postures, thereby improving classification robustness in complex environments.

[0076] The present invention adopts a multi-stage prediction architecture and heat map regression mechanism through the OpenPose network model, which can accurately identify the key points of the limbs under multi-angle shooting such as top view and side view. It can be well applied to the posture detection of personnel in the vertical structure scene of the power tower. Its branch processing capability can effectively distinguish overlapping and interlaced human targets in dense working scenes, reducing the posture false detection rate. At the same time, the present invention realizes high-precision instance segmentation of power equipment while maintaining real-time processing speed through the architecture design of the YOLACT model based on parallel mask generation and target detection. Its prototype generation network has a strong characterization capability for power equipment with complex textures (such as cables and towers / poles), and can clearly distinguish the boundary features of equipment and background, providing reliable spatial information for subsequent feature fusion. In addition, the present invention adopts the incremental learning characteristics of the online sequential extreme learning machine to support dynamic model updates for newly emerging working postures and equipment types, significantly enhancing the system's adaptability to changes in power working scenes (such as new safety equipment and seasonal differences in work clothes), and through the hidden layer node adaptive adjustment mechanism, it can effectively solve the classification boundary fuzzy problem caused by the expansion of the working posture library, ensuring long-term stability.

[0077] In order to better introduce the technical solution of the present invention, this embodiment is described through the following parts.

[0078] 1. OpenPose Network Model

[0079] like Figure 2 As shown in the figure, the OpenPose network is a network model for predicting and estimating human pose, obtained through supervised learning training on large-scale datasets. It mainly consists of three parts: image preprocessing, feature extraction, and pose estimation. First, spatial features are extracted from the image using several layers of CNNs composed of filters of different sizes. Second, a feature pyramid network structure is used to capture image features of different scales from different hierarchical structures, thereby better detecting various human poses. Finally, a specially designed branch network outputs the coordinates of the key points of the human body. The confidence level of the output is thresholded and low-confidence points are removed to ensure the reliability of the final output data. The final output key point information is saved and plotted on the original image for visual analysis. During training, the cross-entropy loss function is used to optimize the key point prediction, and the backpropagation algorithm is used to continuously update the weights to improve prediction accuracy.

[0080] The key points of the human body in the image are captured by the pre-trained OpenPose network: the processed 700×700 image dataset is input into the network, all the key points of the human body in the image are detected, and these key points are connected to form the human limbs for display. In the power scenario of the present invention, the key point features of the human body posture of the target extracted by the OpenPose network model include the horizontal and vertical coordinates (x and y) of 25 key points. i ,y i ) and confidence c i ; The key point features of human body posture include 75 feature data, which is a data matrix of [25,3].

[0081] The applicant found that in the process of identifying and estimating the posture of people in power background images, the background of the specific power high-altitude scene is complex, the human body accounts for a small proportion of the picture, and posture estimation errors often occur. The traditional pre-trained OpenPose model cannot adapt well to the power high-altitude environment and is prone to false detection or missed detection. Therefore, the present invention replaces the VGG-19 network of the backbone network in the OpenPose network model with a MobileNet V2 module with an attention mechanism. The improved OpenPose network model is as follows: Figure 3As shown in the figure, by adding the MobileNet V2 module with attention mechanism, the pose estimation performance of the OpenPose network and its adaptability to high-altitude power scenes can be improved.

[0082] MobileNet V2 is a lightweight convolutional neural network architecture. Compared with traditional convolutional neural networks, it uses a depth-wise separable convolution method to split the standard convolution operation into depth-wise convolution and point-wise convolution, significantly reducing the computational complexity and number of parameters of the model. It is particularly suitable for scenarios with limited resources and requirements for real-time performance. Figure 4 As shown in the figure, the MobileNet V2 module with attention mechanism includes a 1×1 convolution module, a 3×3 depth convolution module, a spatial and channel combined attention mechanism module, and a 1×1 convolution module connected end to end. Among them, the spatial and channel combined attention mechanism module (CBAM) is mainly composed of a channel attention unit (Channel Attention) and a spatial attention unit (SpatialAttention) connected in series, aiming to enhance the feature expression capability from both the channel dimension and the spatial dimension.

[0083] The workflow of the attention mechanism module that combines space and channels is as follows: First, the input feature map is processed by the channel attention mechanism and compressed in the spatial dimension through global average pooling (GlobalAverage Pooling) and global maximum pooling (GlobalMax Pooling). Then, the correlation between channels is extracted through a fully connected layer (or convolution operation) with shared weights, and the channel attention weights are generated through a Sigmoid activation function. Finally, they are multiplied channel by channel with the original feature map to achieve channel importance weighting. Next, the channel-weighted feature map is input into the spatial attention module. Two spatial description maps are generated by the same maximum pooling and average pooling operations along the channel dimension. After splicing them in the channel dimension, a convolution operation is performed to extract local spatial features. Finally, the spatial attention weight map is obtained through Sigmoid activation and multiplied element-by-element with the input feature map to enhance the spatial features of key areas.

[0084] 2. YOLACT Model

[0085] YOLACT (You Only Look At Coefficients) is an efficient, real-time instance segmentation algorithm first proposed by Bolya et al. at the University of California, Berkeley in 2019. Unlike traditional instance segmentation algorithms (such as Mask R-CNN), YOLACT significantly improves model inference speed by splitting the instance segmentation problem into two parallel subtasks: prototype generation and coefficient prediction. The key idea of YOLACT is to combine a shared mask prototype with the instance coefficients to quickly generate an instance mask for each object.

[0086] The YOLACT algorithm's main architecture consists of two core components: a foreground detection branch for object detection and a mask generation branch for instance segmentation. ResNet101 is used as the backbone feature extraction network to acquire deep features from the input image. A Feature Pyramid Network (FPN) extracts multi-scale features, and an anchor mechanism is used for object detection. The multi-layer residual block structure in ResNet101 effectively alleviates the vanishing gradient problem in deep networks, thereby extracting richer and more semantically informative features. The FPN further integrates these features at multiple scales, enabling the network to effectively detect objects of varying sizes. Building on this foundation, YOLACT uses the foreground detection branch to predict the object's category and bounding box. The mask generation branch independently generates a set of basic prototype masks, which are then linearly combined with the coefficients output by the foreground detection branch to generate the final mask for each instance. This allows YOLACT to simultaneously achieve efficient object detection and instance segmentation, offering the advantages of high real-time performance and excellent segmentation quality.

[0087] In this embodiment, the power equipment features include several power equipment categories, as well as detection box position information, detection box position confidence score, and power equipment semantic segmentation mask corresponding to each power equipment category.

[0088] Specifically:

[0089] ① tensor0 represents the power equipment category label, ranging from 0 to 3 (four types), corresponding to cable, tower-lattice, tower-tucohy, and tower-wooden.

[0090] ② tensor1 represents the position information of the detection box, including the coordinates of the upper left corner and lower right corner of each detection box, forming a 4-dimensional array;

[0091] ③tensor2 represents the confidence score of each detection box position;

[0092] ④tensor3 is an instance segmentation mask filled with 0s and 1s with the same resolution as the original input image, which is used to accurately describe the shape and boundary of each power device.

[0093] The applicant discovered that the YOLACT network had difficulty distinguishing overlapping objects when detecting power equipment in images, and that the same object was detected multiple times, leading to false detections. To address this issue, the present invention made improvements in two aspects: loss function optimization and bounding box selection mechanism.

[0094] 1. Introducing generalized intersection-union into the loss function

[0095] In deep learning image segmentation tasks, commonly used loss functions such as cross-entropy and Dice coefficient can measure the gap between predicted results and true labels to a certain extent. However, when faced with imbalanced datasets, they often lead to poor prediction performance for small objects or uncommon categories. To improve the model's precision and recall for target regions, this paper introduces the generalized intersection-over-union loss as an improved loss function.

[0096] Generalized Intersection over Union (GIoU) is an extension of the traditional Intersection over Union (IoU) that not only considers the intersection and union of the predicted box and the true box, but also introduces the distance information between the boxes. Unlike IoU, which only relies on the ratio of intersection and union, GIoU can better handle predicted boxes that are far apart but not completely overlapping by penalizing those boxes that do not completely overlap, thereby providing more accurate boundary positioning in object detection and image segmentation. Specifically, the GIoU loss function calculates the GIoU value of the predicted box and the true box, and then uses it as an optimization target to minimize the difference between the predicted box and the true box.

[0097] Specifically, when training the YOLACT model, the total loss function of the YOLACT model is expressed as:

[0098] L total =L bbox +λ GIoU ·L GIoU ;

[0099] L GIoU =1-GIoU(A,B);

[0100]

[0101] Where: L toTAl Indicates adding generalized intersection loss L GIoU The total loss of the post-YOLACT model; L bboxrepresents the bounding box regression loss (i.e., the loss of the existing YOLACT model); λ GIoU Represents a hyperparameter used to control the generalized intersection-over-union loss L GIoU Impact on the total loss; GIoU(A,B) represents the generalized intersection-over-union ratio between the predicted box A and the true box B; U represents the union area of the predicted box and the true box, that is, their total area minus the intersection part; A c Represents the minimum enclosed area of the predicted box A and the real box B (the area of the smallest box that contains both the predicted box and the real box).

[0102] By introducing GIoU into the loss function, the present invention can significantly improve the performance of the model when dealing with irregular shapes and areas far from the target, especially when the boundary between the target and the background is blurred or there is little overlap. In addition, the introduction of GIoU also enhances the accuracy of the model for the target position and shape, so that the model can not only predict the category of the target, but also predict the boundary of the target more accurately, thereby improving the overall performance of the image segmentation task. By introducing the generalized intersection-over-union loss, the model can

[0103] 2. Introducing adaptive non-maximum suppression

[0104] The present invention replaces the NMS module in the YOLACT model with an adaptive non-maximum suppression (ANMS) module to reduce the redundant candidate boxes generated during the detection process and retain the most representative detection box when multiple high-confidence boxes overlap.

[0105] Adaptive non-maximum suppression (ANMS) is a technique used to suppress redundant candidate boxes in object detection. Its primary purpose is to reduce the number of candidate boxes generated during detection, particularly when multiple high-confidence boxes overlap, and to retain the most representative boxes. By introducing an adaptive weighting mechanism, it more intelligently considers the relative position, size, and category information between objects when selecting candidate boxes, resulting in more precise suppression. Traditional non-maximum suppression (NMS) methods suppress redundant boxes based on the degree of overlap between candidate boxes (typically measured using the intersection-over-union (IoU) ratio). Unlike the fixed threshold method of traditional NMS, ANMS automatically adjusts the suppression strength based on the local context and features of the candidate box, enabling better adaptation to the distribution of objects in different scenarios and avoiding the loss of valuable detection boxes due to overly strict thresholds. Therefore, ANMS calculates an adaptive weight for each candidate box, which comprehensively considers factors such as the box's confidence, the distance between boxes, and the box's size.

[0106] In this way, ANMS can avoid excessive suppression while ensuring that each target region has only one representative box, thereby improving the accuracy and robustness of object detection tasks. In some object detection tasks, ANMS has been proven to effectively improve detection accuracy, especially in images with multiple objects densely packed and overlapping, significantly improving the quality and quantity of target detection boxes. Therefore, ANMS is widely used in tasks such as pedestrian detection, license plate recognition, and multi-target detection in remote sensing images.

[0107] After the present invention introduces adaptive non-maximum suppression into the YOLACT algorithm, its detection performance in multi-target dense scenes can be significantly improved. Traditional NMS suppresses redundant detection frames by calculating the IoU between detection frames, but the fixed threshold is prone to mistakenly suppressing some important frames when the targets overlap a lot. To solve this problem, the NMS module is replaced with the ANMS module in the post-processing stage of YOLACT. ANMS uses an adaptive weight mechanism to not only consider the confidence when retaining candidate frames, but also combines the relative position, size and other information between frames to dynamically adjust the suppression strength. It calculates a weight based on the local context information of the detection frame, and combines the GIoU value to determine whether to retain the current detection frame, thereby effectively avoiding the loss of valid detection frames under a fixed IoU threshold. In this way, it can more accurately process scenes with dense or overlapping targets, retain the most representative detection frames, improve the accuracy and robustness of detection, and ultimately enhance the performance of YOLACT in multi-target complex scenes.

[0108] 3. Feature Fusion

[0109] In this embodiment, feature fusion is performed through the following steps:

[0110] S401: Obtain the horizontal and vertical coordinates (x and y) of each person target, including 25 key points i ,y i ) and confidence c i The human body posture key point features, as well as the power equipment features including several power equipment categories, as well as the detection box position information, detection box position confidence score and power equipment semantic segmentation mask corresponding to each power equipment category;

[0111] In this embodiment, the two data sources, human posture key point data (data1) and power equipment characteristic data (data2), are stored in two different folders and saved in .npy format. During the data processing phase, all .npy files are read from the two folders to check whether the corresponding files exist and whether they correspond one to one. For each pair of files in the same sample, the corresponding data is loaded and extracted. If the data is valid, the extracted task key point information and the power equipment characteristic information for the scenario are fused and classified to ensure that these features are correctly processed and standardized.

[0112] S402: Perform feature extraction (feature dimensionality reduction) on the semantic segmentation mask in each power equipment category through a convolutional neural network to obtain corresponding mask extraction features;

[0113] Combine Figure 5 As shown, the convolutional neural network includes a first convolutional layer with a dimension of 8×700×700, a first pooling layer with a dimension of 8×350×350, a second convolutional layer with a dimension of 16×350×350, a second pooling layer with a dimension of 16×175×175, a third convolutional layer with a dimension of 32×175×175, a third pooling layer with a dimension of 32×87×87, a flattening operation layer (used to flatten 32×87×87 to 241056), and a first fully connected layer and a second fully connected layer, which are connected end to end in sequence; the second fully connected layer outputs a 36-dimensional mask extraction feature.

[0114] In order to effectively extract useful information from these mask features, Figure 5 The convolutional neural network shown is specifically designed for processing mask data. The network first progressively extracts input mask features through multiple layers of convolution operations, with each convolution kernel capturing features at different scales and local locations. Pooling is then used to reduce the spatial size of the feature map, reducing computational complexity and enhancing feature abstraction. The network architecture begins with a four-channel input mask feature map and, through several layers of convolution and max pooling, ultimately produces a smaller, high-level feature representation. After convolution extraction, fully connected layers are used to further compress the convolutional features and output classification or regression results for subsequent tasks. This processing approach fully utilizes the spatial information of the mask and extracts high-dimensional features related to device characteristics for subsequent analysis and classification. This mask processing method not only captures local device details but also, through multiple layers of convolution, constructs global relationships between devices, making it an effective method for efficiently modeling electrical device characteristics.

[0115] S403: Perform tensor splicing on the key point features of the human body posture of each personnel target and the position information, confidence score and mask extraction features corresponding to each power equipment category to obtain the fusion features of each personnel target.

[0116] In this embodiment, to avoid feature redundancy and increased computational complexity, the device features were subjected to dimensionality reduction. Dimensionality reduction techniques such as principal component analysis (PCA) were used to extract the most representative low-dimensional features, preserving the key information of the features and reducing interference from irrelevant dimensions.

[0117] 4. Online Sequential Extreme Learning Machine

[0118] The Extreme Learning Machine (ELM) algorithm is based on a single-hidden-layer neural network structured primarily as an input layer, one hidden layer, and an output layer. Unlike traditional neural networks, which continuously adjust and optimize initialized weight parameters through backpropagation, the ELM algorithm stops adjusting the input and hidden layer parameters after initialization and only fine-tunes the output layer parameters with the goal of minimizing loss. This optimizes the model, reducing computational effort while maintaining accuracy and ensuring speed.

[0119] Although ELM has significant advantages in training speed and computational efficiency, the basic ELM can only learn model parameters once through training. After training, the model parameters are no longer modified, lacking real-time updating capabilities and failing to adapt well to the ever-changing conditions at high-altitude power work sites. This can easily lead to misjudgment risks for power workers working in complex backgrounds, and is incompatible with the real-time requirements of monitoring systems. To better assess personnel and identify other targets during high-altitude power work, an online sequential extreme learning machine (OS-ELM) was designed to continuously learn data generated at high-altitude power work sites. The Online Sequential Extreme Learning Machine (OS-ELM) has low computational complexity and memory requirements, making it suitable for real-time or online learning. It offers significant advantages for dynamic data that may be encountered in real-world scenarios.

[0120] Combine Figure 6 As shown in Figure 2, the processing steps for training an online sequential extreme learning machine are as follows:

[0121] S501: Acquire several groups of samples to construct a training set and a test set, where each group of samples includes key point features of a person's body posture and features of power equipment in the same image;

[0122] S502: For samples in the training set and the test set: the key points in the key point features of the human body posture of the target are connected and sorted according to the human body structure, and the working status label of the target is marked, where the label 0 indicates that no electrical high-altitude work is being performed, and 1 indicates that electrical high-altitude work is being performed; the key point features of the human body posture of the target and the power equipment features are fused to obtain training data, and the input data is associated with the working status label of the target; the key point features of the human body posture, the power equipment features, and the mask extraction features are tensor-concatenated to obtain a 75+20+36=131-dimensional fusion feature as the training data, where the first 75 dimensions are human body posture features and the last 56 dimensions are power equipment features. The working status label is placed in the last column of the training data, so that the training data in each training set is 132-dimensional;

[0123] S503: Using the training data in the training set as input to the online sequential extreme learning machine to train and update the weight β; wherein the supervision label of the online sequential extreme learning machine is two-dimensionally one-hot encoded, where the bit with index 0 represents no high-altitude power work, and the bit with index 1 represents current high-altitude power work;

[0124] The formula for updating the weight β is expressed as:

[0125]

[0126] Where: H is the feature matrix of training data (fusion features), T is the label matrix;

[0127] S504: Repeat step S503 to iteratively train and update the weight β until the online sequential extreme learning machine converges;

[0128] S505: Input the training data and trained weights β in the test set into the online sequential extreme learning machine and output the corresponding prediction result t pre =[t pre0 ,t pre1 ]:When t pre0 >t pre1 When t pre0 <t pre1 When the predicted label is 1, it means that electrical overhead work is in progress. The performance of the online sequential extreme learning machine is evaluated by the labels of the test set.

[0129] Combine Figure 7 As shown in the figure, in actual application, the fusion features of each personnel target are input into the trained online sequential extreme learning machine for classification, and the prediction result t corresponding to each personnel target is output. pre =[t pre0 ,t pre1 ]:When tpre0 >t pre1 When the prediction is 0, it means that the personnel target has not performed electrical high-altitude work; when t pre0 <t pre1 When the prediction is 1, it means that the personnel target is performing electrical high-altitude operations.

[0130] 5. Weight Matrix

[0131] For the 75-dimensional human body key point information, the applicant believes that different key points of the human body have different degrees of influence on classification. Some key points (such as shoulders and keys) contribute more to the classification task, while other key points (such as fingertips and toes) have less influence on the results. To address this problem, the present invention sets a learnable weight matrix for the 25 key points of the human body posture key point features in the input layer of the online sequential extreme learning machine, and each element of the weight matrix corresponds to a specific weight of a key point; during the training process of the online sequential extreme learning machine, the weight matrix is dynamically updated according to the classification loss function to achieve feature weighting of different key points. At the same time, the present invention proposes a dynamic weight learning mechanism, and the model can automatically adjust the weight of each key point during the training process to achieve more accurate classification. This mechanism not only improves the generalization ability of the model, but also effectively reduces the negative impact of unimportant features on the classification results.

[0132] Specifically, the processing steps for dynamically updating the weight matrix include:

[0133] S201: Initialize the weight matrix to the identity matrix, that is, each key point has the same weight;

[0134] S202: Perform weighted calculation on the extracted human posture key point feature X and the weight matrix W to obtain the weighted human posture key point feature; wherein the human posture key point feature X∈R n×75 , where n is the number of samples, 75 is the feature dimension of the key points of the human body; the weight matrix W∈R 25×3 , each element W in the weight matrix W ij represents the weight of the i-th key j-th feature;

[0135] The formula is:

[0136] X′=X·W;

[0137] S203: Fusing the weighted human posture key point features X′ and the power equipment features to obtain corresponding fused features; then inputting the fused features into an online sequential extreme learning machine for classification, and calculating the classification loss function of the online sequential extreme learning machine;

[0138] S204: Using the defined back propagation algorithm, the update formula of the weight matrix W is:

[0139]

[0140] Where: α represents the learning rate; L represents the classification loss of the online sequential extreme learning machine;

[0141] S205: Repeat steps S202 to S204 to iteratively train the online sequential extreme learning machine until convergence, and obtain an updated weight matrix.

[0142] In actual application, in step S4, the key point features of the human body posture are first multiplied element-by-element with the updated weight matrix to adjust the contribution of each key point in the feature space to obtain the weighted key point features of the human body posture; then the weighted key point features of the human body posture and the features of the power equipment are fused to obtain the corresponding fusion features.

[0143] 6. Experimental Description

[0144] 1. Human key point detection in images of power high-altitude work

[0145] After adding the MobileNet V2 module with an attention mechanism, OpenPose adaptively weights different feature data, improving the model's recognition capabilities. It is more adaptable to occlusion between people and between people and objects, as well as for identifying workers in high-altitude power plant operations. The OpenPose model, equipped with the MobileNet V2 module with an attention mechanism, estimates human poses in images of high-altitude power plant operations and saves the human pose feature data corresponding to each sample.

[0146] Figure 8 The improved human pose estimation algorithm was demonstrated for detection on a dataset of electrical power aerial work. The results show that the model correctly identifies all visible key points in the image and also accurately predicts some occluded key points. The poses of people in the electrical power aerial work scene are accurately identified with a low probability of false detection. Virtually all people in the image can be identified, indicating that each person extracted by the model can be considered a sample in subsequent research. Saving the extracted feature vectors per sample can be used in classification models for further fusion analysis.

[0147] 2. Power equipment detection in power high-altitude operation images

[0148] The trained model was used to identify four types of electrical equipment (cable, tower-lattice, tower-tucohy, and tower-wooden). During the evaluation process, the model achieved an FPS of 21.60, demonstrating strong real-time performance. The average precision (AP) of the box and mask at different thresholds was also presented. Box AP was 46.08 at an IoU of 0.5. Mask AP was 30.59 at an IoU of 0.5.

[0149] The model is used to obtain and collect four tensors of power equipment feature data: tensor_0, tensor_1, tensor_2, and tensor_3, which represent the device category, confidence, device location information, and the mask image of the device, respectively. Figure 9 The following demonstrates some of the recognition results in the high-altitude power operation dataset. In order to ensure the consistency of each sample in the dataset, the dimensions of all tensors are adjusted to be consistent. If the number of elements in some tensors is insufficient, we supplement the corresponding data. For the equipment category (tensor_0), we use -1 as a placeholder, and for the confidence (tensor_1), location information and mask features, we fill in 0. In this way, the features of each sample are supplemented to the same dimension to ensure data integrity. The mask feature represents the location and category information of each power equipment in the image. These features are multi-channel and are used to describe the semantic information of different equipment (such as utility poles, transmission lines, etc.).

[0150] 3. Staff status judgment

[0151] In this experiment, OS-ELM was used to perform a binary classification of high-altitude work by power workers. The model's effectiveness in risk detection at power work sites was verified, particularly in terms of real-time performance and adaptability. The dataset used in the experiment contained images of power workers both working at height and not working at height. After feature extraction and fusion, each image generated a fused feature vector z, which served as the input for OS-ELM. Label data was encoded using a two-dimensional one-hot encoding, where bits indexed 0 indicate no high-altitude work and bits indexed 1 indicate active high-altitude work. The experiment consisted of two phases: training and testing. In the training phase, the OS-ELM model was trained using the training set data and corresponding labels to obtain weights β. In the testing phase, predictions were made using the test set input data and the trained weights β to obtain output values. Finally, the predicted category was determined using the argmax function.

[0152] Figure 10The improved human pose estimation algorithm demonstrates the effectiveness of the proposed algorithm in determining whether a person is performing high-altitude electrical work in an image of high-altitude electrical work. Label 1 indicates that a person is performing high-altitude work, and label 0 indicates that a person is not performing high-altitude work.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the technical solutions. Those skilled in the art should understand that modifications or equivalent replacements of the technical solutions of the present invention that do not depart from the purpose and scope of the technical solutions of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting the working status of power workers at high altitudes based on image feature fusion, characterized in that: include: S1: Get the original image to be detected; S2: Input the original image into the OpenPose network model to extract human posture features and obtain the key point features of the human posture of each person target in the original image; S3: Input the original image into the YOLACT model to extract the power equipment features and obtain the power equipment features; S4: performing feature fusion on the key point features of the human body posture of each person target and the features of the power equipment to obtain the fusion features of each person target; S5: Input the fusion features of each personnel target into the trained online sequential extreme learning machine for classification, determine whether each personnel target is performing high-altitude power work, and output the personnel targets performing high-altitude power work in the original image.

2. The method for detecting the working status of electric power workers at high altitudes based on image feature fusion according to claim 1, wherein: In step S2, the VGG-19 network of the backbone network in the OpenPose network model is replaced with a MobileNet V2 module with an attention mechanism; The MobileNet V2 module with attention mechanism includes a 1×1 convolution module, a 3×3 depth convolution module, a spatial and channel attention mechanism module, and a 1×1 convolution module connected end to end. The spatial and channel attention mechanism module is composed of a channel attention unit and a spatial attention unit in series, which is used to enhance feature expression capabilities from both the channel dimension and the spatial dimension.

3. The method for detecting the working status of electric power workers at high altitudes based on image feature fusion according to claim 1, wherein: In step S2, the key features of the human body posture of the target extracted by the OpenPose network model include the horizontal and vertical coordinates (x i ,y i ) and confidence c i .

4. The method for detecting the working status of electric power workers at high altitudes based on image feature fusion according to claim 1, wherein: In step S3, when training the YOLACT model, the generalized intersection-over-union loss L is introduced into the loss function of the YOLACT model. GIoU ; The total loss function of the YOLACT model is expressed as: L total =L bbox +λ GIoU ·L GIoU ; L GIoU =1-GIoU(A,B); Where: L total Indicates adding generalized intersection loss L GIoI The total loss of the post-YOLACT model; L bbox represents the bounding box regression loss; λ GIoU Represents a hyperparameter used to control the generalized intersection-over-union loss L GIoU Impact on the total loss; GIoU(A,B) represents the generalized intersection-over-union ratio between the predicted box A and the true box B; U represents the union area of the predicted box and the true box; A c Represents the minimum enclosed area of the predicted box A and the true box B.

5. The method for detecting the working status of electric power workers at high altitudes based on image feature fusion according to claim 1, characterized in that: In step S3, the power equipment features include several power equipment categories, as well as the detection box position information, detection box position confidence score and power equipment semantic segmentation mask corresponding to each power equipment category.

6. The method for detecting the working status of electric power workers at high altitudes based on image feature fusion according to claim 5, characterized in that: In step S3, the NMS module in the YOLACT model is replaced with an adaptive non-maximum suppression module to reduce the redundant candidate boxes generated during the detection process and retain the most representative detection box when multiple high-confidence boxes overlap.

7. The method for detecting the working status of electric power workers at high altitudes based on image feature fusion according to claim 1, characterized in that: In step S4, feature fusion is performed through the following steps: S401: Obtain the horizontal and vertical coordinates (x and y) of each person target, including 25 key points i ,y i ) and confidence c i The human body posture key point features, as well as the power equipment features including several power equipment categories and the detection box position information corresponding to each power equipment category, the detection box position confidence score and the power equipment semantic segmentation mask; S402: extracting features from the semantic segmentation mask in each power equipment category using a convolutional neural network to obtain corresponding mask extraction features; S403: Perform tensor splicing on the key point features of the human body posture of each personnel target and the position information, confidence score and mask extraction features corresponding to each power equipment category to obtain the fusion features of each personnel target.

8. The method for detecting the working status of electric power workers at heights based on image feature fusion according to claim 7, characterized in that: In step S402, the convolutional neural network includes a first convolutional layer with a dimension of 8×700×700, a first pooling layer with a dimension of 8×350×350, a second convolutional layer with a dimension of 16×350×350, a second pooling layer with a dimension of 16×175×175, a third convolutional layer with a dimension of 32×175×175, a third pooling layer with a dimension of 32×87×87, a flattening operation layer, and a first fully connected layer and a second fully connected layer, which are connected end to end in sequence; the second fully connected layer outputs a mask to extract features.

9. The method for detecting the working status of electric power workers at high altitudes based on image feature fusion according to claim 1, characterized in that: In step S5, the processing steps for training the online sequential extreme learning machine are as follows: S501: Acquire several groups of samples to construct a training set and a test set, where each group of samples includes key point features of a person's body posture and features of power equipment in the same image; S502: For samples in the training set and the test set: connect and sort the key points in the key point features of the human body posture of the target according to the human body structure, and label the working status label of the target, where the label 0 indicates that no high-altitude power work is being performed, and 1 indicates that high-altitude power work is being performed; perform feature fusion on the key point features of the human body posture of the target and the features of the power equipment to obtain training data, and associate the input data with the working status label of the target; S503: Using the training data in the training set as input to the online sequential extreme learning machine to train and update the weight β; wherein the supervision label of the online sequential extreme learning machine is two-dimensionally one-hot encoded, where the bit with index 0 represents no high-altitude power work, and the bit with index 1 represents current high-altitude power work; The formula for updating the weight β is expressed as: Where: H is the feature matrix of training data, T is the label matrix; S504: Repeat step S503 to iteratively train and update the weight β until the online sequential extreme learning machine converges; S505: Input the training data and trained weights β in the test set into the online sequential extreme learning machine and output the corresponding prediction result t pre =[t pre0 ,t pre1 ]:When t pre0 >t pre1 When t pre0 <t pre1 When the predicted label is 1, it means that electrical high-altitude work is in progress.

10. The method for detecting the working status of electric power workers at heights based on image feature fusion according to claim 3, characterized in that: In step S5, a learnable weight matrix is set for the 25 key points of the human body posture key point features in the input layer of the online sequential extreme learning machine, where each element of the weight matrix corresponds to a specific weight of a key point; During the training process, the online sequential extreme learning machine dynamically updates the weight matrix according to the classification loss function to achieve feature weighting of different key points; The processing steps for dynamically updating the weight matrix include: S201: Initialize the weight matrix to the identity matrix, that is, each key point has the same weight; S202: performing weighted calculation on the extracted human posture key point features X and the weight matrix W to obtain weighted human posture key point features; The formula is: X′=X·W; S203: Fusing the weighted human posture key point features X′ and the power equipment features to obtain corresponding fused features; then inputting the fused features into an online sequential extreme learning machine for classification, and calculating the classification loss function of the online sequential extreme learning machine; S204: Using the defined back propagation algorithm, the update formula of the weight matrix W is: Where: α represents the learning rate; L represents the classification loss of the online sequential extreme learning machine; S205: Repeat steps S202 to S204 to iteratively train the online sequential extreme learning machine until convergence, and obtain an updated weight matrix.

Images