Method and system for identifying character interaction behaviors on electric power operation site based on image information

Abstract

The invention discloses an image information-based electric power operation site character interaction behavior identification method and system, and belongs to the technical field of electric power operation. The method comprises the following steps: collecting image information for a power operation site, detecting the collected image information by using a target detection network, and outputting key information; based on a human body posture estimation network, according to the key information, performing human body posture estimation, and obtaining position information of skeleton points of the operator in the image information; and on the basis of a new partition strategy network, according to the position information of the skeleton points of the operating personnel, identifying the character interaction behavior of the operating personnel in the operating field. According to the method, the electric power operation can be carried out orderly according to the steps in the electric power operation process, whether the working state of an operator in the key steps of the electric power operation meets the stipulated risk condition or not can be seen, and safe operation of the electric power operation is greatly guaranteed.

Description

Technical Field

[0001] This invention relates to the field of power operation technology, and more specifically, to a method and system for recognizing human interaction behavior at power operation sites based on image information. Background Technology

[0002] Image recognition refers to the process of processing, analyzing, and understanding images using information processing and computer technology and mathematical methods. It is currently widely used in fields such as barcode recognition, handwriting recognition, facial recognition, intelligent transportation, and computer vision.

[0003] The power industry is vigorously promoting the integration of information technology and safe production, and actively adopting advanced technologies such as video surveillance and image analysis, accumulating a large amount of video and image information, which provides a solid data foundation for the development of intelligent safety management and control technologies at work sites. Patent applications in the field of image data analysis and recognition at power work sites, addressing risks related to equipment, personnel, and the environment, mainly involve applications such as the detection of hazardous materials and the identification of simple violations. With the continuous improvement of computer performance and the mature application of new algorithm structures such as deep learning, patents in the field of image and video data analysis are developing rapidly.

[0004] Existing technologies include a method, system, medium, and electronic device for identifying power equipment. Based on acquired images and a pre-set convolutional neural network model, the location and identification results of equipment at the power operation site are obtained. This patent enhances the model's ability to detect small targets, using a convolutional neural network to obtain the location and identification of power equipment based on acquired image information. While it solves the problem of power equipment location and identification, this technology does not address the risk of personnel and environmental information associated with the image equipment, and the effective information obtained by the algorithm is limited.

[0005] One existing video analysis method combines object detection and human pose estimation. This method acquires video streams from on-site cameras via a streaming media server, extracts video frames, and obtains decoded images. Deep learning algorithms are used to quickly detect objects in each image, identifying all identifiable object categories. Risk information is analyzed, and real-time video stream analysis is continuously performed. While this method uses two-dimensional images for hazard warnings, it suffers from issues like overlapping and occlusion in two-dimensional images. False alarms are easily generated when workers and hazardous areas appear in the same image, and the image data can only be used for basic motion detection. Summary of the Invention

[0006] To address the above problems, this invention proposes a method for recognizing human interaction behavior at power operation sites based on image information, comprising:

[0007] For power operation sites, image information is collected, and the collected image information is detected using a target detection network to output key information;

[0008] Based on the human pose estimation network, human pose estimation is performed according to the key information to obtain the position information of the skeletal points of the worker in the image information;

[0009] Based on the new partitioning strategy network, the human interaction behavior of the workers at the work site is identified according to the location information of the workers' skeleton points.

[0010] Optional, key information includes: the number of operators and the type and location of the equipment.

[0011] Optionally, the acquired image information can be used to perform object detection using a network, including:

[0012] The target detection network is used to extract deep features from the image information to obtain the categories of workers and power equipment in the image information, as well as the number and location information of workers and the location of power equipment.

[0013] Optional, object detection networks, including: lightweight network models.

[0014] Optional, object detection networks include:

[0015] Projection extended projection structure PEP, extended projection structure EP, lightweight fully connected attention structure FCA;

[0016] in:

[0017] PEP is obtained by concatenating two 1×1 convolutional layers, a 3×3 DW convolution, and then another 1×1 convolution.

[0018] EP is obtained by concatenating two 1×1 convolutional layers with a 3×3 DW convolution inserted in between;

[0019] FCA is obtained by connecting two fully connected layers in series;

[0020] By stacking PEP, EP, and FCA, and then using some of the stacked layers as inputs to the next layer in parallel, the target detection network is obtained.

[0021] Optionally, the predicted bounding boxes of the object detection network include: four predicted bounding box offsets, and the predicted bounding boxes are redundant using the non-maximum suppression algorithm (NMS).

[0022] Optional, the human pose estimation network HRNet includes: four stages. The first stage includes: Bottleneck residual units with 4 output channels, where the input feature map is processed by 1×1, 3×3 and 1×1 convolutions (including convolutional layers, BN layers and ReLU layers) to generate a new feature map. The newly generated feature map is fused with the input feature map and used as the output feature map.

[0023] The second, third, and fourth stages have 1, 4, and 3 resolution blocks, respectively. Each resolution block includes 4 Basic residual units. The input feature map is processed by two 3×3 convolutions to generate a new feature map. The newly generated feature map is fused with the input feature map to serve as the output feature map.

[0024] Optionally, the new partitioning strategy network extracts and learns features from the human skeleton information through a human motion recognition model, and then identifies the human interaction behavior of the workers at the work site based on the position information of the workers' skeletal points.

[0025] Furthermore, this invention also proposes a power operation site human interaction behavior recognition system based on image information, comprising:

[0026] The power equipment positioning and identification module is used to collect image information at power operation sites, use a target detection network to detect the collected image information, and output key information.

[0027] The human pose estimation module is used to perform human pose estimation based on the key information and the human pose estimation network to obtain the position information of the skeletal points of the worker in the image information.

[0028] The power operation identification module is used to identify the human interaction behavior of the workers at the work site based on the location information of the worker's skeleton points, according to the new partitioning strategy network.

[0029] Optional, key information includes: the number of operators and the type and location of the equipment.

[0030] Optionally, the acquired image information can be used to perform object detection using a network, including:

[0031] The target detection network is used to extract deep features from the image information to obtain the categories of workers and power equipment in the image information, as well as the number and location information of workers and the location of power equipment.

[0032] Optional, object detection networks, including: lightweight network models.

[0033] Optional, object detection networks include:

[0034] Projection extended projection structure PEP, extended projection structure EP, lightweight fully connected attention structure FCA;

[0035] in:

[0036] PEP is obtained by concatenating two 1×1 convolutional layers, a 3×3 DW convolution, and then another 1×1 convolution.

[0037] EP is obtained by concatenating two 1×1 convolutional layers with a 3×3 DW convolution inserted in between;

[0038] FCA is obtained by connecting two fully connected layers in series;

[0039] By stacking PEP, EP, and FCA, and then using some of the stacked layers as inputs to the next layer in parallel, the target detection network is obtained.

[0040] Optionally, the predicted bounding boxes of the object detection network include: four predicted bounding box offsets, and the predicted bounding boxes are redundant using the non-maximum suppression algorithm (NMS).

[0041] Optional, the human pose estimation network HRNet includes: four stages. The first stage includes: Bottleneck residual units with 4 output channels, where the input feature map is processed by 1×1, 3×3 and 1×1 convolutions (including convolutional layers, BN layers and ReLU layers) to generate a new feature map. The newly generated feature map is fused with the input feature map and used as the output feature map.

[0042] The second, third, and fourth stages have 1, 4, and 3 resolution blocks, respectively. Each resolution block includes 4 Basic residual units. The input feature map is processed by two 3×3 convolutions to generate a new feature map. The newly generated feature map is fused with the input feature map to serve as the output feature map.

[0043] Optionally, the new partitioning strategy network extracts and learns features from the human skeleton information through a human motion recognition model, and then identifies the human interaction behavior of the workers at the work site based on the position information of the workers' skeletal points.

[0044] In another aspect, the present invention also provides a computing device, comprising: one or more processors;

[0045] A processor is used to execute one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the method described above is implemented.

[0047] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described above.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] This invention provides a method for recognizing human interaction behavior at power operation sites based on image information. The method includes: acquiring image information at the power operation site; using a target detection network to detect the acquired image information and output key information; performing human pose estimation based on the key information using a human pose estimation network to obtain the position information of the worker's skeletal points in the image information; and identifying the human interaction behavior of the worker at the operation site based on the position information of the worker's skeletal points using a new partitioning strategy network. This invention not only ensures that power operations are carried out in an orderly manner according to steps, but also allows for monitoring of the worker's working status at key steps to ensure compliance with regulations and identify any risks, greatly guaranteeing the safety of power operations. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method of the present invention;

[0051] Figure 2 This is a structural diagram of the system of the present invention. Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described with reference to the accompanying drawings. However, the invention may be embodied in many different forms and is not limited to the embodiments described herein. These embodiments are provided to fully and completely disclose the invention and to fully convey its scope to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the drawings is not intended to limit the invention. In the drawings, the same units / elements are referred to by the same reference numerals.

[0053] Unless otherwise stated, the terms used herein (including technical terms) have their common meaning as understood by one of ordinary skill in the art. Furthermore, it is understood that terms defined in commonly used dictionaries should be understood to have a meaning consistent with the context of their relevant field, and not to be interpreted as having an idealized or overly formal meaning.

[0054] Example 1:

[0055] This invention proposes a method for recognizing human interaction behavior at power operation sites based on image information, such as... Figure 1 As shown, it includes:

[0056] Step 1: Collect image information at the power operation site, use a target detection network to detect the collected image information, and output key information;

[0057] Step 2: Based on the human pose estimation network, perform human pose estimation according to the key information to obtain the position information of the skeletal points of the worker in the image information;

[0058] Step 3: Based on the new partitioning strategy network, identify the human interaction behavior of the workers at the work site according to the location information of the worker skeleton points.

[0059] Key information includes: the number of workers, the type of equipment, and its location.

[0060] The process involves using an object detection network to detect the acquired image information, including:

[0061] The target detection network is used to extract deep features from the image information to obtain the categories of workers and power equipment in the image information, as well as the number and location information of workers and the location of power equipment.

[0062] The object detection network includes a lightweight network model.

[0063] The object detection network includes:

[0064] Projection-Expansion-Projection (PEP), Extended Projection (EP), and Lightweight Fully-Connected Attention (FCA) structures;

[0065] in:

[0066] PEP is obtained by concatenating two 1×1 convolutional layers, a 3×3 DW convolution, and then another 1×1 convolution.

[0067] EP is obtained by concatenating two 1×1 convolutional layers with a 3×3 DW convolution inserted in between;

[0068] FCA is obtained by connecting two fully connected layers in series;

[0069] By stacking PEP, EP, and FCA, and then using some of the stacked layers as inputs to the next layer in parallel, the target detection network is obtained.

[0070] The predicted bounding boxes of the object detection network include four predicted bounding box offsets and redundant predicted bounding boxes using the Non-Maximum Suppression (NMS) algorithm.

[0071] The human pose estimation network HRNet consists of four stages. The first stage includes four output channels of Bottleneck residual units. The input feature map is processed by 1×1, 3×3 and 1×1 convolutions (including convolutional layers, BN layers and ReLU layers) to generate a new feature map. The newly generated feature map is fused with the input feature map to serve as the output feature map.

[0072] The second, third, and fourth stages have 1, 4, and 3 resolution blocks, respectively. Each resolution block includes 4 Basic residual units. The input feature map is processed by two 3×3 convolutions to generate a new feature map. The newly generated feature map is fused with the input feature map to serve as the output feature map.

[0073] The new partitioning strategy network extracts and learns features from human skeleton information through a human motion recognition model, and then identifies the human interaction behavior of the workers at the work site based on the position information of the workers' skeletal points.

[0074] The invention will be further illustrated below with specific examples:

[0075] The implementation process for a specific case is as follows:

[0076] (1) Facial recognition is used in the system, and only personnel with work permits can be identified to perform the following risk identification operations. Then, in the image information acquisition part, the risk operation images of the operation site are mainly images of the operation of equipment such as metering boxes, wiring terminals, and current transformers, and are uploaded to the system;

[0077] The specific steps for information collection include:

[0078] First, based on CNN (Convolutional Neural Network) face recognition, deep feature extraction of workers' faces is performed using convolutional neural networks. After the equipment collects the image information of the on-site workers, it is compared with the face information in the database to confirm their identity before the risk identification image information is collected.

[0079] Then, the operators upload various power operation equipment to the server. The system can adaptively scale the acquired power equipment images to meet the image size requirements of our deep learning network. Black borders are used to fill the acquired image information. The ratio of our input image size to the required image size is calculated, and then the original image size is multiplied by the obtained ratio to get the output image size. This output image size is compared with the actual required image size, and any excess is filled with black borders.

[0080] (2) After acquiring the collected image information, the target detection network outputs the number of workers and the type and location information of the equipment in the image information:

[0081] First, a target detection network is used to extract deep features from the standard-sized image information to obtain the categories of people and electrical equipment in the image, and to locate the number and location information of people and the location and category of electrical equipment in the image.

[0082] The object detection network uses the YOLO Nano network model, a lightweight network model that significantly reduces computational cost while maintaining high object recognition accuracy. This network consists of three structures: PEP (Projected Extended Projection), EP (Extended Projection), and FCA (Lightweight Fully Connected Attention Structure), implemented in a cascaded and parallel manner. The PEP module is formed by concatenating two 1×1 convolutional layers, a 3×3 DW convolution, and another 1×1 convolution. The EP module is formed by concatenating two 1×1 convolutional layers with a 3×3 DW convolution in between. The FCA module is formed by concatenating two fully connected layers. These three modules are stacked, with the stacked layers serving as input to the next layer, performing a parallel operation to obtain the final object detection network. The network predicts location information at multiple scales, detecting small, medium, and large-scale targets at 52×52, 26×26, and 13×13 scales. The original object detection network pre-sets a Nums value for each scale. channal = 3 prior boxes of different sizes. Therefore, the total number of predicted channels is: Nums channal =13*13+26*26+52*52=3549, and the total number of predicted boxes is: Nums box =Nums channal *Nums anchor = 10647.

[0083] The predicted bounding box for object detection includes four predicted bounding box offsets: t x ,t y ,tw ,t h With one target confidence level C and num=5 (types of different power equipment) category confidence levels P, the number of parameters to be predicted is:

[0084] Nums box =Nums channal *Nums anchor *[4+1+5]=106470

[0085] Since the number of prediction boxes needed in these prediction boxes is much smaller than the total number of prediction boxes, the Non-Maximum Suppression (NMS) algorithm is used to reduce redundant prediction boxes, and the Intersection over Union (IOU) value is relied upon:

[0086]

[0087] Find the box M with the highest score; remove box M from B and add it to the final predicted box set D; calculate the Intersection over Union (IOU) of all boxes in B with box M; remove boxes from B with an IOU greater than the threshold s; repeat the operation until B is empty to achieve the prediction of model location information.

[0088] Finally, the target detection network uses four offset coordinates for each target to be detected: t x ,t y ,t w ,t h Therefore, it is necessary to use the prior bounding box and the predicted offset coordinates to calculate the predicted coordinates: b x ,b y ,b w ,b h If the offset of the cell from the top left corner of the image is (c x ,c y The formula for calculating the predicted coordinate values ​​is as follows:

[0089] b x =σ(t) x )+c x ,b y =σ(t) y )+c y

[0090]

[0091]

[0092] In the above formula, t x ,t y t represents the coordinate offset of the predicted target bounding box center position relative to the top-left corner of the corresponding network grid; w ,t hb is the ratio of the width and height of the predicted target bounding box to the size of the prior bounding box; x ,b y To predict the coordinates of the center position of the target bounding box; b w ,b h To predict the length and width of the target bounding box; P w ,P h Let x be the length and width of the prior bounding box; σ(x) is the sigmoid activation function.

[0093] (3) Based on the obtained target position information, perform human pose estimation to obtain the position information of each skeletal point of the human body in the image;

[0094] The Human Pose Estimation Network (HRNet) comprises four stages. The first stage consists of four Bottleneck residual units with 64 output channels. The input feature map is processed through 1×1, 3×3, and 1×1 convolutions (including convolutional layers, batch normalization (BN) layers, and ReLU layers) to generate new feature maps. These newly generated feature maps are then fused with the input feature map to form the output feature map. The second, third, and fourth stages have 1, 4, and 3 resolution blocks, respectively, each containing four Basic residual units. The input feature map is processed through two 3×3 convolutions to generate new feature maps, which are then fused with the input feature map to form the output feature map.

[0095] Two stages are connected by a swap unit. A new subnetwork is added by 2x downsampling (3×3 convolutions with a stride of 2), reducing the resolution of the new subnetwork to half that of the previous subnetwork while doubling the number of channels. Cross-resolution feature fusion is achieved through nx downsampling (n can be 2, 4, or 8; 4x downsampling involves concatenating two 3×3 convolutions with a stride of 2, and 8x is similar), convolutional units (using 3×3 convolutions with a stride of 1 for same-resolution mapping), and nx upsampling. By achieving multiple multi-resolution scale fusion through the swap unit, the high-resolution subnetwork can obtain features from different receptive fields from the other subnetworks, thus enabling the prediction of skeletal point information in the fourth stage. Through downsampling by the swap unit, the HRNet network has four parallel subnetworks, each with a different resolution, with subnetwork resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively. Through this multi-resolution fusion and high-resolution feature mapping, the HRNet network predicts more accurate keypoint heatmaps and is more spatially precise.

[0096] (4) After obtaining the location information of the skeleton points, the new partitioning strategy network is used to realize human motion recognition to obtain the skeleton information of the human body. Then, the human motion recognition model is used to extract and learn features from the human skeleton information to realize the category of power operation and detect the safety of power operation.

[0097] The ST-GCN network model, which employs a novel partitioning strategy, takes a skeleton sequence as input. This skeleton sequence is represented by the two-dimensional or three-dimensional coordinates of human skeletal points within an image frame. Feature vectors representing these skeletal points are then obtained, and a convolutional neural network is used to recognize the skeletal action sequence.

[0098] The skeleton sequence is represented by the two-dimensional or three-dimensional coordinate position information of human skeletal points in the image frame. Feature vectors representing human skeletal point information can be obtained, and convolutional neural networks are then used to recognize skeletal action sequences.

[0099] The skeletal spatiotemporal graph has a set of skeletal points V = {v} ti |t=1,...,T,i=1,...N};, which consists of the coordinate information of all skeletal points in the human skeleton series, including two parts. The first part is the skeletal edge formed by two adjacent skeletal points in space, denoted as E. s ={v ti v tj (i,j)∈H}; the second part is that the skeleton edge formed by two identical skeletal points in time consists of two subsets, E. F ={v ti v (t+1)i It can be observed that constructing a skeleton spatiotemporal graph can contain the change information of skeletal points in an action sequence. Furthermore, it can be observed that for a skeletal point i, its change curve is E. F All skeleton edges in the skeletal spatiotemporal graph. A partitioning strategy network model can be constructed by using spatiotemporal graph convolution on the skeletal spatiotemporal graph.

[0100] The partitioning strategy network model contains 9 partitioning strategy units. Each strategy unit uses a 3×9 spatiotemporal convolution kernel and employs feature residual fusion to achieve cross-region feature fusion, thereby increasing the model's learning ability. In each partitioning strategy network unit, units 1-3 contain 64 output dimensions, units 4-6 contain 128 output dimensions, and units 7-9 contain 256 output dimensions. The final output dimensions are then classified using a Softmax classifier to determine the action category within the skeleton sequence.

[0101] Finally, the spatiotemporal graph convolution constructs discrete feature points in space. Traditional two-dimensional convolution can treat the input and output images as two-dimensional matrix grids, which is a non-linear mapping from input features to output features. For a filter of size K×K and an input feature matrix f with c channels... in The output of a 2D convolution operation at position x can be defined:

[0102]

[0103] In the above formula, P is the sampling function P(v) ti ,v tj ) = v tj The function acts on the neighborhood (h, w) of position x; w is the weight function W(v ti ,v tj )=W'(l ti (vtj)) provides a weight vector in c-dimensional space; · represents the inner product. By constructing spatial graph convolution through two-dimensional convolution, we can redefine the graph convolution formula by defining the sampling function and weight function:

[0104]

[0105] Among them, normalization Z ti (v ti )=|{v tk |l ti (v tk )=l ti (v tj The value of )} equals the cardinality of the corresponding subset. With this term, to balance the contributions of different subsets to the output, we finally obtain the spatial graph convolution formula:

[0106]

[0107] This invention not only ensures that electrical work is carried out in an orderly manner according to procedures, but also allows for monitoring of the workers' working status at critical stages to ensure compliance with regulations and identify any potential risks, thus greatly guaranteeing the safety of electrical work. Compared with existing technologies, it achieves the following beneficial effects:

[0108] (1) Through the power operation identification module, the network model is used to identify the image information of the power operation site uploaded by the operator, and the posture information of the operator is judged. The type and location of the power equipment are also located. If there is any abnormal risk, it will be promptly alerted and fed back. By combining the key information of each step of the power operation, it is determined whether each step is carried out, thereby ensuring that the operator strictly follows the operation procedure specifications.

[0109] (2) The lightweight YOLO Nano target detection network was used. Since most power operations are carried out outdoors, this network has a great advantage in terms of computational cost. Therefore, this network model has a very good detection effect on power equipment and workers while reducing the model size.

[0110] (3) Using the Human Pose Estimation Network HRNet, the skeletal information of the worker's body can be located, which makes it easier for the subsequent new partitioning strategy ST-GCN network to better identify the worker's pose information and quickly and effectively determine the abnormal behavior pose of the worker.

[0111] Example 2:

[0112] This invention also proposes a power operation site human interaction behavior recognition system 200 based on image information, such as... Figure 2 As shown, it includes:

[0113] The power equipment positioning and identification module 201 is used to collect image information at the power operation site, use a target detection network to detect the collected image information, and output key information.

[0114] The human pose estimation module 202 is used to perform human pose estimation based on the key information and the human pose estimation network to obtain the position information of the skeletal points of the worker in the image information.

[0115] The power operation identification module 203 is used to identify the human interaction behavior of the operator at the work site based on the location information of the operator's skeleton points, according to the new partitioning strategy network.

[0116] Key information includes: the number of workers, the type of equipment, and its location.

[0117] The process involves using an object detection network to detect the acquired image information, including:

[0118] The target detection network is used to extract deep features from the image information to obtain the categories of workers and power equipment in the image information, as well as the number and location information of workers and the location of power equipment.

[0119] The object detection network includes a lightweight network model.

[0120] The object detection network includes:

[0121] Projection extended projection structure PEP, extended projection structure EP, lightweight fully connected attention structure FCA;

[0122] in:

[0123] PEP is obtained by concatenating two 1×1 convolutional layers, a 3×3 DW convolution, and then another 1×1 convolution.

[0124] EP is obtained by concatenating two 1×1 convolutional layers with a 3×3 DW convolution inserted in between;

[0125] FCA is obtained by connecting two fully connected layers in series;

[0126] By stacking PEP, EP, and FCA, and then using some of the stacked layers as inputs to the next layer in parallel, the target detection network is obtained.

[0127] The predicted bounding boxes of the object detection network include four predicted bounding box offsets and redundant predicted bounding boxes using the Non-Maximum Suppression (NMS) algorithm.

[0128] The human pose estimation network HRNet consists of four stages. The first stage includes four output channels of Bottleneck residual units. The input feature map is processed by 1×1, 3×3 and 1×1 convolutions (including convolutional layers, BN layers and ReLU layers) to generate a new feature map. The newly generated feature map is fused with the input feature map to serve as the output feature map.

[0129] The second, third, and fourth stages have 1, 4, and 3 resolution blocks, respectively. Each resolution block includes 4 Basic residual units. The input feature map is processed by two 3×3 convolutions to generate a new feature map. The newly generated feature map is fused with the input feature map to serve as the output feature map.

[0130] The new partitioning strategy network extracts and learns features from human skeleton information through a human motion recognition model, and then identifies the human interaction behavior of the workers at the work site based on the position information of the workers' skeletal points.

[0131] This invention not only ensures that power operations are carried out in an orderly manner according to the steps, but also allows for monitoring of the working status of personnel at key stages of the power operation to ensure compliance with regulations and identify any risks, thus greatly guaranteeing the safe conduct of power operations.

[0132] Example 3:

[0133] Based on the same inventive concept, this invention also provides a computer device, which includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement corresponding method flows or corresponding functions, thereby implementing the steps of the methods in the above embodiments.

[0134] Example 4:

[0135] Based on the same inventive concept, this invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of the method in the above embodiments.

[0136] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0137] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0141] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for recognizing human interaction behavior in a power operation site based on image information, characterized in that, The method comprises the following steps: For a power operation site, image information is collected, and a target detection network is used to detect the collected image information to output key information; Based on a human pose estimation network, human pose estimation is performed according to the key information to obtain position information of skeleton points of an operator in the image information; Based on a new partition strategy network, a human interaction behavior of an operator at the operation site is identified according to the position information of the skeleton points of the operator.

2. The power work site human interactive behavior recognition method of claim 1, wherein, The key information comprises the number of operators and the category and corresponding position information of equipment.

3. The power work site human interactive behavior recognition method of claim 1, wherein, The detection of the collected image information using the target detection network comprises the following steps: The target detection network is used to perform deep feature extraction on the image information to obtain the category of the operator and the category of the power equipment, as well as the number, position information of the operator and the position of the power equipment.

4. The power work site human interactive behavior recognition method of claim 1, wherein, The target detection network comprises a lightweight network model.

5. The power work site human interactive behavior recognition method of claim 1, wherein, The target detection network comprises: a projection expansion projection structure (PEP), an expansion projection structure (EP) and a lightweight fully connected attention structure (FCA); The PEP is obtained by two 1x1 convolution layers, a 3x3 DW convolution and a 1x1 convolution in series; The EP is obtained by two 1x1 convolution layers with a 3x3 DW convolution inserted in series; The FCA is obtained by two fully connected layers in series; The PEP, EP and FCA are stacked continuously, and several layers in the stack are used as the input of the next layer, and a parallel operation is performed to obtain the target detection network. The prediction box of the target detection network comprises four prediction box offsets, and a non-maximum suppression algorithm (NMS) is used to remove redundant prediction boxes.

6. The power work site human interactive behavior recognition method of claim 1, wherein, The human pose estimation network (HRNet) comprises four stages, the first stage comprises four output channel Bottleneck residual units, wherein a new feature map is generated after the input feature map is subjected to 1x1, 3x3 and 1x1 convolution (including convolution layers, BN layers and ReLU layers), and the new generated feature map is fused with the input feature map as an output feature map; 7. The power work site human interactive behavior recognition method of claim 1, wherein, The second, third and fourth stages have 1, 4 and 3 resolution blocks respectively, each resolution block comprises four Basic residual units, and a new feature map is generated after the input feature map is subjected to two 3x3 convolutions, and the new generated feature map is fused with the input feature map as an output feature map. The new partition strategy network identifies the human interaction behavior of the operator at the operation site according to the position information of the skeleton points of the operator after feature extraction and feature learning of the human skeleton information by a human action recognition model.

8. The power work site human interactive behavior recognition method of claim 1, wherein, The method comprises the following steps:

9. An image information-based power operation site human interaction behavior recognition system, characterized by, The power equipment positioning and identification module is used to collect image information for a power operation site, and a target detection network is used to detect the collected image information to output key information; The human pose estimation module is used to perform human pose estimation based on a human pose estimation network according to the key information to obtain position information of skeleton points of an operator in the image information; ​ The power operation recognition module is configured to recognize the human interaction behavior of the operation personnel in the operation site based on a new partition strategy network and according to the position information of the skeleton points of the operation personnel.

10. The power work site human interactive behavior recognition system of claim 9, wherein, The key information includes the number of operation personnel and the category and corresponding position information of the equipment.

11. The power work site human interactive behavior recognition system of claim 9, wherein, The target detection network is configured to detect the collected image information, including: The target detection network is configured to perform deep feature extraction on the image information to obtain the category of the operation personnel and the category of the power equipment, and the number, position information of the operation personnel and the position of the power equipment.

12. The power work site human interactive behavior recognition system of claim 9, wherein, The target detection network includes a lightweight network model.

13. The power work site human interactive behavior recognition system of claim 9, wherein, The target detection network includes: a projection expansion projection structure (PEP), an expansion projection structure (EP) and a lightweight fully connected attention structure (FCA); The PEP is obtained by two 1x1 convolution layers, a 3x3 DW convolution and a 1x1 convolution in series. The EP is obtained by two 1x1 convolution layers with a 3x3 DW convolution inserted in series. The FCA is obtained by two fully connected layers in series. The PEP, EP and FCA are stacked continuously, and several layers of the stacked PEP, EP and FCA are used as the input of the next layer, and a parallel operation is performed to obtain the target detection network. The prediction box of the target detection network includes four prediction box offsets, and a non-maximum suppression algorithm (NMS) is used to remove redundant prediction boxes.

14. The power work site human interactive behavior recognition system of claim 9, wherein, The human pose estimation network (HRNet) includes four stages, the first stage includes four output channel Bottleneck residual units, wherein the input feature map is generated after 1x1, 3x3 and 1x1 convolution (including convolution layer, BN layer and ReLU layer), and the newly generated feature map is fused with the input feature map as the output feature map.

15. The power work site human interactive behavior recognition system of claim 9, wherein, The second, third and fourth stages have 1, 4 and 3 resolution blocks respectively, each resolution block includes four Basic residual units, and the input feature map is generated after two 3x3 convolutions, and the newly generated feature map is fused with the input feature map as the output feature map. The new partition strategy network extracts and learns features from the human skeleton information through a human action recognition model, and recognizes the human interaction behavior of the operation personnel in the operation site according to the position information of the skeleton points of the operation personnel.

16. The power work site human interactive behavior recognition system of claim 9, wherein, The method includes:

17. A computer device, comprising: one or more processors; the processor is configured to execute one or more programs; when the one or more programs are executed by the one or more processors, the method of any one of claims 1-8 is implemented. The computer program is stored on the computer readable medium, and when the computer program is executed, the method of any one of claims 1-8 is implemented.

18. A computer-readable storage medium, characterized in that, ​

Images