Method for understanding human interaction relationship in power production operation scene

Abstract

The application discloses a method for understanding the interactive relationship of characters in a power production operation scene, comprising the following steps: extracting basic features from a power operation image, and generating a heat map according to the basic features; predicting a human body, an object and an interaction point according to the heat map and a convolution module, and then obtaining the position where the interaction occurs; combining the basic features with the position to obtain a corresponding frame; taking the interaction point as an anchor point, predicting the displacement between the interaction point and the corresponding human body point and object point, matching each interaction point with the human body point and the object point according to the displacement, and generating a triple representing the interactive relationship. The method can improve the accuracy and response speed of the prediction result, and further reduces the probability of power production safety accidents caused by personnel violation behaviors.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image recognition, and relates to a method for understanding human interaction relationship in a power production operation scene. BACKGROUND

[0002] With the increasing expansion of the power system, accidents occur frequently in system operation. Among them, due to the non-standard wearing of protective tools, dynamic violation of personnel behavior, power production safety violation cases occur frequently, and part of the power production safety risk shows strong scene dependence. The operation scene is usually composed of multiple elements such as personnel, equipment and protective tools and their production interaction relationship. And for the production interaction relationship of the operation scene, the human interaction relationship is mainly used, therefore, how to accurately identify the interaction relationship between people and various tools in the power scene has been widely concerned by various power units.

[0003] In recent years, with the continuous development of the field of computer vision, human-object interaction detection (HOI) has gradually become a new task in the field of computer vision after target detection, target tracking and image segmentation. The current human-object interaction detection is mainly divided into two different types of methods, which are two-stage method and one-stage method, among which the two-stage method (i.e. target detection first and then interaction prediction) can accurately predict the interaction relationship of the personnel in the operation scene, but the response speed is relatively slow. The one-stage detection method adopts a parallel detection method, which can directly detect the HOI triplets from the image, greatly improving the response speed and greatly saving the computing resources, but the accuracy is less than that of the two-stage method. SUMMARY

[0004] The purpose of the present application is to provide a method for understanding the human interaction relationship in the power production operation scene, which solves the problem that the response speed and accuracy cannot be guaranteed at the same time in the prior art.

[0005] The technical solution adopted by the present application is a method for understanding the human interaction relationship in the power production operation scene, comprising the following steps:

[0006] Step 1, extracting basic features from the power operation image, and generating a heat map according to the basic features;

[0007] Step 2, predicting the human body, object and interaction point according to the heat map and convolution module, and then obtaining the position of the interaction occurrence, combining the basic features with the position to obtain the corresponding frame; taking the interaction point as an anchor point, predicting the displacement between the interaction point and the corresponding human body point and object point, matching each interaction point with the human body point and the object point according to the displacement, and generating a triple representing the interaction relationship.

[0008] The present application also has the following characteristics:

[0009] Step 1 specifically involves: using the CenterNet detection network to extract basic features from power operation images. The CenterNet detection network Backbone uses the Hourglass network for key point detection and generates heatmaps based on the basic features.

[0010] The Decoder part of the CenterNet detection network is obtained by replacing UpSample with the DCNv2 network.

[0011] In step 2, the size and offset in the basic features are combined with the location where the interaction occurs to obtain the human body frame, object frame, and interaction frame.

[0012] In step 2, each interaction point is matched with human body points and object points based on displacement. The specific process is as follows:

[0013] The detected human and object points are sorted according to the following formula to obtain the matched human and object points:

[0014] (6);

[0015] In the above formula, As an interaction point, For the human body The confidence score, For displacement.

[0016] The loss function of the CenterNet detection network is:

[0017] (5);

[0018] In the formula, The losses are for the center points of the human body, objects, and interaction points, respectively. This is the loss due to the displacement from the interaction point to the human body point and the object point. These are the loss values ​​for size and offset;

[0019] (1);

[0020] In the formula, N is the number of interaction points. For the predicted heatmap The score of category k at position (x,y); the loss for object and interaction point center is the same as above;

[0021] (2);

[0022] In the formula, For human frame loss, The bounding box loss is denoted by M, where M is the number of human body points and D is the number of object points.

[0023] (3);

[0024] (4);

[0025] In the formula, To predict the displacement, The set of interaction points in the training set is N, where N is the number of interaction points in the set; the loss for the displacement from the interaction point to the object point is the same as above.

[0026] The beneficial effects of this invention are as follows: The method for understanding human-human interaction relationships in power production operation scenarios replaces the 3x3 convolution before deconvolution in the Decoder part of the CenterNet network with the Hourglass network as its backbone with a DCNv2 network. This performs non-linear deformation on the input image, thereby better adapting to the shape of the object and improving the accuracy of the prediction results. Point detection and point matching are performed as parallel tasks, and in the second stage, two proposals are merged to form meaningful HOI triples, which improves the response speed compared to the traditional two-stage method. Furthermore, it reduces the probability of power production safety accidents caused by personnel violations. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method for understanding human interaction relationships in a power production operation scenario according to the present invention;

[0028] Figure 2 This is a structural diagram of the CenterNet detection network in the method for understanding human interaction relationships in power production operation scenarios of the present invention;

[0029] Figure 3 This is a comparison diagram of the sampling positions of DCNv2 and DCNv1 in the method for understanding the interaction relationship between characters in the power production operation scenario of this invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] Example 1

[0032] Methods for understanding human interaction relationships in power production operation scenarios, such as Figure 1 As shown, it includes the following steps:

[0033] Step 1: Extract basic features from power operation images and generate heatmaps based on these features;

[0034] Step 2: Based on the heatmap and the prediction of the convolution module, the human body, object, and interaction points are obtained, and the location where the interaction occurs is obtained. The basic features are combined with the location to obtain the corresponding bounding box. At the same time, the interaction point is used as an anchor point to predict the displacement between the interaction point and the corresponding human body point and object point. Based on the displacement, each interaction point is matched with the human body point and object point to generate a triple representing the interaction relationship.

[0035] Example 2

[0036] The method for understanding human interaction relationships in power production operation scenarios includes the following steps:

[0037] Step 1: Extract basic features from power operation images and generate a heatmap based on these features; Step 1 specifically involves using the CenterNet detection network to extract basic features from the power operation images, such as... Figure 2 As shown, the CenterNet detection network Backbone employs the Hourglass network for keypoint detection and utilizes it to generate heatmaps based on fundamental features. The Decoder part of the CenterNet detection network is obtained by replacing UpSample with a DCNv2 network. DCNv2 (Deformable Convolutional Networks V2) is an operation used in convolutional neural networks that can non-linearly deform the input image, thereby better adapting to the shape of objects. DCNv2 can be used to enhance the receptive field of the model, thus better capturing the interaction information between people and objects. Figure 3 As shown, it can be seen that DCN v2 is more focused on the complete and effective region of the object. Compared with DCN v1, which only used three deformable convolutions in conv 5, DCN v2 replaced conv3 to conv5 with deformable convolutions, thus improving the algorithm's ability to model geometric deformations.

[0038] Step 2: Based on the heatmap and the prediction of the convolution module, the human body, object, and interaction center point are obtained, and the location where the interaction occurs is obtained. The basic features are combined with the location to obtain the corresponding bounding box. At the same time, the interaction point is used as an anchor point to predict the displacement between the interaction point and the corresponding human body point and object point. Based on the displacement, each interaction point is matched with the human body point and object point to generate a triple representing the interaction relationship.

[0039] Example 3

[0040] The method for understanding human interaction relationships in power production operation scenarios includes the following steps:

[0041] Step 1: Use the CenterNet detection network to extract basic features from the power operation images. The basic features include size, offset, etc. The backbone of the CenterNet detection network adopts the Hourglass key point detection network. The Decoder part of the CenterNet detection network is obtained by replacing UpSample with DCNv2 network. Use the Hourglass key point detection network to generate heatmaps based on the basic features.

[0042] In this embodiment, the Hourglass network is used to generate corresponding feature maps. For each point on the feature map, the probability density of a category and the regression of their spatial location are calculated, thereby obtaining a heatmap of human body, object and action interaction points.

[0043] Step 2: Based on the heatmap and convolutional module prediction, the human body, objects, and interaction points are obtained, and the location of the interaction is obtained. The size and offset are combined with the location of the interaction to obtain the human body bounding box, object bounding box, and interaction box. At the same time, the interaction point is used as an anchor point to predict the displacement between the interaction point and the corresponding human body point and object point. Based on the displacement, each interaction point is matched with the best human body point and object point to generate a triple representing the interaction relationship. That is, the interaction relationship between people and various tools in the work scene is obtained, realizing intelligent understanding of human interaction in the power operation scene.

[0044] In this embodiment, the center point of the human body frame is used. This represents the human body frame, along with its corresponding size (width and height). Local point offset The same applies to object frames. At the same time, the character's interaction points... Defined as the midpoint of a pair of person points and object points.

[0045] This embodiment uses a keypoint estimation method to map points onto a heatmap with a Gaussian kernel, thus transforming point detection into a heatmap estimation task. Three real low-resolution points... , , It is mapped onto three Gaussian heatmaps (the point heatmap has low resolution, so it also calculates the low-resolution center point). ), ), and apply element-wise focal loss to reduce the loss for point locations. For example, given an estimated interaction point heatmap and the corresponding heatmap The loss function for the center point of the human body is:

[0046] (1);

[0047] In the formula, N is the number of interaction points. For the predicted heatmap The score of category k at position (x, y); the loss for the center point of objects and interaction points is the same as above; in addition to the center point, the size of the bounding box and the local offset of the center point are also needed to form the person / object bounding box. The offset loss function is:

[0048] (2);

[0049] In the formula, For human frame loss, The bounding box loss is denoted by M, where M is the number of human body points and D is the number of object points.

[0050] Regarding displacement loss, let's take the displacement loss from the interaction point to the person's point as an example:

[0051] (3);

[0052] (4);

[0053] In the formula, To predict the displacement, The set of interaction points in the training set is N, where N is the number of interaction points in the set; the loss for the displacement from the interaction point to the object point is the same as above.

[0054] The loss function of the CenterNet detection network is then:

[0055] (5);

[0056] In the formula, The losses are for the center points of the human body, objects, and interaction points, respectively. This is the loss due to the displacement from the interaction point to the human body point and the object point. This represents the loss value due to size and offset.

[0057] Sort the human body points and object points according to the following formula, and select the best human body point;

[0058] (6);

[0059] In the above formula, As an interaction point, For the human body The confidence score, For displacement.

[0060] Through the above methods, the present invention's method for understanding human-human interaction relationships in power production operation scenarios replaces the 3x3 convolution before deconvolution in the Decoder part of the CenterNet network with a DCNv2 network, performing non-linear deformation on the input image to better adapt to the shape of the object and improve the accuracy of the prediction results; it treats point detection and point matching as parallel tasks, merging two proposals in the second stage to form meaningful HOI triples, which improves the response speed compared to the traditional two-stage method; and further reduces the probability of power production safety accidents caused by personnel violations.

Claims

1. A method for understanding human interaction relationships in a power production work scenario, characterized by, The method comprises the following steps: Step 1, extracting basic features from power operation images, and generating a heat map according to the basic features; Step 2, predicting human bodies, objects and interaction points according to the heat map and a convolution module, and then obtaining the positions of interactions, combining the basic features with the positions to obtain corresponding frames, taking the interaction points as anchor points, predicting the displacement between the interaction points and corresponding human body points and object points, matching each interaction point with the human body points and object points according to the displacement, and generating a triple representing an interaction relationship; Step 1 specifically comprises: extracting basic features from power operation images by using a CenterNet detection network, wherein the Backbone of the CenterNet detection network adopts a key point detection Hourglass network, and the heat map is generated according to the basic features by using the key point detection Hourglass network; The Decoder part of the CenterNet detection network is obtained by replacing UpSample with a DCNv2 network; In step 2, the size and offset in the basic features are combined with the positions of interactions to obtain human body frames, object frames and interaction frames; In step 2, each interaction point is matched with human body points and object points according to the displacement, and the specific process is as follows: The human body points and object points are sorted according to the following formula to obtain matched human body points and object points: （6）； In the above formula, is an interaction point, is a human point a confidence score, is a displacement; The loss function of the CenterNet detection network is as follows: （5）； In the formula, respectively, the loss of the human body, the object, and the interaction point center point, is the loss of the displacement of the interaction point to the human body point and the object point, is the loss value of the size and the offset. （1）； where N is the number of interaction points, the fraction of class k in the predicted heat map at position (x, y), the fraction of class k in the predicted heat map at position (x, y), The loss of the object and interaction point center points is the same as above; （2）； In the formula, is a human body bounding box loss, is an object bounding box loss, M is the number of human body points, and D is the number of object points; （3）； （4）； In the formula, is a predicted displacement, is a set of interaction points in the training set, and N is the number of interaction points in the set of interaction points. The loss of the displacement from the interaction point to the object point is the same as above.

Images