An AI-based method for detecting violations in the maintenance operations of distribution rooms

Abstract

The present invention discloses a method for detecting violations in the maintenance operation of a power plant distribution room based on artificial intelligence. The present invention simultaneously utilizes object detection technology, two-dimensional to three-dimensional coordinate technology, and head pose estimation technology to achieve the detection of the above-mentioned violation actions, and can detect the violation actions existing in the video images of the distribution room captured by a monocular camera, without the need for a depth camera and complex three-dimensional object detection technology; moreover, for the scenario where most of the personnel captured during the maintenance operation of the maintenance personnel in the power plant distribution room taking maintenance tools from the protection cabinet are in the side or back view, a head pose estimation method without frontal face key points is used to achieve the estimation of the head forward tilt pose, improving the accuracy of head pose estimation, and finally combining object detection technology and two-dimensional coordinate to three-dimensional coordinate technology to accurately detect the violation action of the head reaching into the protection cabinet.

Description

Technical Field:

[0001] The present invention relates to the technical field of power plants, and particularly to a method for detecting violations in the maintenance operations of a distribution room. Background Art:

[0002] As an important production link in the power system, the safe production of a power plant has a significant impact on the stable operation of the power system. The maintenance operations of the staff in the power plant distribution room are an important measure to ensure the safe operation of the power plant equipment. When the staff performs maintenance operations in the power plant distribution room, they need to take out the maintenance equipment from the protection cabinet and place it in the specified ground area of the distribution room for maintenance. During this process, it should be ensured that the head is not put into the protection cabinet. However, some staff will inadvertently put their heads into the protection cabinet to facilitate the taking of maintenance tools. When the protection cabinet is energized, this operation will directly endanger the lives of the staff. Therefore, it is particularly important to establish a perfect intelligent detection system for violations in the maintenance operations of the distribution room.

[0003] The traditional detection of the operation behavior in the power plant distribution room is carried out by on-site staff through video monitoring, but the efficiency is low, which brings difficulties in monitoring and identification to the monitoring personnel. At the same time, due to the low accuracy of manual detection, it is very easy to miss detections. With the development of industrial automation, some intelligent detection technologies are relatively mature. In the field of safety management of the distribution room operations, the entry of personnel into the distribution room can be strictly controlled by intelligent identification of personnel identities, or the misoperation behavior of the operator walking into the wrong interval can be detected, but there is no automatic detection and alarm for the violation operations during the maintenance operations in the distribution room, such as the violation behavior of putting the head into the protection cabinet to take maintenance tools. Summary of the Invention:

[0004] The purpose of the present invention is to provide a method for detecting violations in the maintenance operations of a power plant distribution room based on artificial intelligence. This method can obtain the operation execution situation of the maintenance personnel in the distribution room in real time, and at the same time use object detection technology, the technology of converting the two-dimensional coordinates of the image plane to the three-dimensional coordinates of the world, and head pose estimation technology to detect and give early warnings in real time whether the head extends into the protection cabinet, improving the personal safety of the maintenance personnel and better ensuring the safe and stable operation of the power system.

[0005] The present invention is implemented by the following technical solutions: A method for detecting violations in the maintenance operations of a power plant distribution room based on artificial intelligence, which includes the following steps:

[0006] Step S1: Use the cameras installed in the distribution room to obtain the video images of the maintenance personnel performing maintenance operations in the distribution room;

[0007] Step S2: Manually mark the target positions and categories of the opened cabinet doors, personnel heads, and personnel areas in the video images obtained in Step S1 to obtain the object detection training data set;

[0008] Step S3: Train the neural network model Faster RCNN using the target detection training dataset obtained in Step S2 to obtain a target detection model that can detect open cabinet doors, human heads, and human targets.

[0009] Step S4: Use the target detection model obtained in Step S3 to perform target detection on the video images in the monitoring area of the distribution room, and determine whether there is an open cabinet door in the target detection result; if no open cabinet door is detected in the target detection result, repeat Step S4; otherwise, proceed to the next step.

[0010] Step S5: Determine whether there are still human targets in the target detection result of Step S4. If not, repeat Step S4; otherwise, proceed to the next step.

[0011] Step S6: Obtain the position coordinates of the human target in the target detection result of Step S5, calculate the position coordinates of the human feet based on the human position coordinates in the target detection result, then determine the three-dimensional coordinates of the human feet in the world three-dimensional coordinate system, and then determine whether the distance between the feet of the maintenance personnel and the protection cabinet exceeds the set distance threshold based on the three-dimensional coordinates of the human feet; if it exceeds, it indicates that the person cannot put their head into the protection cabinet, and repeat Step S4; otherwise, it means that the maintenance personnel have walked in front of the protection cabinet. At this time, proceed to the next step.

[0012] Step S7: At this time, use the head pose estimation model for predicting the head pitch angle to perform head pose estimation on the human head in the target detection result of Step S5, and determine whether the human head has a forward tilt movement; when it is determined that the human head has a forward tilt movement, it indicates that the maintenance personnel's head is about to extend into the protection cabinet, and an alarm is issued.

[0013] Further, the distance threshold is 0 - 30 cm.

[0014] Further, in Step S7, the head pose estimation model for predicting the head pitch angle is obtained by training the head pose estimation model based on the residual network using the publicly available head pose estimation training dataset.

[0015] Further, the specific process of Step S7 is as follows: When using the head pose estimation model to perform head pose estimation on the human head in the target detection result of Step S5, the Euler angle θ in the direction of head pose pitch and roll is obtained. If the Euler angle θ is greater than the set angle threshold, it is determined that the head has a forward tilt movement. At this time, it indicates that the person has extended their head into the protection cabinet, and an alarm is issued; otherwise, return to Step S4 to continue the detection.

[0016] Further, the Euler angle θ is -30° to -90°.

[0017] Furthermore, the specific process of step S6 is as follows:

[0018] S601: Establish a three-dimensional coordinate system in the substation scene: The origin is located at the center point of the bottom side length of the overall protection cabinet. The X-axis extends to the left along the bottom side of the cabinet, the Y-axis is obtained by rotating the X-axis counterclockwise by 90 degrees in the ground plane, and the Z-axis extends upward along the height of the cabinet, perpendicular to the ground;

[0019] S602: The conversion formula between a certain pixel point in the personnel target area and the three-dimensional coordinates in the three-dimensional coordinate system in the substation scene is as follows:

[0020]

[0021] where f x , f y denote the focal lengths of the camera on the x-axis and y-axis; u 0 , v 0 is the center of the camera aperture; (u, v) represents the pixel coordinates of a certain pixel point in the personnel target area, Z c is the Z value in the camera coordinate system corresponding to this point, (X w , Y w , Z w ) represents the coordinate values in the three-dimensional coordinate system in the substation scene corresponding to this point, R is the external reference rotation matrix of the camera, and T is the translation vector.

[0022] S603: According to the vertex coordinates (up1, vp1, up2, vp2) of the upper left corner and the lower right corner of the personnel area detection box in the target detection result of step S5, calculate the pixel coordinates (up0, vp0) of the approximate point of the personnel's feet. The calculation formula is as follows:

[0023] up0 = (up1 + up2) / 2 (2)

[0024] vp0 = vp2 (3)

[0025] where up1 and vp1 are the horizontal and vertical coordinates of the upper left corner vertex of the personnel area detection box; up2 and vp2 are the horizontal and vertical coordinates of the lower right corner vertex of the personnel area detection box; since the personnel's feet are on the ground, the Z w in its three-dimensional coordinate system is 0. Substitute the pixel coordinates (up0, vp0) of the approximate point of the personnel's feet and Z w = 0 into formula (1) to obtain the corresponding coordinate values (X d , Y d , 0) in the three-dimensional coordinate system;

[0026] S604: If Y in the three-dimensional coordinates (X d , Y d , 0)d If the difference from 0 is greater than the set threshold, it indicates that there is still a certain distance between the maintenance personnel's footsteps and the protection cabinet. At this time, it is impossible to put the head into the protection cabinet, so step S4 is repeated; otherwise, it indicates that the maintenance personnel have walked in front of the protection cabinet, and the next step is executed.

[0027] Advantages of the present invention: The present invention detects and warns the behavior of the staff putting their heads into the protection cabinet through artificial intelligence methods to prevent harm to the staff. The present invention also uses object detection technology, two-dimensional to three-dimensional coordinate technology, and head pose estimation technology to detect the above-mentioned illegal actions, and can detect the illegal actions existing in the video images of the power distribution room captured by a monocular camera, without the need for a depth camera and complex three-dimensional object detection technology; moreover, in the case where most of the personnel captured in the scenario of maintenance personnel taking maintenance tools from the protection cabinet in the power plant power distribution room are in the side or back view, the head pose estimation method without frontal face key points of the human face is used to estimate the forward tilt pose of the head, improving the accuracy of head pose estimation. At the same time, combined with object detection technology and two-dimensional coordinate to three-dimensional coordinate technology, the accurate detection of the illegal action of putting the head into the protection cabinet is finally realized. Description of the drawings:

[0028] Figure 1 It is a schematic diagram of the detection process of the present invention.

[0029] Figure 2 It is a schematic diagram of the construction of a three-dimensional coordinate system.

[0030] Figure 3 It is a schematic diagram of the checkerboard used for camera external parameter calibration.

[0031] Figure 4 It is a head pose estimation residual network model for predicting the Euler angle θ in the pitch direction of the head pose. Detailed implementation manners:

[0032] In the description of the present invention, it should be noted that when terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, when terms such as "first", "second", "third" appear, they are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0033] As Figures 1 to 4 shown, a method for detecting illegal maintenance operations in a power plant power distribution room based on artificial intelligence includes the following steps:

[0034] Step S1: Obtain the video images of maintenance personnel performing maintenance operations in the switchgear room by using the cameras installed in the switchgear room;

[0035] Step S2: Manually annotate the target positions and categories of the opened cabinet doors, personnel heads, and personnel areas in the obtained video images to obtain a target detection training data set;

[0036] Step S3: Use the above target detection training data set to train the neural network model Faster RCNN to obtain a target detection model that can detect opened cabinet doors, personnel heads, and personnel targets;

[0037] Step S4: Use the target detection model obtained in Step S3 to perform target detection on the video images in the monitoring area of the switchgear room, and determine whether there are any opened cabinet doors in the target detection results; if no opened cabinet doors are detected in the target detection results, repeat Step S4; otherwise, start to execute the next step;

[0038] Step S5: Determine whether there are still personnel targets in the target detection results of Step S4. If not, repeat Step S4; otherwise, execute the next step;

[0039] Step S6: Obtain the position coordinates of the personnel targets in the target detection results of Step S5, calculate the position coordinates of the feet of the personnel based on the personnel position coordinates in the target detection results, determine the three-dimensional coordinates of the personnel in the world three-dimensional coordinate system based on the three-dimensional coordinates of the personnel feet position, and then determine whether the distance between the feet of the maintenance personnel and the protection cabinet exceeds the set distance threshold according to the three-dimensional coordinates of the personnel feet position; if it exceeds, it indicates that the personnel cannot put their heads into the protection cabinet, and repeat Step S4; otherwise, it means that the maintenance personnel have walked in front of the protection cabinet. At this time, execute the next step;

[0040] Among them, the specific process of Step S6 is as follows:

[0041] S601: As Figure 2 shown, establish a three-dimensional coordinate system in the switchgear room scenario: The origin is located at the center point of the bottom side length of the overall protection cabinet. The X-axis extends to the left along the bottom side of the cabinet, the Y-axis is obtained by rotating the X-axis counterclockwise by 90 degrees in the ground plane, and the Z-axis extends upward along the height of the cabinet, perpendicular to the ground;

[0042] S602: The conversion formula between a certain pixel point in the personnel target area and the three-dimensional coordinates in the three-dimensional coordinate system in the switchgear room scenario is as follows:

[0043]

[0044] Among them, f x ,f y refer to the focal lengths of the camera on the x-axis and y-axis, u 0 ,v0 is the aperture center of the camera; (u, v) represents the pixel coordinates of a point in the image, and Z c is the Z value in the camera coordinate system corresponding to this point, (X w , Y w , Z w ) represents the coordinate values of the three-dimensional coordinate system corresponding to this point, R is the external camera parameter rotation matrix of size 3*3, and T is a three-dimensional translation vector.

[0045] Among them, the camera internal parameter matrix can be obtained through the parameters of the camera. For example, the parameters of the Hikvision DS-2CD2355F(D)-IS network camera are as follows: lens focal length f = 4mm, maximum image resolution: 2560×1920, sensor size: 4.8×3.6mm. Then, there are: u 0 = 2560 / 2 = 1280, v 0 = 1920 / 2 = 960, dx = 4.8 / 2560, dy = 3.6 / 1920, f x = f / dx = 2133.3mm, f y = f / dy = 2133.3mm. It can also be obtained through the Zhang's camera calibration method.

[0046] The process of solving the external camera parameters includes the following steps:

[0047] Such as Figure 3The checkerboard shown is a 9*6 checkerboard in the A4 standard, with the corresponding number of corner points being 5*8, and the side length of each grid being 30mm. Placing the checkerboard in the image captured by the camera can include the position of the complete checkerboard. For example, in this embodiment, it is placed on the ground in front of the protection cabinet, near the origin of the three-dimensional space coordinate system, and the actual space coordinates of each corner point on the checkerboard are easy to calculate. By collecting the calibration image of the checkerboard with the camera and using the corresponding functions in openCV to solve the external parameters of the camera: for the collected calibration image image, first use gray=(image, cv2.COLOR_RGB2GRAY) to convert it into a grayscale image; then use cv2.findChessboardCorners(gray, (5, 8), ) to return the pixel coordinate array corners of the detected 40 corner points; further, cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria) can be called to obtain a more accurate corner point position, which is stored in the array exact_corners, where criteria is the termination condition of the corner point refinement iteration process, and here it is set as criteria=(cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001); after obtaining the accurate pixel coordinates of the 40 corner points, and based on their actual space coordinates (stored in the world_point array), when the internal parameters of the camera (stored in the matrix mtx) are known, by calling cv2.solvePnPRansac(world_point, exact_corners, mtx, dist), this function returns the rotation matrix rvec and the translation matrix tvec, where rvec is a 3×1 vector. To restore it to a 3×3 rotation matrix, the Rodrigues transform Rodrigues is required, and cv2.Rodrigues(rvec) is called to obtain the 3×3 rotation matrix, and thus the external parameter rotation matrix R and translation vector T of the camera can be obtained.

[0048] S603: According to the vertex coordinates (up1, vp1, up2, vp2) of the upper left corner and the lower right corner of the person area detection box in the target detection result of step S5, calculate the pixel coordinates (upO, vpO) of the approximate point of the person's foot, and the calculation formula is as follows:

[0049] up0 = (up1 + up2) / 2 (2)

[0050] vp0 = vp2 (3)

[0051] Among them, up1 and vp1 are the horizontal and vertical coordinates of the upper left vertex of the personnel area detection frame; up2 and vp2 are the horizontal and vertical coordinates of the lower right vertex of the personnel area detection frame; since the feet of the personnel are on the ground, the Z in its three-dimensional coordinate system w = 0. Substitute the pixel coordinates (up0, vp0) and Z w = 0 of the approximate point of the personnel's feet into formula (1) to obtain the corresponding coordinate values (X d 、Y d 、0) in the three-dimensional coordinate system. The specific steps are as follows: Formula (1) can be further transformed into:

[0052]

[0053] Among them, the M matrix is the product of the camera internal parameter matrix and the external parameter matrix, and the internal parameter matrix and the external parameter matrix have been obtained in the above steps, so they are known terms. From this formula, three equations can be obtained:

[0054] Z c up0 = m 11 X w + m 12 Y w + m 14

[0055] Z c vp0 = m 21 X w + m 22 Y w + m 24

[0056] Z c = m 31 X w + m 32 Y w + m 34

[0057] Substitute the values of (upO, vpO) and Z w = 0 into the above equations to obtain the coordinate values of the pixel point (upO, vpO) in the corresponding three-dimensional coordinate system, which is represented by (X d 、Y d 、0) here.

[0058] S604: If the difference between Y d 、Y d 、0) in the three-dimensional coordinates (X d and 0 is greater than the set distance threshold, it means that there is still a certain distance between the maintenance personnel's feet and the protection cabinet. At this time, it is impossible to put the head into the protection cabinet, so repeat step S4; otherwise, it means that the maintenance personnel have walked in front of the protection cabinet, and then execute the next step.

[0059] Step S7: When using the head pose estimation model for predicting the head pitch angle to perform head pose estimation on the human head in the object detection result of step S6, if the Euler angle θ in the head pose pitch direction is obtained and θ is greater than the set angle threshold, it is determined that the head has a forward tilting action. At this time, it indicates that the person has put their head into the protection cabinet, and an alarm is issued; otherwise, return to step S4 to continue the detection.

[0060] In step S7, the head pose estimation model for predicting the head pitch angle is obtained by training the head pose estimation model based on the residual network using the publicly available head pose estimation training dataset;

[0061] Specifically, in this embodiment, the AFLW2000 public dataset is selected as the training sample. AFLW2000 contains rich side head images and is suitable for the situation where only the side face can be captured instead of the front face in this scenario.

[0062] Such as Figure 4 , the head pose estimation model based on the residual network is specifically: using ResNet-50 as the backbone network, followed by a fully connected layer and softmax to obtain the output category. For the Euler angle in the pitch direction, the angle range is specified as [-90°, 90°], with each 3 degrees as a category, and a total of 60 categories are included; according to the output category id obtained by softmax, the predicted Euler angle in the pitch direction is calculated using the formula id * 3 - 90. Since the present invention only considers the angle change in the pitch direction, only the head pose estimation in the pitch direction is predicted. The loss function loss consists of two parts: the cross-entropy loss for classification and the mean squared error (MSE) regression loss.

[0063] Use the AFLW2000 dataset to train the above head pose estimation model based on the residual network to obtain the optimal head pose estimation model.

[0064] The present invention detects and warns the behavior of the staff putting their heads into the protection cabinet through artificial intelligence methods to prevent harm to the staff. The present invention simultaneously uses object detection technology, two-dimensional to three-dimensional coordinate technology, and head pose estimation technology to achieve the detection of the above-mentioned illegal actions, and can detect the illegal actions existing in the video images of the power distribution room captured by a monocular camera, without the need for a depth camera and complex three-dimensional object detection technology; moreover, for the scenario where most of the personnel captured during the maintenance operation of taking maintenance tools from the protection cabinet by the maintenance personnel in the power plant power distribution room are in the side or back, a head pose estimation method without frontal face key points of the human face is used to estimate the forward tilting pose of the head, improving the accuracy of head pose estimation. At the same time, combined with object detection technology and two-dimensional coordinate to three-dimensional coordinate technology, the accurate detection of the illegal action of putting the head into the protection cabinet is finally achieved.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An illegal detection method for maintenance operations in a power plant distribution room based on artificial intelligence, characterized in that, it includes the following steps: Step S1: Use the cameras installed in the distribution room to obtain video images of maintenance personnel during maintenance operations in the distribution room; Step S2: Manually annotate the target positions and categories of the opened cabinet doors, personnel heads, and personnel areas in the video images obtained in Step S1 to obtain a target detection training data set; Step S3: Use the target detection training data set obtained in Step S2 to train the neural network model Faster RCNN to obtain a target detection model that can detect opened cabinet doors, personnel heads, and personnel targets; Step S4: Use the target detection model obtained in Step S3 to perform target detection on the video images in the monitoring area of the distribution room, and determine whether there is an opened cabinet door in the target detection result; if no opened cabinet door is detected in the target detection result, repeat Step S4; otherwise, perform the next step; Step S5: Determine whether there are still personnel targets in the target detection result of Step S4. If not, repeat Step S4; otherwise, perform the next step; Step S6: Obtain the position coordinates of the personnel targets in the target detection result of Step S5, calculate the position coordinates of the feet of the personnel according to the personnel position coordinates in the target detection result, determine the three-dimensional coordinates of the personnel in the world three-dimensional coordinate system according to the three-dimensional coordinates of the personnel's feet position, and then judge whether the distance between the feet of the maintenance personnel and the protection cabinet exceeds the set distance threshold; if it exceeds, it means that the personnel cannot put their heads into the protection cabinet, and repeat Step S4; otherwise, it means that the maintenance personnel have walked in front of the protection cabinet. At this time, perform the next step; Step S7: At this time, use the head pose estimation model for predicting the head pitch angle to perform head pose estimation on the personnel heads in the target detection result of Step S5, and judge whether the personnel heads have a forward tilting action; When it is judged that the personnel heads have a forward tilting action, it means that the heads of the maintenance personnel are about to extend into the protection cabinet, and an alarm is issued.

2. An illegal detection method for maintenance operations in a power plant distribution room based on artificial intelligence according to claim 1, characterized in that, in Step S7, the head pose estimation model for predicting the head pitch angle is obtained by training the head pose estimation model based on the residual network using the publicly available head pose estimation training data set.

3. An illegal detection method for maintenance operations in a power plant distribution room based on artificial intelligence according to claim 1, characterized in that, The specific process of Step S7 is as follows: When using the head pose estimation model to perform head pose estimation on the personnel heads in the target detection result of Step S5, obtain the Euler angle θ in the head pose pitch direction. If the Euler angle θ is greater than the set angle threshold, it is judged that there is a forward tilting action of the head. At this time, it means that the personnel put their heads into the protection cabinet and an alarm is issued; otherwise, return to Step S4 to continue the detection.

4. An illegal detection method for maintenance operations in a power plant distribution room based on artificial intelligence according to any one of claims 1 to 3, characterized in that, The specific process of Step S6 is as follows: S601: Establish a three-dimensional coordinate system in the distribution room scenario: The origin is located at the center point of the bottom side length of the overall protection cabinet. The X-axis extends to the left along the bottom side of the cabinet, the Y-axis is obtained by rotating the X-axis counterclockwise by 90 degrees in the ground plane, and the Z-axis extends upward along the height of the cabinet, perpendicular to the ground; S602: The conversion formula between a pixel point in the personnel target area and the three-dimensional coordinates in the three-dimensional coordinate system in the distribution room scenario is as follows: (1) Among them, , refers to the focal lengths of the camera on the x-axis and y-axis; , is the aperture center of the camera; ([[]] , ) represents the pixel coordinates of a certain pixel point in the personnel target area, is the Z value in the camera coordinate system corresponding to this point, ([[]] , , ) represents the coordinate values in the three-dimensional coordinate system under the substation scene corresponding to this point, R is the external camera rotation matrix, and T is the translation vector; S603: Based on the vertex coordinates (up1, vp1, up2, vp2) of the upper left corner and the lower right corner of the person area detection box in the target detection result of step S5, calculate the pixel coordinates (up0, vp0) of the approximate point of the person's feet. The calculation formula is as follows: / 2 (2) （3） Among them, up1 and vp1 are the horizontal and vertical coordinates of the upper left vertex of the person area detection box; up2 and vp2 are the horizontal and vertical coordinates of the lower right vertex of the person area detection box; since the person's feet are on the ground, so in its three-dimensional coordinate system = 0. Substitute the pixel coordinates (up0, vp0) of the approximate point of the person's feet and = 0 into formula (1) to obtain the coordinate values of the three-dimensional coordinate system corresponding to the pixel coordinates (up0, vp0), represented by ( , , 0); S604: If the difference between and in the three-dimensional coordinates ( , , 0) and 0 is greater than the set threshold, it indicates that there is still a certain distance between the maintenance personnel's feet and the protection cabinet. At this time, it is impossible to put the head into the protection cabinet, so repeat step S4; otherwise, it indicates that the maintenance personnel have walked in front of the protection cabinet, and the next step is executed at this time.

Images