Real-time personnel positioning method based on 3D model of substation

Abstract

This invention discloses a real-time personnel positioning method based on a 3D model of a substation, comprising the following steps: S1, constructing a 3D point cloud map of the substation; S2, extracting the 2D pixel coordinates of each key point of the pedestrian from the acquired pedestrian video images, and outputting the 2D pixel coordinates and confidence scores of the main key points; S3, calculating the 2D projection coordinates of the pedestrian's center point on the ground based on the 2D pixel coordinates and confidence scores of the pedestrian's main key points; S4, obtaining the 3D coordinates based on the camera's intrinsic and extrinsic parameters, the 3D point cloud map, and the 2D projection coordinates of the pedestrian's center point on the ground. This invention has the beneficial effect of enabling real-time, seamless positioning of personnel in substations.

Description

Technical Field

[0001] This invention relates to the field of real-time personnel positioning technology. More specifically, this invention relates to a method for real-time personnel positioning based on a three-dimensional model of a substation. Background Technology

[0002] A substation is a location in a power system that transforms voltage and current, receiving and distributing electrical energy. Substations within power plants are step-up substations, whose function is to boost the voltage of electricity generated by generators before feeding it into the high-voltage power grid. With the rapid development of internet and IoT technologies, substations are also facing a transformation and upgrade from digitalization to intelligentization. How to integrate substation production safety with internet technology has become a pressing issue for the intelligentization of substations. While improving production efficiency, intelligent substations also prioritize the personal safety of workers. Controlling unsafe human behavior is the foundation of all safety management work; therefore, personnel location management is paramount in the construction of intelligent substations.

[0003] Currently, personnel positioning in substations mainly relies on positioning modules carried by personnel, using feedback information from these modules for real-time location tracking. However, this requires additional equipment, which can disrupt normal operations. Therefore, achieving seamless personnel positioning in substations while improving accuracy is a pressing issue that needs to be addressed. Summary of the Invention

[0004] One object of the present invention is to solve at least the above-mentioned problems and to provide at least the advantages that will be described later.

[0005] Another objective of this invention is to provide a real-time personnel positioning method based on a three-dimensional model of a substation, which enables real-time, seamless positioning of personnel in the substation while improving positioning accuracy.

[0006] To achieve these objectives and other advantages according to the present invention, a method for real-time personnel positioning based on a three-dimensional model of a substation is provided, comprising the following steps:

[0007] S1. Construct a 3D point cloud map of the substation;

[0008] S2. Extract the two-dimensional pixel coordinates of each key point of the pedestrian from the acquired pedestrian video image, and output the two-dimensional pixel coordinates and confidence of the main key point;

[0009] S3. Based on the two-dimensional pixel coordinates and confidence level of the pedestrian's main joints, calculate the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0010] S4. Based on the camera's intrinsic and extrinsic parameters, the 3D point cloud map, and the 2D projection coordinates of the pedestrian's center point on the ground, obtain the 3D coordinates.

[0011] Preferably, step S2 includes the following steps:

[0012] S2a: Perform pedestrian detection on the pedestrian video image and output a two-dimensional detection box and a cropped image within the two-dimensional detection box.

[0013] S2b: Obtain pedestrian cutout based on cropped image. Extract 2D pixel coordinates of each joint of pedestrian based on pedestrian cutout and 2D detection box, and output the 2D pixel coordinates and confidence of main joint. The main joint of pedestrian includes multiple feature point groups, arranged in order of priority. Multiple feature point groups include left and right heels, left and right ankles, left and right knees, left and right hips, left and right shoulders, neck, and nose.

[0014] Preferably, step S3 includes the following steps:

[0015] S3a1. Determine the human body proportions of pedestrians;

[0016] S3a2. Determine the two-dimensional line segment, specifically: according to priority, traverse multiple feature point groups with priority no greater than that of the left and right ankles in turn, determine whether there is a feature point group whose confidence of the two-dimensional pixel coordinates is greater than the first threshold. If so, define the feature point group as the first key group, connect the two-dimensional pixel coordinates of the first key group to obtain the two-dimensional line segment and proceed to step S3a3. If not, proceed to step S3a4.

[0017] S3a3. According to priority, starting from the left and right hips, traverse multiple feature point groups in sequence to determine whether there is a feature point group whose confidence of the two-dimensional pixel coordinates is greater than the first threshold. If so, define the feature point group as the second key group. If not, judge it as invalid extraction.

[0018] Based on the second key group, the coordinates of the pedestrian's center point are obtained. These coordinates are then projected vertically downwards onto a two-dimensional line segment to obtain the coordinates of the projection point of the pedestrian's center point on the two-dimensional line segment, where:

[0019] When the first key group is the left and right heels, the coordinates of the projection point are the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0020] When the first key group is the left and right ankles, determine whether there is a joint point in the two-dimensional pixel coordinates of the left and right knees with a confidence level greater than the first threshold. If so, calculate the joint distance between the knee and the ankle. Based on this distance and the human body proportion of the pedestrian, calculate the height H1 of the joint point corresponding to the first key group from the ground. Subtract the height H1 from the projection point coordinates to obtain the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0021] S3a4. According to priority, starting from the left and right hips, traverse multiple feature point groups in sequence to determine whether there are two feature point groups whose confidence in the two-dimensional pixel coordinates is greater than the first threshold.

[0022] If not, the extraction is deemed invalid.

[0023] If so, define them as the third key group and the fourth key group in order of priority, and calculate the distance X2 between the third key group and the fourth key group; based on the distance X2 and the proportion of the pedestrian's body, calculate the height H2 of the feature point group with higher priority from the ground; obtain the coordinates of the pedestrian's center point based on the third key group, subtract the height H2 from the coordinates of the pedestrian's center point, and obtain the coordinates of the projection point of the pedestrian's center point on the two-dimensional line segment.

[0024] Preferably, according to priority, multiple feature point groups are located on the left and right heels, left and right ankles, left and right little toes, and left and right big toes. Specifically, in step S3a3, when the first key group is the left and right heels, left and right little toes, or left and right big toes, the projection point coordinates are the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0025] Preferably, step S3 further includes determining the pedestrian's orientation three-dimensional vector relative to the camera coordinates, including the following steps:

[0026] Determine if the confidence levels of the left and right hips and shoulders are both greater than the first threshold. If so, determine the three-dimensional pixel coordinates of the left and right hips and shoulders when the center point of the pedestrian's upper body is at the camera coordinate origin and the human body is facing forward towards the camera. Using these coordinates as reference coordinates, calculate the pedestrian camera orientation three-dimensional vector relative to the camera coordinates using the solve PNP method, based on the camera intrinsic parameters, camera distortion parameters, reference coordinates, and the two-dimensional pixel coordinates of the left and right hips and shoulders extracted from the pedestrian video image. When the angle between the pedestrian camera orientation three-dimensional vector and the camera's forward vector is 75-105°, calculate the toe direction two-dimensional vector based on the two-dimensional pixel coordinates of one of the little toe and big toe on the same side of the pedestrian and both greater than the first threshold, as well as the heel. If the angle between the two-dimensional vectors of the pedestrian camera orientation in the x and y axes and the two-dimensional vector of the toe direction is acute, do not change the pedestrian camera orientation three-dimensional vector; if it is obtuse, change the pedestrian camera orientation three-dimensional vector to the opposite direction.

[0027] Preferably, step S4 further includes converting the camera orientation 3D vector into a pedestrian 3D orientation vector on a 3D point cloud map.

[0028] Preferably, step S4 includes the following steps:

[0029] Based on the 3D point cloud map, obtain the 3D point cloud of the ground and generate a 3D coordinate image;

[0030] Based on the camera's intrinsic and extrinsic parameters, the pedestrian's two-dimensional projection coordinates on the ground are projected onto the ground's three-dimensional point cloud with the camera's optical center as the origin. It is then determined whether the corresponding three-dimensional coordinates have been found. If so, the three-dimensional coordinates are obtained; otherwise, the two-dimensional projection coordinates are moved down one row at intervals until the corresponding three-dimensional coordinates are found.

[0031] Preferably, the pedestrian's joint points also include the left and right eyes, left and right ears, left and right elbows, left and right wrists, and the center.

[0032] Preferably, step S2 further includes S2c, calculating the average confidence level of the two-dimensional pixel coordinates of all the pedestrian's joints, determining whether the average confidence level is greater than the second threshold, and if so, outputting the two-dimensional pixel coordinates and confidence level of the pedestrian's main joints; if not, determining it as invalid extraction, wherein the second threshold is greater than the first threshold.

[0033] The present invention has at least the following beneficial effects:

[0034] The two-dimensional projection coordinates of pedestrians on the ground are obtained by using the main pedestrian joints, and then combined with the three-dimensional point cloud to convert them into three-dimensional coordinates. Specifically: j. The accuracy of obtaining two-dimensional projection coordinates is improved by setting the priority of the main joints and the method of two-dimensional projection coordinates; k. The relationship between two-dimensional projection coordinates and three-dimensional point cloud is used to accurately convert them into three-dimensional coordinates.

[0035] By detecting information from the left and right hips and shoulders, as well as the direction of the toes, the three-dimensional vector of the pedestrian camera's orientation can be determined across the entire range, solving the problem that traditional methods cannot determine the left and right orientation when the three-dimensional vector of the pedestrian camera's orientation is nearly perpendicular to the camera's forward vector.

[0036] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0037] Figure 1 A flowchart illustrating real-time personnel positioning based on a three-dimensional model of a substation, as described in one of the technical solutions of this invention;

[0038] Figure 2 This is a schematic diagram showing the positions of each joint point according to one of the technical solutions of the present invention;

[0039] Figure 3 This is a flowchart illustrating the process of obtaining the two-dimensional projection coordinates of the pedestrian's center point on the ground, as described in one of the technical solutions of the present invention. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to embodiments, so that those skilled in the art can implement it based on the description.

[0041] like Figure 1 As shown, the real-time personnel positioning method based on a 3D model of a substation includes the following steps:

[0042] (1) Construction of 3D point cloud map of substation

[0043] Substation scanning data is obtained; the scanning data is modeled and output as a 3D point cloud map;

[0044] (2) Acquisition of pedestrian video images

[0045] The optical cameras used for monitoring in the physical scene of the substation are managed in a unified manner, and a large amount of image information (real-time video stream) is acquired in a round-robin manner. The real-time video stream is obtained from the video source through RTSP streaming media and decoded by the video decoding module to output the decoded video stream.

[0046] Based on the decoded video stream, determine the pedestrian video images that need to be detected;

[0047] (3) For example Figure 3 As shown, the two-dimensional pixel coordinates and confidence scores of each pedestrian joint are extracted from the acquired pedestrian video images, and the two-dimensional pixel coordinates and confidence scores of the main joints are output.

[0048] Specifically, the following steps are included:

[0049] (3.1) The pedestrian detection module performs pedestrian detection on the pedestrian video image and outputs a two-dimensional detection box and a cropped image within the two-dimensional detection box. The two-dimensional detection box corresponds to the coordinates of four vertices (top left, top right, bottom right and bottom left), or it can be the coordinates of two vertices that are diagonally related (top left and bottom right, or bottom left and top right).

[0050] (3.2) The pedestrian joint detection module obtains the pedestrian cutout based on the cropped image, extracts the two-dimensional pixel coordinates of each joint of the pedestrian based on the pedestrian cutout and the two-dimensional detection box, and obtains the confidence of each joint simultaneously.

[0051] Output the two-dimensional pixel coordinates and confidence scores of the main joints, including the following steps:

[0052] (3.2.1) Using skeleton extraction, the correspondence between each joint and the human body is obtained, specifically as follows: Figure 2 As shown, Figure 2 The corresponding part for each number in the table is shown in Table 1 below:

[0053] Table 1. Pedestrian Key Point Statistics

[0054] 15: Right eye; 16: Left eye; 17: Right ear; 0: Nose; 18: Left ear; 2: Right shoulder; 1: Neck; 5: Left shoulder; 3: Right elbow; 6: Left elbow; 4: Right wrist; 9: Right hip; 8: Middle hip; 12: Left hip; 7: Left wrist; 10: Right knee; 13: Left knee; 11: Right ankle; 14: Left ankle; 22: Right - both toes; 23: Right - little toe; 24: Right - heel; 19: Left - both toes; 20: Left - little toe; 21: Left - heel;

[0055] In this step, the accuracy of data extraction is improved by utilizing multiple joints and the relationships between them.

[0056] Preferably, in step 3.3, while extracting the two-dimensional pixel coordinates of each joint of the pedestrian, the confidence level of the two-dimensional pixel coordinates of each joint is obtained, the average confidence level of the two-dimensional pixel coordinates of all joints of the pedestrian is calculated, and it is determined whether the average confidence level is greater than the second threshold. If so, the two-dimensional pixel coordinates and confidence level of the main joint of the pedestrian are output. If not, it is determined to be an invalid extraction and is not output. In one specific embodiment, the second threshold is set to 0.1.

[0057] (3.2.2) Output the two-dimensional pixel coordinates and confidence scores of the main joints, where the main joints of the pedestrian include multiple feature point groups;

[0058] In one of the technical solutions (Solution 1), multiple feature point groups are arranged in order of priority, including left and right heels, left and right ankles, left and right knees, left and right hips, left and right shoulders, neck, and nose.

[0059] In another technical solution (Solution 2), multiple feature point groups are arranged in order of priority, including left and right heels, left and right little toes, left and right big toes, left and right ankles, left and right knees, left and right hips, left and right shoulders, neck, and nose.

[0060] (4) Using the pedestrian two-dimensional positioning module, based on the two-dimensional pixel coordinates and confidence level of the pedestrian's main joints, calculate the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0061] (4.1) For Scheme 1, the following steps are included:

[0062] S3a1. Determine the human proportions of pedestrians, including:

[0063] ①: Determine the height ratio of the upper body to the lower body. Specifically, define the distance between the heel and the center point of the left and right hips as the foot-hip distance. Determine the ratio of the foot-hip distance to the distance between any two characteristic point groups of the left and right hips, left and right shoulders, neck, and nose. The distance ratio between any two characteristic point groups of the left and right hips, left and right shoulders, neck, and nose includes:

[0064] Basic distances: The distance between the center points of the left and right hips and the center points of the left and right shoulders is the hip-shoulder distance; the distance between the center points of the left and right shoulders and the neck is the shoulder-neck distance; and the distance between the shin and the nose is the shin-nose distance.

[0065] Extended distance: Based on the basic distance, the distance between the center point of the left and right hips and the neck is the hip-neck distance, the distance between the center point of the left and right hips and the nose is the hip-nose distance, and the distance between the center point of the left and right shoulders and the nose is the shoulder-nose distance. Among them, the hip-nose distance is equal to the hip-shoulder distance + shoulder-neck distance + shin-nose distance.

[0066] ②: Determine the ratio of the distance between the knee and the ankle to the distance between the ankle and the heel;

[0067] S3a2. Determine the two-dimensional line segment, specifically: according to priority, traverse multiple feature point groups with priority no greater than that of the left and right ankles in turn, determine whether there is a feature point group whose confidence of the two-dimensional pixel coordinates is greater than the first threshold. If so, define the feature point group as the first key group, connect the two-dimensional pixel coordinates of the first key group to obtain the two-dimensional line segment and proceed to step S3a3. If not, proceed to step S3a4.

[0068] S3a3. According to priority, starting from the left and right hips, traverse multiple feature point groups in sequence to determine whether there is a feature point group whose confidence of the two-dimensional pixel coordinates is greater than the first threshold. If so, define the feature point group as the second key group. If not, judge it as invalid extraction.

[0069] The pedestrian center point coordinates are obtained based on the second key group. When the second key group is the left and right hips or the left and right shoulders, the corresponding pedestrian center point coordinates are the midpoint coordinates of the line connecting the left and right hips or the midpoint coordinates of the line connecting the left and right shoulders. When the second key group is the neck or nose, the shin coordinates or nose coordinates are the pedestrian center point coordinates.

[0070] Projecting the pedestrian's center point coordinates vertically downwards onto a two-dimensional line segment yields the coordinates of the projected point of the pedestrian's center point on the two-dimensional line segment, where:

[0071] When the first key group is the left and right heels, the coordinates of the projection point are the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0072] When the first key group is the left and right ankles, determine whether there is a joint point in the two-dimensional pixel coordinates of the left and right knees with a confidence level greater than the first threshold. If so, calculate the inter-joint distance X1 between the knee and ankle (the pixel distance displayed on the pedestrian video image, and at this time the knee and ankle are on the same side of the pedestrian). Based on this distance and the proportion of the pedestrian's body (set the ratio of the inter-joint distance between the knee and ankle and the distance between the ankle and the heel in the proportion of the pedestrian's body to be X0:H0), calculate the height H1 of the joint point corresponding to the first key group from the ground, H1=(X1×H0) / X0. Subtract the height H1 from the projection point coordinates to obtain the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0073] S3a4. Based on priority, starting from the left and right hips, traverse multiple feature point groups sequentially to determine whether there are two sets of features.

[0074] The confidence level of the two-dimensional pixel coordinates of the point group is greater than the first threshold;

[0075] If not, the extraction is deemed invalid.

[0076] If so, define them as the third key group and the fourth key group in order of priority. Calculate the distance X2 between the third key group and the fourth key group, where X2 is the pixel distance displayed on the pedestrian video image. When the third key group and the fourth key group are the left and right hips or the left and right shoulders, take the center point of each group of feature points for calculation.

[0077] Based on distance X2 and the human body proportions of pedestrians (the ratio of distance between the third and fourth key groups and the distance between the feet and hips is X3:H3), calculate the height H2 of the high-priority feature point group from the ground, H2=(X2×H3) / X3;

[0078] The pedestrian center point coordinates are obtained based on the third key group. When the second key group is the left and right hips or the left and right shoulders, the corresponding pedestrian center point coordinates are the midpoint coordinates of the line connecting the left and right hips or the midpoint coordinates of the line connecting the left and right shoulders. When the second key group is the neck, the shin coordinates are the pedestrian center point coordinates.

[0079] Subtract the height H2 from the coordinates of the pedestrian's center point to obtain the coordinates of the projection point of the pedestrian's center point on the two-dimensional line segment;

[0080] In one embodiment, specifically, when the first key group is the left and right ankles and the second key group is the left and right hips, the following steps are included:

[0081] Process the two-dimensional pixel coordinates of the left and right hips to obtain the coordinates of the center point of the hips, which are the coordinates of the pedestrian's center point.

[0082] The two-dimensional line segment connecting the two ankles (left ankle and right ankle) was calculated;

[0083] Based on the distance between the left ankle and the left and right knees (the pixel distance displayed on the pedestrian video image), combined with the pedestrian's body proportions (the ratio of the distance between the heel and ankle and the distance between the ankle and knee), the pixel height H1 between the heel and ankle is calculated;

[0084] Project the coordinates of the center point of the hip vertically downwards onto a two-dimensional line segment to obtain the projection point of the pedestrian's center point on the line connecting the ankle and the groin, and obtain the coordinates of the projection point.

[0085] Subtract the calculated foot height H1 from the projection point to obtain the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0086] (4.2) For Scheme 2, the following steps are included:

[0087] S3a1. Determine the human proportions of pedestrians, including:

[0088] ①: Determine the height ratio of the upper body to the lower body. Specifically, define the distance between the heel and the center point of the left and right hips as the foot-hip distance. Determine the ratio of the foot-hip distance to the distance between any two characteristic point groups of the left and right hips, left and right shoulders, neck, and nose. The distance ratio between any two characteristic point groups of the left and right hips, left and right shoulders, neck, and nose includes:

[0089] Basic distances: The distance between the center points of the left and right hips and the center points of the left and right shoulders is the hip-shoulder distance; the distance between the center points of the left and right shoulders and the neck is the shoulder-neck distance; and the distance between the shin and the nose is the shin-nose distance.

[0090] Extended distance: Based on the basic distance, the distance between the center point of the left and right hips and the neck is the hip-neck distance, the distance between the center point of the left and right hips and the nose is the hip-nose distance, and the distance between the center point of the left and right shoulders and the nose is the shoulder-nose distance. Among them, the hip-nose distance is equal to the hip-shoulder distance + shoulder-neck distance + shin-nose distance.

[0091] ②: Determine the ratio of the distance between the knee and the ankle to the distance between the ankle and the heel;

[0092] S3a2. Determine the two-dimensional line segment, specifically: according to priority, traverse multiple feature point groups with priority no greater than that of the left and right ankles in turn, determine whether there is a feature point group whose confidence of the two-dimensional pixel coordinates is greater than the first threshold. If so, define the feature point group as the first key group, connect the two-dimensional pixel coordinates of the first key group to obtain the two-dimensional line segment and proceed to step S3a3. If not, proceed to step S3a4.

[0093] S3a3. According to priority, starting from the left and right hips, traverse multiple feature point groups in sequence to determine whether there is a feature point group whose confidence of the two-dimensional pixel coordinates is greater than the first threshold. If so, define the feature point group as the second key group. If not, judge it as invalid extraction.

[0094] The pedestrian center point coordinates are obtained based on the second key group. When the second key group is the left and right hips or the left and right shoulders, the corresponding pedestrian center point coordinates are the midpoint coordinates of the line connecting the left and right hips or the midpoint coordinates of the line connecting the left and right shoulders. When the second key group is the neck or nose, the shin coordinates or nose coordinates are the pedestrian center point coordinates.

[0095] Projecting the pedestrian's center point coordinates vertically downwards onto a two-dimensional line segment yields the coordinates of the projected point of the pedestrian's center point on the two-dimensional line segment, where:

[0096] When the first key group is the left and right heels, left and right little toes, or left and right big toes, the coordinates of the projection point are the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0097] When the first key group is the left and right ankles, determine whether there is a joint point in the two-dimensional pixel coordinates of the left and right knees with a confidence level greater than the first threshold. If so, calculate the inter-joint distance X1 between the knee and ankle (the pixel distance displayed on the pedestrian video image, and at this time the knee and ankle are on the same side of the pedestrian). Based on this distance and the proportion of the pedestrian's body (set the ratio of the inter-joint distance between the knee and ankle and the distance between the ankle and the heel in the proportion of the pedestrian's body to be X0:H0), calculate the height H1 of the joint point corresponding to the first key group from the ground, H1=(X1×H0) / X0. Subtract the height H1 from the projection point coordinates to obtain the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0098] S3a4. Based on priority, starting from the left and right hips, traverse multiple feature point groups sequentially to determine whether there are two sets of features.

[0099] The confidence level of the two-dimensional pixel coordinates of the point group is greater than the first threshold;

[0100] If not, the extraction is deemed invalid.

[0101] If so, define them as the third key group and the fourth key group in order of priority. Calculate the distance X2 between the third key group and the fourth key group, where X2 is the pixel distance displayed on the pedestrian video image. When the third key group and the fourth key group are the left and right hips or the left and right shoulders, take the center point of each group of feature points for calculation.

[0102] Based on distance X2 and the human body proportions of pedestrians (the ratio of distance between the third and fourth key groups and the distance between the feet and hips is X3:H3), calculate the height H2 of the high-priority feature point group from the ground, H2=(X2×H3) / X3;

[0103] The pedestrian center point coordinates are obtained based on the third key group. When the second key group is the left and right hips or the left and right shoulders, the corresponding pedestrian center point coordinates are the midpoint coordinates of the line connecting the left and right hips or the midpoint coordinates of the line connecting the left and right shoulders. When the second key group is the neck, the shin coordinates are the pedestrian center point coordinates.

[0104] Subtract the height H2 from the coordinates of the pedestrian's center point to obtain the coordinates of the projection point of the pedestrian's center point on the two-dimensional line segment;

[0105] In one embodiment, specifically, when the first key group is the left and right ankles and the second key group is the left and right hips, the following steps are included:

[0106] Process the two-dimensional pixel coordinates of the left and right hips to obtain the coordinates of the center point of the hips, which are the coordinates of the pedestrian's center point.

[0107] The two-dimensional line segment connecting the two ankles (left ankle and right ankle) was calculated;

[0108] Based on the distance between the left ankle and the left and right knees (the pixel distance displayed on the pedestrian video image), combined with the pedestrian's body proportions (the ratio of the distance between the heel and ankle and the distance between the ankle and knee), the pixel height H1 between the heel and ankle is calculated;

[0109] Project the coordinates of the center point of the hip vertically downwards onto a two-dimensional line segment to obtain the projection point of the pedestrian's center point on the line connecting the ankle and the groin, and obtain the coordinates of the projection point.

[0110] Subtract the calculated foot height H1 from the projection point to obtain the two-dimensional projection coordinates of the pedestrian's center point on the ground.

[0111] (5) Using the 3D mapping module, based on the camera's intrinsic and extrinsic parameters, the 3D point cloud map, and the 2D projection coordinates of the pedestrian's center point on the ground, the 3D coordinates are obtained. This includes the following steps:

[0112] Based on the 3D point cloud map, obtain the 3D point cloud of the ground and generate a 3D coordinate image;

[0113] Based on the camera's intrinsic and extrinsic parameters, the pedestrian's two-dimensional projection coordinates on the ground are projected onto the ground's three-dimensional point cloud with the camera's optical center as the origin. It is then determined whether the corresponding three-dimensional coordinates have been found. If so, the three-dimensional coordinates are obtained; otherwise, the two-dimensional projection coordinates are moved down one row until the corresponding three-dimensional coordinates are found.

[0114] (6) Determine the three-dimensional orientation vector of the pedestrian relative to the camera coordinates using the pedestrian camera pose detection module, including the following steps:

[0115] Determine if the confidence levels of the left and right hips and shoulders are both greater than the first threshold. If so, determine the three-dimensional pixel coordinates of the left and right hips and shoulders when the center point of the pedestrian's upper body is at the camera coordinate origin and the human body is facing forward towards the camera. Using these coordinates as reference coordinates, and based on the camera intrinsic parameters, camera distortion parameters, reference coordinates, and the two-dimensional pixel coordinates of the left and right hips and shoulders extracted from the pedestrian video image, calculate the three-dimensional vector of the pedestrian's camera orientation relative to the camera coordinates using the solve PNP method. When the angle between the three-dimensional vector of the pedestrian's camera orientation and the forward vector of the camera is 75-105°, calculate the two-dimensional vector of the toe direction based on the two-dimensional pixel coordinates of one of the little toe and big toe on the same side of the pedestrian and both greater than the first threshold, as well as the heel. If the angle between the two-dimensional vectors of the pedestrian's camera orientation along the x and y axes and the two-dimensional vector of the toe direction is acute, do not change the three-dimensional vector of the pedestrian's camera orientation; if it is obtuse, change the three-dimensional vector of the pedestrian's camera orientation to the opposite direction.

[0116] In the above scheme, Solvnp requires the camera coordinates to be three-dimensional coordinates, where the X-axis direction is left and right, with right being the positive direction; the Y-axis direction is up and down, with upward being the positive direction; and the Z-axis direction is forward and backward, with forward being the positive direction.

[0117] (7) The pedestrian orientation two-dimensional vector is converted into a three-dimensional orientation vector on the three-dimensional point cloud map through the three-dimensional mapping module;

[0118] (8) Through the pedestrian three-dimensional posture detection module, based on the temporal correlation of the input pedestrian three-dimensional coordinates and the previously calculated three-dimensional orientation vector, the current three-dimensional orientation information of the pedestrian is further calculated.

[0119] Interface design:

[0120] The entire system runs in a ROS environment, with multiple ROS functional node processes exchanging information via ROS messages, specifically:

[0121] The video decoding node runs independently as a ROS Node. It is responsible for obtaining real-time compressed video streams from the front-end network camera in the form of RTSP streams, decoding them, and then publishing them out through the ROS standard Image message.

[0122] The pedestrian detection node receives std_msg / Image messages from the video decoding node, performs target detection on the input video frame, and if a target is found, sends the detected target information through the custom Ros message ObjectDetect2d.

[0123] The pedestrian joint detection node receives the ObjectDetect2d message published by the pedestrian detection node, processes the target detection box and pedestrian cropped image contained in the message, and obtains the joint information of each pedestrian in the pedestrian cropped image. The joint message is published through the custom ROS message PersonPose2d.

[0124] The pedestrian pose detection node receives the PersonPose2d message published by the pedestrian joint detection node, processes the joint points contained in the message to obtain the pedestrian's current 3D coordinates and 3D orientation vector, and publishes it in the PersonPose3d custom message format.

[0125] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for real-time personnel positioning based on a 3D model of a substation, characterized in that, Includes the following steps: S1. Construct a 3D point cloud map of the substation; S2. Extract the two-dimensional pixel coordinates of each key point of the pedestrian from the acquired pedestrian video image, and output the two-dimensional pixel coordinates and confidence of the main key point; S3. Based on the two-dimensional pixel coordinates and confidence level of the pedestrian's main joints, calculate the two-dimensional projection coordinates of the pedestrian's center point on the ground. S4. Based on the camera's intrinsic and extrinsic parameters, the 3D point cloud map, and the 2D projection coordinates of the pedestrian's center point on the ground, obtain the 3D coordinates. Step S3 includes the following steps: S3a1. Determine the human body proportions of pedestrians; S3a2. Determine the two-dimensional line segment, specifically: according to priority, traverse multiple feature point groups with priority no greater than that of the left and right ankles in turn, determine whether there is a feature point group whose confidence of the two-dimensional pixel coordinates is greater than the first threshold. If so, define the feature point group as the first key group, connect the two-dimensional pixel coordinates of the first key group to obtain the two-dimensional line segment and proceed to step S3a3. If not, proceed to step S3a4. S3a3. According to priority, starting from the left and right hips, traverse multiple feature point groups sequentially to determine whether there is a feature point group whose confidence in the two-dimensional pixel coordinates is greater than the first threshold. If so, define this feature point group as the second key group. No, this is considered an invalid extraction; Based on the second key group, the coordinates of the pedestrian's center point are obtained. These coordinates are then projected vertically downwards onto a two-dimensional line segment to obtain the coordinates of the projection point of the pedestrian's center point on the two-dimensional line segment, where: When the first key group is the left and right heels, the coordinates of the projection point are the two-dimensional projection coordinates of the pedestrian's center point on the ground. When the first key group is the left and right ankles, determine whether there is a joint point in the two-dimensional pixel coordinates of the left and right knees with a confidence level greater than the first threshold. If so, calculate the joint distance between the knee and the ankle. Based on this distance and the human body proportion of the pedestrian, calculate the height H1 of the joint point corresponding to the first key group from the ground. Subtract the height H1 from the projection point coordinates to obtain the two-dimensional projection coordinates of the pedestrian's center point on the ground. S3a4. According to priority, starting from the left and right hips, traverse multiple feature point groups in sequence to determine whether there are two feature point groups whose confidence in the two-dimensional pixel coordinates is greater than the first threshold. If not, the extraction is deemed invalid. If so, define them as the third key group and the fourth key group in order of priority, and calculate the distance X2 between the third key group and the fourth key group; based on the distance X2 and the proportion of the pedestrian's body, calculate the height H2 of the feature point group with higher priority from the ground; obtain the coordinates of the pedestrian's center point based on the third key group, subtract the height H2 from the coordinates of the pedestrian's center point, and obtain the coordinates of the projection point of the pedestrian's center point on the two-dimensional line segment.

2. The method for real-time personnel positioning based on a three-dimensional model of a substation as described in claim 1, characterized in that, Step S2 includes the following steps: S2a: Perform pedestrian detection on the pedestrian video image and output a two-dimensional detection box and a cropped image within the two-dimensional detection box. S2b: Obtain pedestrian cutout based on cropped image. Extract 2D pixel coordinates of each joint of pedestrian based on pedestrian cutout and 2D detection box, and output the 2D pixel coordinates and confidence of main joint. The main joint of pedestrian includes multiple feature point groups, arranged in order of priority. Multiple feature point groups include left and right heels, left and right ankles, left and right knees, left and right hips, left and right shoulders, neck, and nose.

3. The method for real-time personnel positioning based on a three-dimensional model of a substation as described in claim 1, characterized in that, According to priority, multiple feature point groups are located on the left and right heels, left and right ankles, left and right little toes, and left and right big toes. Specifically, in step S3a3, when the first key group is the left and right heels, left and right little toes, or left and right big toes, the projection point coordinates are the two-dimensional projection coordinates of the pedestrian's center point on the ground.

4. The method for real-time personnel positioning based on a three-dimensional model of a substation as described in claim 3, characterized in that, Step S3 also includes determining the pedestrian's orientation 3D vector relative to the camera coordinates, including the following steps: Determine if the confidence levels of the left and right hips and shoulders are both greater than the first threshold. If so, determine the three-dimensional pixel coordinates of the left and right hips and shoulders when the center point of the pedestrian's upper body is at the camera coordinate origin and the human body is facing forward towards the camera. Using these coordinates as reference coordinates, calculate the pedestrian camera orientation three-dimensional vector relative to the camera coordinates using the solve PNP method, based on the camera intrinsic parameters, camera distortion parameters, reference coordinates, and the two-dimensional pixel coordinates of the left and right hips and shoulders extracted from the pedestrian video image. When the angle between the pedestrian camera orientation three-dimensional vector and the camera's forward vector is 75-105°, calculate the toe direction two-dimensional vector based on the two-dimensional pixel coordinates of one of the little toe and big toe on the same side of the pedestrian and both greater than the first threshold, as well as the heel. If the angle between the two-dimensional vectors of the pedestrian camera orientation in the x and y axes and the two-dimensional vector of the toe direction is acute, do not change the pedestrian camera orientation three-dimensional vector; if it is obtuse, change the pedestrian camera orientation three-dimensional vector to the opposite direction.

5. The method for real-time personnel positioning based on a three-dimensional model of a substation as described in claim 4, characterized in that, Step S4 also includes converting the camera orientation 3D vector into a pedestrian 3D orientation vector on a 3D point cloud map.

6. The method for real-time personnel positioning based on a three-dimensional model of a substation as described in claim 1, characterized in that, Step S4 includes the following steps: Based on the 3D point cloud map, obtain the 3D point cloud of the ground and generate a 3D coordinate image; Based on the camera's intrinsic and extrinsic parameters, the pedestrian's two-dimensional projection coordinates on the ground are projected onto the ground's three-dimensional point cloud with the camera's optical center as the origin. It is then determined whether the corresponding three-dimensional coordinates have been found. If so, the three-dimensional coordinates are obtained; otherwise, the two-dimensional projection coordinates are moved down one row at intervals until the corresponding three-dimensional coordinates are found.

7. The method for real-time personnel positioning based on a three-dimensional model of a substation as described in claim 3, characterized in that, The key points for pedestrians also include the left and right eyes, left and right ears, left and right elbows, left and right wrists, and the center of the body.

8. The method for real-time personnel positioning based on a three-dimensional model of a substation as described in claim 7, characterized in that, Step S2 also includes S2c, calculating the average confidence level of the two-dimensional pixel coordinates of all the pedestrian's joints, and determining whether the average confidence level is greater than the second threshold. If so, outputting the two-dimensional pixel coordinates and confidence level of the pedestrian's main joints; otherwise, determining it as invalid extraction. The second threshold is greater than the first threshold.

Images