Image-based insulator digital three-dimensional posture detection method and system

Abstract

The application discloses a kind of based on image insulator digitization three-dimensional posture detection method and system, method includes: pre-constructing insulator posture recognition model and insulator three-dimensional model library;Obtain insulator image and input to insulator posture recognition model and carry out posture recognition, obtain the posture Euler angle of target insulator, the insulator bottom center coordinates under camera coordinate system and the pixel mask of insulator body;Insulator three-dimensional model is rotated according to Euler angle, and is translated and insulator bottom center coordinates align, obtain the projection of insulator three-dimensional model on insulator image;Traverse the depth of insulator three-dimensional model, obtain the intersection and union of projection and the pixel mask of insulator body ratio;When ratio is at maximum, corresponding insulator model on insulator image is regarded as insulator depth posture model, and then insulator three-dimensional posture information is obtained.The application has the advantages of short time consumption, low cost, high precision and the like.

Description

Technical Field

[0001] This invention mainly relates to the field of insulator attitude detection technology, specifically to an image-based digital three-dimensional attitude detection method and system for insulators. Background Technology

[0002] High-voltage, high-altitude insulators are located in hazardous environments, and their maintenance and cleaning often require the assistance of specialized robotic arms. However, manual operation of these arms is prone to accidents due to limited experience. To enable autonomous operation, digital modeling of the target insulator is necessary, allowing the robotic arm to accurately determine its positional relationship with the insulator. Two common methods for creating 3D models of targets are as follows: The first method involves obtaining point cloud data through laser scanning, followed by modeling using other software. The disadvantages are that laser scanners are expensive and time-consuming, and they increase the weight on the robotic arm, affecting structural design.

[0003] The second method is depth camera modeling. There are different types of depth cameras, including binocular, structured light, and time-of-flight (TOF) cameras. They emit infrared light, light pulses, etc., and receive signals reflected from the object's surface, using algorithms to calculate depth data. However, this method requires specialized cameras, and its performance is poor in outdoor high-altitude scenes due to the influence of the solar spectrum. It also requires significant computational resources and time, making it unsuitable for rapid insulator modeling. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide an image-based digital three-dimensional attitude detection method and system for insulators that is time-efficient, low-cost, and highly accurate, addressing the technical problems existing in the prior art.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows: A method for digital three-dimensional pose detection of insulators based on images, comprising the following steps: S1. Pre-construct an insulator attitude recognition model and an insulator 3D model library; the insulator 3D model library contains 3D models of various types of insulators. S2. Acquire the insulator image and input it into the insulator attitude recognition model for attitude recognition, obtaining the Euler angles of the target insulator's attitude and the coordinates of the insulator's bottom center in the camera coordinate system. And the pixel mask of the insulator body; S3. Rotate the preset 3D insulator model in the insulator 3D model library according to the Euler angles in step S2, and translate it relative to the coordinates of the bottom center of the insulator in the camera coordinate system. Align the insulator to obtain the projection of the 3D model of the insulator onto the insulator image; S4. Traverse the depth of the insulator's 3D model and obtain the ratio of the intersection and union of the pixel mask of the projection and the insulator body; when the ratio is at its maximum value, the corresponding 3D model of the insulator on the insulator image is taken as the insulator's depth attitude model, and thus the 3D attitude information of the insulator is obtained.

[0006] Preferably, the specific process of constructing the insulator attitude recognition model in step S1 is as follows: Collect images of insulators and perform attitude annotation, center annotation, and segmentation annotation on each image; A shared convolutional neural network is used as the backbone network to extract general features of the image; An attention module is embedded in the backbone network, and multi-scale features are fused through a skip connection structure. Three task branches are constructed at the end of the backbone network: including pose estimation branch, circle center localization branch, and segmentation branch; Construct a composite loss function by establishing geometric constraints between the outputs of the three task branches; End-to-end joint training was performed to obtain the insulator attitude recognition model.

[0007] Preferably, the center location branch adopts a heatmap regression method, which converts the marked center coordinates into a Gaussian heatmap as a learning target, predicts the heatmap, and obtains the center coordinates by finding the location of the maximum value.

[0008] Preferably, the attention module uses a CBAM or SE module to redistribute weights on the channel dimension and spatial dimension; the multi-scale feature fusion uses an FPN or U-Net-style skip connection structure to fuse low-level high-resolution features with high-level semantic features.

[0009] Preferably, the composite loss function is as follows:

[0010] in: This is a composite loss; Loss is estimated for attitude; For the regression loss of the heatmap centered on the circle; The cross-entropy and Dice loss of the segmented regions; This is a geometric consistency constraint term used to ensure the consistency of the three outputs in geometric space; where... , , and For the corresponding weights.

[0011] Preferably, in step S2, the pixel coordinates of the bottom center of the insulator are obtained. The specific process is as follows: Detect the coordinates of the bottom center of the insulator in the insulator image. ; Then move the camera a certain distance in both the horizontal and vertical directions. Acquire an insulator image and then detect the coordinates of the bottom center of the insulator circle on the insulator image again. The coordinates of the moved insulator in the camera coordinate system are: The coordinates of the insulator in the original camera coordinate system before the movement are then... ; Will and Substitute into the formula respectively In and The solution formula is obtained as follows:

[0012] The coordinates of the bottom center of the insulator in the original camera coordinate system are obtained by solving the formula. ;in Indicates the column number in the image coordinate system. Indicates the row number; , For image scaling parameters, , This refers to the imaging offset parameter.

[0013] Preferably, an image of the insulator is acquired by a camera, which is located on a robotic arm; the robotic arm moves the camera a certain distance in both the horizontal and vertical directions.

[0014] Preferably, the plane formed by the x-axis and y-axis of the camera coordinate system formed by the installed camera is pre-calibrated to be parallel to the plane formed by the x-axis and y-axis of the robot arm coordinate system, and distortion correction is performed on the camera image.

[0015] Preferably, in step S1, an insulator in a stationary safe position is scanned using a laser scanner to establish a three-dimensional model library of insulators.

[0016] The present invention also discloses an image-based digital three-dimensional attitude detection system for insulators, comprising a memory and a processor connected to each other, wherein the memory stores a computer program, and the computer program executes the steps of the method described above when run by the processor.

[0017] Compared with the prior art, the advantages of the present invention are as follows: The detection method of the present invention is based on the principle of camera pinhole imaging. It presets three-dimensional models of various types of insulators by establishing a three-dimensional insulator model library. Then, the three-dimensional models are reduced to two-dimensional images by rotating, translating and scaling to remove depth information. The images are then matched with insulators in the camera coordinate system. The matched modeling depth map replaces the scanning modeling of the current target insulator. The effect is similar, but it is faster and cheaper. Attached Figure Description

[0018] Figure 1 This is a flowchart of the detection method in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart illustrating the construction process of the insulator attitude recognition model in this embodiment of the invention.

[0020] Figure 3 This is the original image of the insulator in an embodiment of the present invention.

[0021] Figure 4 This is a segmentation diagram of the insulator mask in an embodiment of the present invention.

[0022] Figure 5 This is a 3D generated image of an insulator in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 As shown in the figure, the image-based digital three-dimensional attitude detection method for insulators according to an embodiment of the present invention specifically includes the following steps: S1. Pre-build an insulator attitude recognition model and an insulator 3D model library; like Figure 2 As shown, a deep learning model for insulator attitude recognition is pre-trained. The input to this model is a color image containing insulators. The specific training process of the insulator attitude recognition model is as follows: S101. Data Preparation and Labeling Stage We collected a dataset of color images of insulators of various types, at different angles, and under different lighting conditions; each image underwent the following three annotation processes: Attitude annotation: Annotate the Euler angles (Yaw, Pitch, Roll) of the insulator; where Yaw is the yaw angle, Pitch is the pitch angle, and Roll is the roll angle. Center mark: Marks the pixel coordinates (x, y) of the center of the bottom circle of the insulator; Segmentation labeling: The pixel mask used to label the insulator body.

[0025] The labeled image data is then divided into training, validation, and test sets in a ratio of 7:2:1. S102. Construct a shared feature extraction network A shared convolutional neural network (CNN) is used as the backbone network to extract common features from images. The backbone network can adopt ResNet, VGG, or a custom CNN structure. The number of parameters is reduced by using a shared CNN (feature extraction layer), while enabling the three tasks to utilize common high-level semantic representations. Specifically, a unified encoder is designed to extract multi-level feature maps from the input image. By extracting features through a shared CNN, the three related tasks in the subsequent output can mutually promote learning, improve data utilization efficiency, and enhance model generalization ability.

[0026] S103. Introducing Attention Modules and Multi-Scale Feature Fusion Embed attention modules and skip connection structures in convolutional neural networks, specifically: Add a CBAM (Convolutional Block Attention Module) or SE (Squeeze-and-Excitation) attention module after each convolutional layer or residual block to redistribute the weights of the channel dimension and spatial dimension, making the model more focused on the insulator target region; By employing a skip connection structure similar to FPN (Feature Pyramid Network) or U-Net, low-level high-resolution detailed features are fused with high-level abstract features rich in semantic information. Additionally, this step implements attention-guided multi-scale feature fusion: The channel attention module is used to capture the differences in the importance of insulators in different channels; Enhance the model's response to critical regions of the insulator by utilizing a spatial attention module; By fusing features at multiple scales, the accuracy of segmentation boundaries and the precision of center localization are maintained simultaneously.

[0027] S104. Construct the three main task output branches Three parallel task branches (Heads) are connected at the end of the backbone network. The specific design of each branch is as follows: 1. Pose estimation branch: Output the Euler angle vector of the insulator [Yaw, Pitch, Roll]; Training is performed using a regression loss function (such as MSE or MAE).

[0028] 2. Center positioning branch: The marked center coordinates (x, y) are converted into a Gaussian heatmap, with these coordinates as the peak center, and the peak decays in a Gaussian distribution within the radius. The model predicts a heatmap with the same resolution as the input image, rather than directly regressing coordinate values; The final center coordinates are obtained by finding the location of the maximum pixel value in the predicted heatmap.

[0029] Among them, heatmap regression can learn the contextual information around key points and is more robust to occlusion, blurring and other conditions.

[0030] 3. Region partitioning branch: Output a pixel mask with the same size as the input image, where the pixel value of the insulator region is 1 and the pixel value of the background region is 0; We employ a joint optimization of Dice Loss and cross-entropy loss to address the imbalance between positive and negative samples.

[0031] S105. Design of Geometric Consistency Consistency Loss Function Establish geometric constraints between the model's output layers; the minimum circumcircle of the segmentation map should have some consistency with the predicted circle's center and radius. Design a consistency loss function. This ensures that the center of the circle predicted by the segmentation map is as close as possible to the center of the circle in the direct regression, while the regions in the segmentation map and the circular projection used in pose estimation are mutually constrained; specifically: Calculate the minimum circumcircle based on the mask data output from the split branches, and obtain its center coordinates. and radius ; The model predicts the center coordinates of the circle. ,radius coordinates of the circumcircle center ,radius Perform a consistency comparison.

[0032] Finally, the following composite loss function is designed:

[0033] in: This is a composite loss; Loss is estimated for attitude; For the regression loss of the heatmap centered on the circle; The cross-entropy and Dice loss of the segmented regions; This is a geometric consistency constraint term used to ensure the consistency of the three outputs in geometric space; where... , , and For the corresponding weights.

[0034] S106. Model Training and Optimization Phase Use the training set to perform end-to-end joint training of the model: The Adam optimizer was used, with an initial learning rate of 1×10⁻⁶. -4 And adopt a cosine annealing scheduling strategy; The combined loss on the validation set is used as the trigger condition for early stopping to prevent overfitting; Record the changes in metrics for the three task branches in each epoch: pose estimation error, circle center localization error, and segmentation IOU value; After training is complete, save the optimal model weight file for use in step S2.

[0035] Furthermore, although insulators vary in length and size, any substation or power plant uses only a few types of insulators. Therefore, a laser scanner is used to scan insulators in stationary, safe positions to obtain three-dimensional models of various types of insulators, forming a three-dimensional insulator model library. The specific process is as follows: A laser scanner was used to perform high-precision scanning of various types of insulators placed in a safe location. Point cloud data of various types of insulators are acquired, and after denoising, registration and surface reconstruction, a three-dimensional geometric model is formed. The 3D models are categorized and stored according to model number for later matching and retrieval.

[0036] S2. Acquire images of the insulator using a camera (e.g., ...) Figure 3 (As shown) and input into the insulator attitude recognition model for attitude recognition, to obtain the insulator's attitude Euler angles (Yaw, Pitch, Roll) and the coordinates of the insulator's bottom center in the camera coordinate system. The detected pixel mask for segmenting the insulator body has a height and width [h,w] that are consistent with the input image, such as... Figure 4 As shown; S3. Rotate the corresponding 3D insulator model from the insulator 3D model library according to the Euler angles in step S2, and then translate it to the camera coordinate system position. Location (coordinates of the center of the bottom circle of the insulator) Alignment), calculate the projection of the 3D model of the insulator onto the image; where Initialize to 0 or a maximum distance of 300 mm from the insulator; S4. Use the binary search method to traverse the depth of the insulator's 3D model ( The value is maximized when the ratio of the intersection to the union of the projection and the insulator segmentation map (pixel mask) on the image is at its maximum. The value is closest to the depth value of the real insulator in the camera coordinate system, and the output is a 3D model of the insulator after rotation and translation transformation. As the final insulator depth attitude model, the three-dimensional attitude information of the insulator is obtained, such as... Figure 5 As shown.

[0037] The projection image refers to the image obtained by projecting a 3D model onto a 2D image plane; the projection depends on the camera parameters and the depth of the model. Value); when changed When the value is changed, the size and position of the projection map will change.

[0038] The insulator segmentation image (mask data) is a binary image where a pixel value of 1 represents an insulator region and a pixel value of 0 represents a non-insulator region. The segmentation result comes from the image segmentation algorithm described above and marks the precise location of the insulators in the image.

[0039] The intersection refers to the set of pixels that are both equal to 1 in both images. In other words, it represents the region that is an insulator in both the projection image and the mask data.

[0040] The union refers to the set of pixels in two images where at least one pixel is 1. In other words, in the projection image or mask data, it represents the region of the insulator.

[0041] The detection method of this invention is based on the principle of camera pinhole imaging (the four-parameter model mentioned later). It presets three-dimensional models of various types of insulators by establishing a three-dimensional insulator model library. Then, it removes the depth information of the three-dimensional model by rotation, translation and scaling to reduce the dimension to a two-dimensional image. Then, it matches the insulator in the camera coordinate system. The matched modeling depth map replaces the scanning modeling of the current target insulator. The effect is similar, but it is faster and cheaper.

[0042] In step S2, the coordinates of the bottom center of the insulator in the camera coordinate system are obtained. The specific process is as follows: Because the camera coordinate system originates from the optical center, the difference between it and the robotic arm coordinate system originating from the end point is very small. In practice, this difference can be eliminated through calibration. Specifically, calibration ensures that plane a, formed by the x-axis and y-axis of the image coordinate system captured by the camera mounted on the robotic arm, is parallel to plane b, formed by the x-axis and y-axis of the lower part of the robotic arm coordinate system, and distortion correction is applied to the camera image. There is a spatial point in the camera coordinate system O-xyz. After passing through the optical center, the imaging point is obtained on the physical imaging plane. Then, the final pixel point is obtained by scaling and offsetting. ; The four-parameter model of the camera is as follows:

[0043] This formula describes how points in space... The image point is obtained on the imaging plane after passing through the optical center of the camera. Indicates the column number in the image coordinate system. Indicates the row number; , For image scaling parameters, , These imaging offset parameters, together, form the camera's intrinsic parameter matrix. It can be obtained through calibration; it can be eliminated through transformation. The value influences the following formula:

[0044]

[0045]

[0046] The coordinates of the bottom center of the insulator in the image are detected using an insulator attitude recognition model. This allows the robotic arm to move a small distance in both the horizontal and vertical directions. The coordinates of the bottom center of another insulator were obtained through a second test. Because the moving robotic arm changes the camera coordinate system, the true coordinates of the insulator after the movement are: The coordinates of the camera in the original camera coordinate system before the movement are: ; The center coordinates of the circles in the two images and To replace respectively and The following solution formula is obtained:

[0047] The result is obtained from the above two simultaneous equations. and This also reveals the actual center of the bottom circle of the insulator in the camera coordinate system. , Coordinate values, at this point only The value is unknown.

[0048] In addition, the Euler angles (Yaw, Pitch, Roll) and center coordinates of the insulators in the image are obtained through the insulator attitude recognition model. and the set of points in the segmented region { ,in They represent the first Each segmentation result pixel coordinates and coordinate; and These represent the width and height of the segmentation result, respectively; Let represent the set of all pixel coordinates of the insulator on the image, where and Indicates the width and height of the image; In practical applications, step S3 involves creating a three-dimensional model of the insulator. After Euler angle rotation and translation, the result is obtained from the above steps. Initialize in coordinates =300mm (a robotic arm manually controlled can easily achieve an accuracy within 3 meters), therefore, in the coordinate system of the camera mounted at the end of the robotic arm, the insulator... The coordinates must be within 3 meters; because Given that the values ​​are known, calculate the rotation and translation through Euler angles (Yaw, Pitch, Roll) to... The projection of the insulator model Mt onto the image, i.e., the set It is obtained through the four-parameter model of the camera; In practical applications, step S4 uses a binary search method to traverse the data. When the coordinates are [300, 150, 75, 37.5, 18.75, ..., 5.859], calculate the ratio of the intersection and union of the set U and the set of points in the segmented region. When this value is maximized, the insulator model in the current camera coordinate system can be considered to be obtained. The insulator depth and attitude model, which is similar to high-precision scanning modeling, obtains the digital three-dimensional attitude and depth information of the high-voltage insulator from the perspective of the robotic arm. The robotic arm can perform maintenance and cleaning operations based on this digital three-dimensional attitude and depth information, ensuring safe and reliable operation.

[0049] This invention can quickly and cost-effectively obtain digital 3D pose and depth information of high-voltage insulators from the perspective of a robotic arm, taking less than 3 seconds. It also boasts a fast and computationally efficient digital modeling process for insulator depth and pose, which can be accomplished using an industrial computer with edge computing capabilities. Furthermore, the overall cost is very low, as a common camera can be used to establish the depth and pose digital model. The corresponding laser scanner is only needed once when building the model library. Finally, this invention offers high accuracy, as the theoretical accuracy of the 3D model library built using a laser scanner is comparable to that of the laser scanner itself.

[0050] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, performs the steps of the method described above. This invention further provides an image-based digital three-dimensional attitude detection system for insulators, including a memory and a processor interconnected, wherein the memory stores a computer program that, when run by a processor, performs the steps of the method described above.

[0051] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium includes: any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0052] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for digital three-dimensional attitude detection of insulators based on images, characterized in that, Including the following steps: S1. Pre-construct an insulator attitude recognition model and an insulator 3D model library; the insulator 3D model library contains 3D models of various types of insulators. S2. Acquire the insulator image and input it into the insulator attitude recognition model for attitude recognition, obtaining the Euler angles of the target insulator's attitude and the coordinates of the insulator's bottom center in the camera coordinate system. And the pixel mask of the insulator body; S3. Rotate the preset 3D insulator model in the insulator 3D model library according to the Euler angles in step S2, and translate it relative to the coordinates of the bottom center of the insulator in the camera coordinate system. Align the insulator to obtain the projection of the 3D model of the insulator onto the insulator image; S4. Traverse the depth of the insulator's 3D model and obtain the ratio of the intersection and union of the pixel mask of the projection and the insulator body; when the ratio is at its maximum value, the corresponding 3D model of the insulator on the insulator image is taken as the insulator's depth attitude model, and thus the 3D attitude information of the insulator is obtained.

2. The image-based digital three-dimensional attitude detection method for insulators according to claim 1, characterized in that, The specific process of constructing the insulator attitude recognition model in step S1 is as follows: Collect images of insulators and perform attitude annotation, center annotation, and segmentation annotation on each image; A shared convolutional neural network is used as the backbone network to extract general features of the image; An attention module is embedded in the backbone network, and multi-scale features are fused through a skip connection structure. Three task branches are constructed at the end of the backbone network: including pose estimation branch, circle center localization branch, and segmentation branch; Construct a composite loss function by establishing geometric constraints between the outputs of the three task branches; End-to-end joint training was performed to obtain the insulator attitude recognition model.

3. The image-based digital three-dimensional attitude detection method for insulators according to claim 2, characterized in that, The center localization branch uses a heatmap regression method, which converts the marked center coordinates into a Gaussian heatmap as the learning target, predicts the heatmap, and obtains the center coordinates by finding the location of the maximum value.

4. The image-based digital three-dimensional attitude detection method for insulators according to claim 2, characterized in that, The attention module uses CBAM or SE to redistribute weights on the channel and spatial dimensions; the multi-scale feature fusion uses FPN or U-Net-style skip connection structures to fuse low-level high-resolution features with high-level semantic features.

5. The image-based digital three-dimensional attitude detection method for insulators according to claim 2, 3, or 4, characterized in that, The composite loss function is as follows: in: This is a composite loss; Loss is estimated for attitude; For the regression loss of the heatmap centered on the circle; The cross-entropy and Dice loss of the segmented regions; This is a geometric consistency constraint term used to ensure the consistency of the three outputs in geometric space; where... , , and For the corresponding weights.

6. The image-based digital three-dimensional attitude detection method for insulators according to any one of claims 1-4, characterized in that, In step S2, the pixel coordinates of the bottom center of the insulator are obtained. The specific process is as follows: Detect the coordinates of the bottom center of the insulator in the insulator image. ; Then move the camera a certain distance in both the horizontal and vertical directions. Acquire an insulator image and then detect the coordinates of the bottom center of the insulator circle on the insulator image again. The coordinates of the moved insulator in the camera coordinate system are: The coordinates of the insulator in the original camera coordinate system before the movement are then... ; Will and Substitute into the formula respectively In and The solution formula is obtained as follows: The coordinates of the bottom center of the insulator in the original camera coordinate system are obtained by solving the formula. ;in Indicates the column number in the image coordinate system. Indicates the row number; , For image scaling parameters, , This refers to the imaging offset parameter.

7. The image-based digital three-dimensional attitude detection method for insulators according to claim 6, characterized in that, Images of insulators are acquired using a camera located on a robotic arm; the robotic arm moves the camera a distance in both the horizontal and vertical directions.

8. The image-based digital three-dimensional attitude detection method for insulators according to claim 7, characterized in that, Beforehand, the plane formed by the x-axis and y-axis of the camera coordinate system formed by the installed camera is parallel to the plane formed by the x-axis and y-axis of the robot arm coordinate system, and distortion correction is performed on the camera image.

9. The image-based digital three-dimensional attitude detection method for insulators according to any one of claims 1-4, characterized in that, In step S1, an insulator in a stationary safe position is scanned using a laser scanner to establish a three-dimensional model library of insulators.

10. An image-based digital three-dimensional attitude detection system for insulators, comprising an interconnected memory and a processor, wherein the memory stores a computer program, characterized in that... The computer program, when run by a processor, performs the steps of the method as described in any one of claims 1-9.

Images