Pose estimation method for disordered grabbing of maintenance tools and instruments of hydropower station

Abstract

The invention discloses a pose estimation method for disordered grabbing of maintenance tools of a hydropower station, and the method comprises the steps: collecting multiple types of maintenance tool images, and constructing a maintenance tool image data set; performing target segmentation on the maintenance tool image data set through a first segmentation model to obtain a candidate mask set, extracting a target area based on the candidate mask set and executing standardization processing to generate a candidate image set; training and verifying the second classification model by using the candidate image set; and in combination with the depth image and the internal reference of the calibration camera, calculating the center-of-mass coordinate and the feature vector matrix of the maintenance tool to obtain the pose of the maintenance tool. According to the method, the problems that the implementation cost is high and the comprehensive requirements of an industrial-grade grabbing task on real-time performance, robustness and precision are difficult to meet due to the fact that a traditional segmentation and classification model depends on large-scale pixel-level labeling data in a tool disordered grabbing task in a hydropower station maintenance scene are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial automation, and in particular relates to a posture estimation method for disorderly grasping of maintenance tools for hydropower stations. Background Art

[0002] In the intelligent management scenario of maintenance tools at hydropower stations, the disordered stacking, multi-category mixing, and complex working conditions of tools pose severe challenges to pose estimation technology. Existing point cloud registration-based methods rely heavily on pre-stored high-precision three-dimensional templates. When faced with a variety of tools with significant morphological differences, a large model library needs to be built to achieve category coverage. However, in actual applications, the local point cloud loss caused by tool stacking leads to a high registration failure rate, making it difficult to meet the accuracy requirements in dynamic environments. Deep learning-based instance segmentation technology improves contour generalization through semantic understanding, but the pixel-level annotation model it relies on faces high costs in industrial scenarios. Massive images need to be annotated pixel by pixel, and the model's ability to recognize unlabeled tool variants is significantly reduced, making it unable to adapt to the diversity of tool morphologies in actual scenarios.

[0003] General-purpose large-scale vision models exhibit significant limitations in industrial scenarios. Segmentation masks lack boundary integrity on complex tools, and a lack of deep information fusion mechanisms results in pose solution errors that fail to meet industrial-grade accuracy standards. Furthermore, existing methods are inefficiently deployed on edge devices. The excessive reliance of high-precision models on computing resources conflicts with the precision loss of lightweight solutions, making it difficult to balance real-time and accuracy requirements. Even more critically, dynamic occlusion and material deformation between tools are not effectively modeled, and traditional methods have yet to overcome the problem of error accumulation during the grasping process. Furthermore, there is still a lack of reliable solutions to the pose ambiguity of symmetrical tools. Summary of the Invention

[0004] The present invention solves the problems of traditional segmentation and classification models in the disordered grasping of tools in hydropower station maintenance scenarios, such as high implementation cost due to reliance on large-scale pixel-level annotation data, and the need to consume a large amount of computing power resources for model training under complex background interference; the monocular vision method lacks depth information, resulting in insufficient accuracy in the three-dimensional pose estimation of tools, which makes mechanical grasping operations prone to pose deviations; and the incomplete target segmentation boundaries and high classification false detection rate caused by multi-scale size differences of tools, stacking occlusions and on-site environmental noise, which reduce the reliability of subsequent pose calculations and make it difficult to meet the comprehensive requirements of industrial-grade grasping tasks for real-time, robustness and accuracy.

[0005] The technical solution provided by the present invention is a posture estimation method for disordered grasping of maintenance tools in hydropower stations, comprising the following steps: Step 1: Collect images of multiple types of maintenance tools and construct a maintenance tool image dataset; Step 2: The maintenance tool image dataset from step 1 is segmented using the first segmentation model to obtain a segmentation result, i.e., a candidate mask set. Target regions are extracted based on the candidate mask set and normalized to generate a candidate image set. Step 2.1: Segment the maintenance tool image dataset in step 1 using the first segmentation model to generate a candidate mask set containing the outer contour of the target tool; Step 2.2: Calculate the minimum circumscribed ROI bounding box of the candidate mask; Step 2.3: Perform background normalization on the non-ROI area of the candidate mask; Step 2.4: Generate a candidate image set that matches the input data size.

[0006] The first segmentation model is a SAM segmentation model, which adopts an automatic edge segmentation mode. The input of the first segmentation model is a maintenance tool image dataset, and the output is a candidate mask set.

[0007] Furthermore, an empty image is created, the background color is set, and its size, type and input data are consistent. According to a single candidate mask area, the input image is copied to the empty image, which is recorded as ImageReduce.

[0008] Furthermore, the outer frame of the candidate mask outline is calculated and the outer frame is cropped by ImageReduce to complete the image cropping. The cropped image is recorded as ImageCrop.

[0009] Furthermore, a candidate image is created, and the background color is set, and the size is 224*224 pixels. The maximum value MaxSize of the width and height of ImageCrop is calculated, and ImageCrop is scaled. The scaled image is recorded as ImageResize.

[0010] Furthermore, the ImageResize data is copied to the candidate image and aligned according to the image center.

[0011] Step 3: Use the candidate image set obtained in step 2 to train and verify the second classification model; Furthermore, the second classification model is a ResNet50 classification model, the input is a candidate image set, the output is the category corresponding to the candidate image, and the segmentation result screening is performed on the candidate image.

[0012] Furthermore, the candidate image set is preprocessed, which includes manually marking the candidate images that completely cover the outer contour of the tool as positive examples, and dividing the candidate image set into a training set, a verification set and a test set in a ratio of 7:2:1.

[0013] The second classification model specifically includes: ‌Input layer, used to extract preliminary features;‌ ‌Residual block, a total of 4 residual blocks are stacked, each residual block is stacked by a Bottleneck structure;‌ ‌Global average pooling layer, used to convert feature maps into vectors;‌ ‌Fully connected layer‌, used to output classification results.

[0014] The second classification model introduces the Spatial Attention mechanism in the residual block. The calculation process of the prediction output with the attention mechanism is: Get the attention weight matrix, attention weight matrix The expression is: ; Where, is the input feature map, They represent the global maximum pooling and global average pooling of the channel dimension of the input feature map, To perform splicing in the channel dimension, The convolution kernel size is The convolution operation, is the Sigmoid activation function; The attention weight matrix acts on the original feature map and outputs a weighted feature map. The calculation formula of the weighted feature map is: ; Where, is the input feature map, is the attention weight matrix, is the weighted feature map.

[0015] Step 4: Generate candidate masks for the real-time captured maintenance tool images using the first segmentation model, and filter the segmentation results using the trained second classification model to obtain a target mask. Step 5: Based on the target mask obtained in step 4, combined with the depth image and the calibrated camera internal parameters, the center of mass coordinates and eigenvector matrix of the maintenance tool are calculated to obtain the posture of the maintenance tool.

[0016] Step 5.1: Based on the target mask obtained in step 4, combined with the depth image and the calibrated camera intrinsic parameters, calculate the point cloud data of the mask area; Furthermore, point cloud data is generated based on the target mask, depth image, and camera intrinsic parameters. The calculation formula for point cloud data is: ; ; ; Where, are the focal lengths of the camera in the x and y directions, are the optical center coordinates, is the coordinate of the point with gray value 255 in the mask, For the depth map The gray value at are the x, y, and z coordinates of the point cloud data respectively; Step 5.2: Calculate the centroid of the point cloud data in step 5.1; Calculate the centroid of the point cloud data. The formula for calculating the centroid is: ; Where, is the point cloud centroid, are the coordinates of the i-th point in the point cloud; Step 5.3: Calculate the covariance matrix of the point cloud data in step 5.1 and decompose the eigenvectors; Calculate the covariance matrix of the point cloud data. The calculation formula of the covariance matrix is: ; Where, is the data point of the point cloud, is the centroid of the point cloud; Perform eigenvalue decomposition on the covariance to obtain three orthogonal vectors corresponding to the maximum, medium, and minimum eigenvalues respectively. The expressions of the three orthogonal vectors are: ; Where, is the eigenvalue focus matrix, is the eigenvector matrix; Step 5.4: Combine the centroid of step 5.2 with the eigenvector of step 5.3 to output the pose matrix. The expression of the recognized object pose matrix is: ; Where, Represents the eigenvector matrix The element in the i-th column and j-th row of They represent the x-coordinate, y-coordinate, and z-coordinate of the center of mass, respectively.

[0017] Furthermore, the depth image and the calibration camera intrinsic parameters are acquired by a binocular structured light 3D camera.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention significantly reduces data annotation costs and model training overhead while ensuring target recognition accuracy through a cascaded architecture of segmentation and classification models. The automatic edge segmentation capability of the SAM model is used to generate candidate masks, avoiding the large-scale pixel-level annotation required by the fully supervised segmentation model. At the same time, based on a small number of candidate image samples, an improved ResNet50 network with a spatial attention mechanism is used to achieve efficient classification and screening. Model training can be completed with only over 10,000 images, effectively overcoming the bottleneck problems of traditional methods, such as strong dependence on labeled data and high computing power consumption.

[0019] (2) This invention improves the robustness and computational efficiency of spatial positioning of tools in complex scenarios by integrating depth information with covariance eigendecomposition. By inverting target point cloud data using depth images and camera intrinsic parameters, and combining covariance matrix eigendecomposition to obtain the main direction orthogonal vectors, it can accurately analyze the three-dimensional center of mass position and attitude direction of tools in a disordered stacked state. This solves the pose estimation error problem caused by the lack of depth information in traditional monocular vision methods, and meets the dual requirements of real-time and precision in industrial grasping operations.

[0020] (3) This invention enhances the model's generalization ability for multi-scale targets and complex background interference through an image preprocessing strategy based on background normalization and adaptive scaling. By performing ROI cropping and background normalization on the candidate mask area, combined with an image size alignment mechanism with adaptive scaling, the interference of background noise on the classification model is effectively eliminated, ensuring the scale consistency of tools of different sizes during the feature extraction process, and providing high-confidence target area input for subsequent pose calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below with reference to the accompanying drawings and examples.

[0022] Figure 1 Flowchart of a method according to an embodiment of the present invention.

[0023] Figure 2 This is a network structure diagram of the improved ResNet50 model according to an embodiment of the present invention.

[0024] Figure 3 This is a segmentation effect diagram of an embodiment of the present invention.

[0025] Figure 4 This is a diagram showing the pose estimation effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] like Figure 1 As shown, a pose estimation method for disordered grasping of maintenance tools in hydropower stations includes the following steps: Step 1: Collect images of multiple types of maintenance tools and construct a maintenance tool image dataset; Step 2: The maintenance tool image dataset from step 1 is segmented using the first segmentation model to obtain a segmentation result, i.e., a candidate mask set. Target regions are extracted based on the candidate mask set and normalized to generate a candidate image set. Step 2.1: Segment the maintenance tool image dataset in step 1 using the first segmentation model to generate a candidate mask set containing the outer contour of the target tool; Step 2.2: Calculate the minimum circumscribed ROI bounding box of the candidate mask; Step 2.3: Perform background normalization on the non-ROI area of the candidate mask; Step 2.4: Generate a candidate image set that matches the input data size.

[0027] The first segmentation model is constructed using the SAM segmentation model. In automatic edge segmentation mode, all edges in the maintenance tool image are segmented to obtain a segmentation result containing multiple candidate masks. The candidate masks are used to perform ROI cropping on the image. Background filling and scaling are used to generate candidate images of the same size and retaining the original proportions of the maintenance tool. Since segmentation using the prompt information mode in the SAM model still consumes a lot of computing resources and high costs for training the target detection model, the automatic edge segmentation mode is adopted in this embodiment.

[0028] After the maintenance tool image dataset is input into the SAM model, it will return a set of candidate masks with the same size as the input image.

[0029] Use a mask to crop the image ROI, and use background filling and scaling to generate candidate images with the same size and retaining the original proportions of the maintenance tool; In the automatic edge segmentation mode, the edges in the image will be segmented, and the segmentation results will include fine-grained segmentation results. Therefore, in this embodiment, a post-processing method is designed to eliminate the fine-grained segmentation results and retain only the segmentation results of the outer contour. The post-processing method mainly generates candidate images through a series of image processing methods, and then screens the candidate images through a classification model. In this way, complex instance segmentation or target detection tasks can be converted into simpler binary classification tasks, greatly reducing the annotation complexity and computing power requirements.

[0030] Create an empty image with the background color set to (128, 128, 128). Its size and type are consistent with the input image. Then, based on a single candidate mask region, copy the input image to the empty image, which is called ImageReduce.

[0031] By calculating the outer border of the mask contour and cropping the image using ImageReduce, the image is cropped and the cropped image is recorded as ImageCrop.

[0032] Create a candidate image with the background color set to (128, 128, 128) and a size of 224*224 pixels. Then calculate the maximum value MaxSize of the width and height of ImageCrop and scale ImageCrop. The scaled image is recorded as ImageResize.

[0033] Copy the ImageResize data to the candidate image and align it according to the image center.

[0034] The segmentation effect is as follows Figure 3 shown.

[0035] Step 3: Use the candidate image set obtained in step 2 to train and verify the second classification model; The second classification model is the ResNet50 classification model. The input is the candidate image set, the output is the category corresponding to the candidate image, and the segmentation result screening is performed on the candidate image.

[0036] The candidate image set is preprocessed, which includes manually marking the candidate images that completely cover the outer contour of the tool as positive examples, and dividing the candidate image set into training set, verification set and test set in a ratio of 7:2:1.

[0037] The second classification model specifically includes: ‌Input layer, the input image first passes through a convolutional layer to extract preliminary features;‌ ‌Residual block, the main component of ResNet50, the core function is to solve the gradient vanishing problem in deep neural networks and improve network performance. In the ResNet50 model, there are 4 residual blocks stacked: conv2_x, conv3_x, conv4_x, and conv5_x. Each residual network block is composed of a stacked Bottleneck structure;‌ ‌Global average pooling layer, after all residual blocks, is a global average pooling layer used to convert feature maps into vectors;‌ ‌Fully connected layer‌, the last one is a fully connected layer to output the classification results; The SAM model automatically segments the image edges. Therefore, the features of positive and negative examples will overlap, and the classification model will suffer performance degradation due to feature confusion. Based on the ResNet50 model, the Spatial Attention mechanism is introduced. Figure 2As shown in the figure, the Spatial Attention module is added after the conv2_x, conv3_x, conv4_x, and conv5_x modules respectively.

[0038] In the Spatial Attention module, the input feature map is first subjected to global maximum pooling and global average pooling in the channel dimension, and then the results are spliced according to the channel. The spliced results are then convolved, and finally the convolution operation is input into the Sigmoid activation function to obtain the attention weight matrix.

[0039] The second classification model introduces the Spatial Attention mechanism in the residual block. The calculation process of the prediction output with the attention mechanism is: Get the attention weight matrix, attention weight matrix The expression is: ; Where, is the input feature map, They represent the global maximum pooling and global average pooling of the channel dimension of the input feature map, To perform splicing in the channel dimension, The convolution kernel size is The convolution operation, is the Sigmoid activation function; The attention weight matrix acts on the original feature map and outputs a weighted feature map. The calculation formula of the weighted feature map is: ; Where, is the input feature map, is the attention weight matrix, is the weighted feature map.

[0040] After training, a comparative experiment was conducted on the test set to compare the ResNet50 model before and after the improvement. The experimental results are shown in Table 1:

[0041] From the experimental results, it can be seen that after the ResNet50 model is improved by the method described in the present invention, the model performance is significantly improved.

[0042] In this embodiment, on a computer equipped with an NVIDIA GeForce RTX 3070 GPU, only 10,880 images were used to complete the classification model training, greatly reducing the labeling cost and computing resource requirements.

[0043] Step 4: Generate candidate masks for the real-time captured maintenance tool images using the first segmentation model, and filter the segmentation results using the trained second classification model to obtain a target mask. Step 5: Based on the target mask obtained in step 4, combined with the depth image and camera intrinsic parameters, the center of mass coordinates and eigenvector matrix of the maintenance tool are calculated to obtain the posture of the maintenance tool.

[0044] Step 5.1: Based on the target mask obtained in step 4, combined with the depth image and the calibrated camera intrinsic parameters, calculate the point cloud data of the mask area; Generate point cloud data based on the target mask, depth image and camera intrinsic parameters. The calculation formula of point cloud data is: ; ; ; Where, are the focal lengths of the camera in the x and y directions, are the optical center coordinates, is the coordinate of the point with gray value 255 in the mask, For the depth map The gray value at are the x, y, and z coordinates of the point cloud data respectively; Step 5.2: Calculate the centroid of the point cloud data in step 5.1; Calculate the centroid of the point cloud data. The formula for calculating the centroid is: ; Where, is the point cloud centroid, are the coordinates of the i-th point in the point cloud; Step 5.3: Calculate the covariance matrix of the point cloud data in step 5.1 and decompose the eigenvectors; Calculate the covariance matrix of the point cloud data. The calculation formula of the covariance matrix is: ; Where, is the data point of the point cloud, is the centroid of the point cloud; Perform eigenvalue decomposition on the covariance to obtain three orthogonal vectors corresponding to the maximum, medium, and minimum eigenvalues respectively. The expressions of the three orthogonal vectors are: ; Where, is the eigenvalue focus matrix, is the eigenvector matrix; Step 5.4: Combine the centroid from step 5.2 with the eigenvector from step 5.3 and output the pose matrix.

[0045] The expression of the recognized object pose matrix is: ; Where, Represents the eigenvector matrix The element in the i-th column and j-th row of They represent the x-coordinate, y-coordinate, and z-coordinate of the center of mass, respectively.

[0046] The depth image and the calibrated camera intrinsic parameters are obtained by a binocular structured light 3D camera.

[0047] The pose calculation results are as follows Figure 4 shown.

Claims

1. A pose estimation method for disordered grasping of maintenance tools in hydropower stations, characterized by: The following steps are involved: Step 1: Collect images of multiple types of maintenance tools and construct a maintenance tool image dataset; Step 2: The maintenance tool image dataset from step 1 is segmented using the first segmentation model to obtain a segmentation result, i.e., a candidate mask set. Target regions are extracted based on the candidate mask set and normalized to generate a candidate image set. Step 3: Use the candidate image set obtained in step 2 to train and verify the second classification model; Step 4: Generate candidate masks for the real-time captured maintenance tool images using the first segmentation model, and filter the segmentation results using the trained second classification model to obtain a target mask. Step 5: Based on the target mask obtained in step 4, combined with the depth image and the calibrated camera internal parameters, the center of mass coordinates and eigenvector matrix of the maintenance tool are calculated to obtain the posture of the maintenance tool.

2. A method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 1, characterized in that: In step 2, the first segmentation model is a SAM segmentation model, which adopts an automatic edge segmentation mode. The input of the first segmentation model is a maintenance tool image dataset, and the output is a candidate mask set.

3. The method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 2 is characterized in that: The step 2 includes the following sub-steps: Step 2.1: Segment the maintenance tool image dataset in step 1 using the first segmentation model to generate a candidate mask set containing the outer contour of the target tool; Step 2.2: Calculate the minimum ROI bounding box of the candidate mask; Step 2.3: Perform background normalization on the non-ROI area of the candidate mask; Step 2.4: Generate a set of candidate images that match the input data size.

4. The method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 3 is characterized in that: In step 3, the second classification model is a ResNet50 classification model, the input is a candidate image set, the output is the category corresponding to the candidate image, and the segmentation result screening is performed on the candidate image.

5. The method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 4 is characterized in that: In step 3, the candidate image set is preprocessed. The preprocessing includes manually marking the candidate images that completely cover the outer contour of the tool as positive examples, and dividing the candidate image set into a training set, a validation set, and a test set.

6. A method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 5, characterized in that: In step 3, the second classification model specifically includes: ‌Input layer, used to extract preliminary features;‌ ‌Residual block, a total of 4 residual blocks are stacked, each residual block is stacked by a Bottleneck structure;‌ ‌Global average pooling layer, used to convert feature maps into vectors;‌ ‌Fully connected layer‌, used to output classification results.

7. The method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 6, characterized in that: In step 3, the second classification model introduces the Spatial Attention mechanism in the residual block. The calculation process of the prediction output with the attention mechanism is: Get the attention weight matrix, attention weight matrix The expression is: ； Where, is the input feature map, They represent the global maximum pooling and global average pooling of the channel dimension of the input feature map, To perform splicing in the channel dimension, The convolution kernel size is The convolution operation, is the Sigmoid activation function; The attention weight matrix acts on the original feature map and outputs a weighted feature map. The calculation formula of the weighted feature map is: ； Where, is the input feature map, is the attention weight matrix, is the weighted feature map.

8. The method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 7, characterized in that: The step 5 includes the following sub-steps: Step 5.1: Based on the target mask obtained in step 4, combined with the depth image and the calibrated camera intrinsic parameters, calculate the point cloud data of the mask area; Step 5.2: Calculate the centroid of the point cloud data in step 5.1; Step 5.3: Calculate the covariance matrix of the point cloud data in step 5.1 and decompose the eigenvectors; Step 5.4: Combine the centroid from step 5.2 with the eigenvector from step 5.3 and output the pose matrix.

9. The method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 8, characterized in that: In step 5, the depth image and the camera intrinsic parameters are acquired by a binocular structured light 3D camera.

10. A method for posture estimation for disordered grasping of maintenance tools for hydropower stations according to claim 9, characterized in that: In step 5, point cloud data is generated based on the target mask, depth image, and camera intrinsic parameters. The calculation formula for point cloud data is: ； ； ； Where, are the focal lengths of the camera in the x and y directions, are the optical center coordinates, is the coordinate of the point with gray value 255 in the mask, For the depth map The gray value at are the x, y, and z coordinates of the point cloud data respectively; Calculate the centroid of the point cloud data. The formula for calculating the centroid is: ； Where, is the point cloud centroid, are the coordinates of the i-th point in the point cloud; Calculate the covariance matrix of the point cloud data. The calculation formula of the covariance matrix is: ； Where, is the data point of the point cloud, is the centroid of the point cloud; Perform eigenvalue decomposition on the covariance to obtain three orthogonal vectors corresponding to the maximum, medium, and minimum eigenvalues respectively. The expressions of the three orthogonal vectors are: ； Where, is the eigenvalue focus matrix, is the eigenvector matrix; The expression of the recognized object pose matrix is: ； Where, Represents the eigenvector matrix The element in the i-th column and j-th row of They represent the x-coordinate, y-coordinate, and z-coordinate of the center of mass, respectively.

Images