Improved pixel voting-based billet end face 6d pose estimation method and system

Abstract

The application discloses a billet end face 6D pose estimation method and system based on improved pixel voting, and belongs to the technical field of industrial machine vision and automation. The method collects a billet end face RGB image through a depth camera and performs enhancement preprocessing; 3D key points are obtained by using farthest point sampling of a three-dimensional model; the image is input into an improved pixel voting network, and a semantic segmentation mask and a scale invariant unit vector field are synchronously output; 2D key point coordinates and a covariance matrix thereof are generated based on foreground pixel voting and random sample consensus; after 2D-3D matching point pairs are established, the covariance matrix is combined to execute an uncertainty-driven perspective n-point algorithm, an initial pose is obtained by minimizing Mahalanobis distance; the initial pose and deep fusion features extracted by the network are input into a pose predictor, and an optimal pose is output. The corresponding system comprises a memory and a processor. The method can obtain a high-precision billet end face 6D pose under weak texture and variable scale conditions.

Description

Technical Field

[0001] This invention relates to the field of industrial machine vision and automation technology, specifically to a method and system for 6D pose estimation of billet end face based on improved pixel voting. Background Technology

[0002] Accurate estimation of the 6D pose of the billet end face is a crucial prerequisite for automated billet clamping, cutting, and grinding. Traditional pose estimation relies on local feature point detection and geometric matching in the image. However, after flame cutting, the billet end face is oxidized and blackened, with highly repetitive textures and a lack of obvious corner points. Furthermore, uneven lighting in industrial environments makes it difficult for conventional feature extractors to obtain stable and repeatable descriptors, leading to frequent failures in feature-matching-based pose estimation. Deep learning-based direct pose regression methods have limited generalization ability for weakly textured objects like billets, failing to meet the accuracy requirements under varying pose conditions. Pixel voting networks predict the direction vector of each pixel in the image pointing to key points of the object, and then locate the two-dimensional key points through voting, providing an effective solution for pose estimation of weakly textured targets.

[0003] However, the unit direction vector output by such methods is sensitive to changes in the target scale. When the working distance between the camera and the billet end face fluctuates, the spatial distribution of voting intersections tends to disperse, causing significant shifts in 2D keypoints. Existing post-voting processing workflows typically only provide maximum likelihood estimates for keypoints, without measuring the uncertainty of each keypoint due to occlusion or different voting densities. Subsequent perspective n-point pose calculations assign equal weights to all keypoints, making them susceptible to interference from points with high uncertainty, resulting in significant jitter in the pose results. Furthermore, using a single perspective n-point solution as the final pose fails to fully exploit the ability of deep features to correct pose residuals.

[0004] Therefore, when relying solely on a single frame of RGB image, it is necessary to address the scale invariance of pixel voting on the billet end face, as well as the difficulty in modeling and effectively utilizing the uncertainty of key points. Summary of the Invention

[0005] This invention proposes a 6D pose estimation method and system for billet end face based on improved pixel voting. The purpose is to solve the defects of existing pixel voting methods, such as the sensitivity of voting vectors to target scale and the neglect of key point uncertainty in pose calculation. The method aims to obtain accurate and robust six-degree-of-freedom pose in scenarios with weak texture, multiple scales and varying illumination on the billet end face.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a 6D pose estimation method for the end face of a steel billet based on improved pixel voting, comprising the following steps:

[0007] RGB images of the billet end face are acquired using a depth camera; Gaussian filtering and histogram equalization are applied to the RGB images to enhance image contrast and suppress noise; based on the three-dimensional model of the billet end face, 3D key points distributed on the model surface are obtained by sampling from the farthest point, so that the key points uniformly cover the end face area, ensuring the comprehensiveness of subsequent pose calculation.

[0008] The preprocessed RGB image is input into an improved pixel voting network, which simultaneously outputs a semantic segmentation mask for the billet end face and a scale-invariant unit vector field. This network includes a shared feature encoder, a semantic segmentation decoder, and a vector field regression decoder. The shared feature encoder extracts multi-scale feature maps from the image, the semantic segmentation decoder generates a semantic segmentation mask based on the multi-scale feature maps to accurately separate the foreground region, and the vector field regression decoder predicts unit vectors pointing to each 3D key point pixel by pixel. The unit vectors are then normalized by a scale-adaptive normalization layer to output a scale-invariant unit vector field, thereby adapting to the scale changes that occur on the billet end face in the image and improving the consistency and robustness of the vector field prediction.

[0009] Based on semantic segmentation mask and scale-invariant unit vector field, foreground pixels are screened and voted on. For each 3D keypoint, the voting position of the foreground pixel is calculated according to the unit vector field and aggregated to generate a 2D candidate point set. A two-dimensional Gaussian distribution is fitted with random sampling consistency. Each candidate hypothesis is voted and scored. The cluster mean with the highest score is used as the coordinate of the 2D keypoint. The covariance matrix is ​​calculated based on the voting in-points. Preferably, the sample covariance of the voting in-point set is calculated and a preset isotropic noise covariance matrix is ​​added for fusion to obtain a covariance matrix that represents the positioning uncertainty, so that the two-dimensional keypoint positioning information has a credibility measure.

[0010] Based on the camera intrinsic parameters, a one-to-one correspondence is established between the 2D key points and the 3D key points to form 2D-3D matching point pairs;

[0011] The algorithm uses the uncertainty of the input of matching point pairs and covariance matrix to drive the perspective n-point algorithm. Based on the covariance matrix, a Mahalanobis distance-weighted reprojection error function is constructed. By iteratively solving the nonlinear least squares problem, the Mahalanobis distance is minimized to solve the pose and output the initial pose. This process can use covariance information to suppress the influence of high uncertainty point pairs and improve the accuracy and noise resistance of the initial pose solution.

[0012] The initial pose and the deep fusion features output by the improved pixel voting network are input into the pose predictor to obtain the optimal pose. Preferably, the deep fusion features are obtained by concatenating and fusioning the high-level semantic feature map and the low-level texture feature map extracted by the shared feature encoder channel by channel, which takes into account both semantic information and detailed texture information. The pose predictor includes a residual network and an iterative optimizer. The residual network takes the concatenation of the rendering features corresponding to the initial pose and the deep fusion features as input and outputs the pose residual. The iterative optimizer repeatedly applies the pose residual to correct the initial pose and generate the optimal pose, realizing pose optimization from coarse to fine.

[0013] As a technical solution of the present invention, the improved pixel voting network is trained by a multi-task loss function, which includes the cross-entropy loss of semantic segmentation and the smoothing L1 loss of vector field regression. In the vector field regression loss, a constraint term for the scale adaptive normalization layer parameters is introduced to ensure network training stability and maintain the consistency of vector field scale, thereby further improving the reliability of overall pose estimation.

[0014] This invention also provides a 6D pose estimation system for billet end faces based on improved pixel voting, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the aforementioned 6D pose estimation method for billet end faces based on improved pixel voting. This system can be deployed in industrial settings, enabling real-time, high-precision 6D pose estimation of billet end faces, providing crucial pose information for automated hoisting or sorting, and enhancing system robustness under complex lighting and dimensional conditions.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0016] An improved pixel voting network incorporates a scale-adaptive normalization layer at the end of the vector field regression decoder. This layer applies scale-invariant constraints to the keypoint direction vectors predicted by each pixel, ensuring that the learned unit vector field maintains consistent pointing response characteristics across different shooting distances on the billet end face. Foreground pixel voting based on this scale-invariant unit vector field significantly improves the clustering of candidate hypothesis points and drastically reduces the localization error of 2D keypoint coordinates. This alleviates the voting divergence problem caused by target scale variations and ensures repeatability of keypoint detection under multi-scale conditions. While generating 2D keypoint coordinates through random sampling consensus voting, the covariance of samples is calculated using the inlier set and isotropic noise is fused to output a covariance matrix reflecting localization uncertainty. This covariance matrix, along with the matching point pairs, is synchronously input into an uncertainty-driven perspective n-point algorithm. A reprojection error function metricd by Mahalanobis distance is constructed. During iterative optimization, higher uncertainty keypoints are adaptively assigned lower contribution weights, while lower uncertainty keypoints dominate pose estimation. This effectively suppresses interference from local occlusion or vector field prediction noise on pose estimation, resulting in a stable initial pose in rough billet end face scenes. The deep fusion feature formed by concatenating and convolving the high-level semantic features and low-level texture features extracted by the shared feature encoder, together with the model rendering features under the initial pose, is fed into the pose predictor. The pose residual is predicted by the residual network and iteratively corrected. The contour and texture information are finely aligned by cross-modal feature complementarity to eliminate the residual bias of coarse pose, so that the final pose can still stably converge to the high-precision optimal solution under the conditions of sparse texture on the steel billet end face and changes in illumination. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of a 6D pose estimation method for the end face of a steel billet based on improved pixel voting.

[0019] Figure 2 This is a flowchart of the sampling process for the farthest point of the 3D model of the billet end face;

[0020] Figure 3 This is a statistical chart of the Euclidean distance distribution of the farthest points of the 3D key points on the end face of the steel billet;

[0021] Figure 4 This is a schematic diagram of the inner and outer point sets and confidence ellipse of 2D key point voting based on random sampling consistency.

[0022] Figure 5It is a comparison curve of iterative pose estimation error based on the reprojection error of Mahalanobis distance weighted and Euclidean distance;

[0023] Figure 6 It is the convergence curve of each loss function during the training process of the improved pixel voting network. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] See Figure 1 This invention provides a 6D pose estimation method for the end face of a steel billet based on improved pixel voting, comprising the following steps:

[0026] RGB images of the billet end face are acquired using a depth camera; Gaussian filtering and histogram equalization are applied to the RGB images; 3D key points distributed on the model surface are obtained by sampling from the farthest point based on the 3D model of the billet end face; the preprocessed RGB images are input into an improved pixel voting network, which simultaneously outputs a semantic segmentation mask and a scale-invariant unit vector field for the billet end face; foreground pixels are selected and voted on based on the semantic segmentation mask and the scale-invariant unit vector field, and 2D candidate hypotheses for each of the 3D key points are generated using random sampling consistency; the spatial distribution of the voting scores is estimated to obtain the coordinates and covariance matrix of the 2D key points; a one-to-one correspondence is established between the 2D key points and the 3D key points according to the camera intrinsic parameters, forming 2D-3D matching point pairs; the matching point pairs and the covariance matrix are input into an uncertainty-driven perspective n-point algorithm to minimize the Mahalanobis distance to calculate the pose and output the initial pose; the initial pose and the deep fusion features output by the improved pixel voting network are input into a pose predictor to obtain the optimal pose.

[0027] Example 1

[0028] In its implementation, the improved pixel voting network consists of three parts: a shared feature encoder, a semantic segmentation decoder, and a vector field regression decoder. The shared feature encoder receives an RGB image preprocessed with Gaussian filtering and histogram equalization as input and outputs a set of multi-scale feature maps with different spatial resolutions. The backbone of the shared feature encoder adopts a deep residual network structure, containing an initial convolutional layer and four sequentially connected residual stages. Each residual stage consists of several stacked residual blocks, where each residual block is composed of two 3×3 convolutional layers and skip connections. In the four residual stages, the feature maps output by the last three residual stages together form the multi-scale feature map, which is denoted as [the three feature maps with different resolutions are denoted as and respectively]. , and ,in This is the lower-level feature map output from the first residual stage. This is the intermediate-level feature map output from the second residual stage. This is a higher-level feature map output from the third residual stage.

[0029] The semantic segmentation decoder receives multi-scale feature maps. , and The feature fusion process is performed from top to bottom to gradually restore spatial resolution, ultimately generating a semantic segmentation mask for the billet end face. The semantic segmentation decoder contains three upsampling modules. The highest-level feature map... First, the feature map is obtained by adjusting the number of channels through a 1×1 convolutional layer. ; The feature map is upsampled by 2x using bilinear interpolation and then compared with the feature map after adjusting the number of channels using a 1×1 convolution. Element-wise addition is performed, followed by smooth fusion through a 3×3 convolutional layer to obtain the feature map. ; Similarly, a 2x upsampling is performed using bilinear interpolation, and the feature map is compared with the feature map after adjusting the number of channels using a 1×1 convolution. Element-wise addition is performed, followed by smooth fusion through a 3×3 convolutional layer to obtain the feature map. Finally, for A 4x bilinear interpolation upsampling is performed to restore the image to the same spatial size as the preprocessed RGB image. A single-channel semantic segmentation mask for the billet end face is then output through a 1×1 convolutional layer and a Sigmoid activation function. Each pixel value in the mask represents the probability that the corresponding pixel belongs to the foreground of the billet end face.

[0030] The vector field regression decoder also receives multi-scale feature maps. , and Its structural layout is similar to that of the semantic segmentation decoder, also containing three upsampling modules, but the final number of output channels differs. The vector field regression decoder employs the exact same feature fusion method as the semantic segmentation decoder. to After progressively fusing upsampling to obtain a feature representation of the same size as the input image, it is fed into a 1×1 convolutional layer to output a feature map with 2K channels, where K represents the number of 3D keypoints pre-determined through farthest point sampling. Each spatial location in the output feature map corresponds to a 2K-dimensional vector, which is composed of K two-dimensional vectors concatenated sequentially. Each two-dimensional vector represents the unnormalized direction prediction from the current pixel to the projection position of a certain 3D keypoint on the two-dimensional image plane.

[0031] The 2K-dimensional feature map output by the vector field regression decoder is then fed into a scale-adaptive normalization layer. This layer performs normalization and scale scaling on each of the K two-dimensional vectors corresponding to each pixel location, outputting a scale-invariant unit vector field. Taking the k-th keypoint as an example, for the pixel location... Two-dimensional prediction vector at the location The scale-adaptive normalization layer calculates its corresponding unit vector as follows: :

[0032]

[0033] in, Represents pixels The unnormalized two-dimensional prediction vector corresponding to the k-th 3D keypoint; Representing vectors The L2 norm; It is a small positive number preset to avoid division by zero, and its value is... ; This is a scale-adaptive factor specifically for predicting the orientation of the k-th keypoint. It is a learnable scalar parameter that is automatically updated via backpropagation during the training of the improved pixel voting network, with an initial value set to 1.0. Scale Adaptive Factor The introduction of this feature enables the network to adjust the vector magnitude range according to the different keypoint locations, thereby alleviating the training instability caused by the difference in regression difficulty between different keypoints. After processing by the scale-adaptive normalization layer, the orientation predictions of all keypoints at all pixel locations are constrained to unit vectors with controllable scale and accurate orientation information, forming a scale-invariant unit vector field for subsequent foreground pixel voting.

[0034] Example 2

[0035] In specific implementation, please refer to Figure 2Based on the 3D model of the billet end face, 3D key points distributed on the model surface were obtained by sampling from the farthest point. The 3D model of the billet end face originated from the computer-aided design model of the billet, which was discretized and converted into a dense point cloud. The conversion process was as follows: points were collected on the surface of the 3D model of the billet end face using a uniform sampling method, with the sampling resolution set to 1 point per square millimeter, so that the point cloud fully covered the contour and internal area of ​​the billet end face. The generated point cloud set is denoted as […]. Each point is represented by its three-dimensional coordinates. express.

[0036] Set the number of sampling points K, The value is determined based on the pose estimation accuracy requirements predicted by the geometric features of the billet end face. Set as The reason is that the end face of the steel billet has an approximately rectangular shape. These key points are distributed across the four corner regions and the midpoint regions of the four sides, thus fully representing the two-dimensional projection shape of the end face and providing spatial constraints for subsequent 2D-3D matching. Selected point set It is initially an empty set.

[0037] Farthest point sampling is performed iteratively. In the first iteration, sampling is performed from the point cloud set. Randomly select a point As the initial point, Add to the selected point set ,at this time .

[0038] In each subsequent iteration, the set of unselected points is calculated. From each point in the set of selected points The distance is defined as the distance between the currently unselected point and the set of selected points. The minimum Euclidean distance between all points. For a point in the set of unselected points. Its arrival at the selected point set distance The calculation method is as follows:

[0039]

[0040] in, Represents the set of currently unselected points. One of the points, This represents the current set of selected points. Indicates the selected point set Any point in, Point With point Between Norm, which is the Euclidean distance between two points.

[0041] After calculating the distance value for each point in the set of unselected points, find the point with the maximum distance value. ,Will Add to the selected point set ,renew .

[0042] Repeat the above iterative process until the selected point set is reached. The number of points included reaches the preset value. Finally, the selected point set In These points are 3D key points distributed on the surface of the billet end face model. These 3D key points have a large spatial dispersion and can approximately cover the main geometric features of the model surface with a small number of points.

[0043] See Figure 3 The figure shows a histogram of Euclidean distance distribution from each point in the point cloud of the 3D model of the billet end face, obtained based on the farthest point sampling method in Example 2, to its nearest 3D key point. The horizontal axis represents the Euclidean distance from a point in the point cloud to its nearest 3D key point, in millimeters, ranging from approximately 5 mm to 95 mm; the vertical axis represents the frequency of points within the corresponding distance range. The legend "Euclidean Distance Distribution" indicates that the statistics represent the spatial distance characteristics between point cloud points and key points.

[0044] As shown in the figure, the distances are concentrated in the range of approximately 35 mm to 70 mm, with frequencies fluctuating between 40 and 76. The frequency peaks in the range of 45 mm to 55 mm, reaching a maximum of approximately 76, indicating a relatively high and dense point cloud within this range. The number of points less than 20 mm is relatively small, suggesting fewer point cloud points near keypoints, consistent with the design intention of maintaining a large spatial dispersion between keypoints. The number of points greater than 75 mm gradually decreases, reflecting that the keypoints have well covered the end face model surface, with very few points being far from the keypoints.

[0045] The overall curve shows a single-peak distribution that rises first and then falls, indicating that the eight selected key points achieve uniform coverage of the point cloud of the billet end face model through sampling at the farthest point. The spacing between the key points is reasonable, which not only ensures spatial dispersion but also effectively characterizes the main geometric features of the billet end face, meeting the accuracy requirements of subsequent 2D-3D matching and pose estimation.

[0046] Example 3

[0047] In practice, the process of selecting and voting for foreground pixels based on semantic segmentation masks and scale-invariant unit vector fields is performed pixel-by-pixel. The foreground region is extracted from the semantic segmentation mask of the billet end face, and pixels with values ​​greater than 0.5 in the mask are identified as foreground pixels. All foreground pixels constitute the foreground pixel set. For each 3D keypoint obtained through sampling from the farthest point, voting is performed using the corresponding layer of the unit vector map in the scale-invariant unit vector field.

[0048] With the first Taking the first 3D key point as an example, the first The 3D keypoint corresponds to the scale-invariant unit vector field in the 3D keypoint. Each channel group contains two channels, representing the unit vector components along the horizontal and vertical axes of the image, respectively. For a given position in the foreground pixel set... The pixel is read from the scale-invariant unit vector field, corresponding to the pixel at position i.e., the pixel at position j. Two-dimensional unit vector of 3D key points Voting location It is obtained by adding the foreground pixel position to a two-dimensional unit vector, i.e. , . Target all foreground pixels for the first The voting positions generated by the 3D key points are aggregated to form the first... 2D candidate point set of 3D key points 2D candidate point set Each point in the graph represents an independent vote from a foreground pixel, expressing that pixel's opinion on the first pixel. Prediction of the projection position of 3D key points onto a 2D image plane.

[0049] For 2D candidate point set A two-dimensional Gaussian distribution is fitted using random sampling consensus. The number of random sampling consensus iterations is set to 200. In each iteration, from the 2D candidate point set... Two points are randomly selected from the dataset. A candidate Gaussian distribution's mean vector and covariance matrix are calculated based on these two points. The mean vector is the average of the coordinates of the two points, and the covariance matrix is ​​obtained by calculating the sample covariance from these two points. The Mahalanobis distance is set to a threshold value. As an interior point determination criterion, the threshold The threshold is set to 9.21, based on the critical value of the chi-square distribution in two-dimensional space at a 97.5% confidence level, corresponding to a coverage area of ​​approximately 95% probability. The number of interior points under the current candidate Gaussian distribution is counted; this number serves as the voting score for that candidate hypothesis. After iteration, the interior point set corresponding to the candidate hypothesis with the highest voting score is selected as the final interior point set. And the mean vector of the candidate hypothesis is used as the first... The coordinates of the 2D key points corresponding to the 3D key points .

[0050] Obtain 2D keypoint coordinates Then, based on the interior point set Calculate the covariance matrix. Calculate the set of vote interior points. The sample covariance matrix, denoted as Sample covariance matrix The calculation method is as follows: subtract the mean from all points in the interior point set. Obtain the mean-reduced vectors, calculate the sum of the outer products of these mean-reduced vectors, and then divide by the number of inliers minus one.

[0051] Add a preset isotropic noise covariance matrix and fuse them to obtain a covariance matrix for subsequent pose calculation. The fusion method is as follows:

[0052]

[0053] in, Indicates the first The final covariance matrix of each 2D keypoint; Represents the sample covariance matrix of the in-point set of the vote; This represents the preset standard deviation of isotropic noise. Set to 2.0 pixels, this setting combines the noise level of the depth camera image sensor with the uncertainty of the scale-invariant unit vector field prediction to prevent the covariance matrix from overfitting to the inlier distribution of the current frame. express The identity matrix. The addition of the isotropic noise covariance matrix makes the covariance matrix... The lower bound of uncertainty in all directions is guaranteed, thus providing a robust Mahalanobis distance metric in subsequent uncertainty-driven perspective n-point algorithms.

[0054] See Figure 4 The figure shows the two-dimensional candidate point set for a certain 3D key point in Example 3. The scatter distribution of the voting results and the two-dimensional Gaussian distribution confidence ellipse obtained by fitting with the random sampling consensus algorithm. The horizontal and vertical axes represent the pixel coordinates of the image, respectively, and their numerical range covers the spatial dimensions of the input image. The set of interior points marked with red dots. Gathered at about The pixel neighborhood reflects the clustering trend of 2D keypoint locations in the foreground pixels based on the scale-invariant unit vector field voting. Blue asterisks indicate voting locations judged as outliers; their distribution is sparse and widely scattered across various image regions, consistent with the random characteristics of voting noise. The green dashed line depicts... The confidence ellipse encloses the main voting points of the inlier set, indicating that the Gaussian distribution model fits the spatial distribution characteristics of the inlier set well. This voting result is used to further calculate the sample covariance matrix, which, combined with the pre-defined isotropic noise covariance matrix, forms the final covariance. And participate in the pose calculation of the subsequent uncertainty-driven perspective n-point algorithm.

[0055] Example 4

[0056] In practical implementation, the uncertainty of the input uncertainty of the matching point pairs and covariance matrix drives the perspective n-point algorithm to minimize the Mahalanobis distance and solve the pose. The matching point pairs obtained from the aforementioned steps include For the 2D-3D correspondence, where the first... Matching point pairs are determined by the coordinates of 2D key points. Corresponding 3D keypoint coordinates Composition, with each 2D keypoint carrying a covariance matrix .

[0057] The uncertainty-driven perspective n-point algorithm constructs a Mahalanobis distance-weighted reprojection error function based on the covariance matrix. The camera projection model uses a pinhole model, and the camera intrinsic parameter matrix is ​​denoted as... Its specific parameters include focal length and and principal point coordinates and These parameters are obtained through the factory calibration of the depth camera. The camera pose to be solved is determined by the rotation matrix. With translation vector Represents the rotation matrix. It is An orthogonal matrix with a determinant of 1, and a translation vector. It is a three-dimensional vector. The first... 3D key points Based on the currently estimated pose Projecting onto a two-dimensional image plane yields the projection point. The projection process is to first project Transformed to camera coordinate system Then Through camera intrinsic parameter matrix Project onto the pixel coordinate system. Reprojection error vector. Defined as 2D keypoint coordinates With projection point The difference, that is , It is a two-dimensional vector.

[0058] Mahalanobis distance-weighted reprojection error function The expression is:

[0059]

[0060] in, This represents the total number of 3D key points, with a value of 8. Indicates the first A pair of matching points in the current pose estimation The reprojection error vector is a two-dimensional column vector. Represents the reprojection error vector Transpose of; Indicates the first Each 2D keypoint corresponds to a covariance matrix. The inverse matrix. Covariance matrix. The inverse matrix serves to scale the error in each direction, assigning lower weights to errors in directions with greater uncertainty and higher weights to errors in directions with less uncertainty.

[0061] Minimize the nonlinear least squares problem by iteratively solving it. The pose parameters are updated until the convergence condition is met. The Levenberg-Marquardt algorithm is used for the nonlinear least squares solution. Pose parameters... The initial values ​​are set as follows: rotation matrix Initialized as an identity matrix, translation vector Initialize to a zero vector. The convergence condition comprises two criteria; iteration stops when either one is satisfied: the norm of the pose parameter update is less than 0.5%. Or, the number of iterations reaches a preset limit of 100. The rotation matrix output when the iteration terminates. With translation vector This establishes the initial pose.

[0062] In some implementations, the pose predictor includes a residual network and an iterative optimizer. The input to the residual network consists of two concatenated parts: the first part is the rendering features corresponding to the initial pose, and the second part is the deep fusion features output by the improved pixel voting network. The rendering features are obtained by: based on the initial pose... A differentiable rendering process was performed on the 3D model of the billet end face. A lightweight convolutional neural network (CNN) was used to extract the rendered feature map from the image. The CNN consisted of two 3×3 convolutional layers and one 1×1 convolutional layer stacked sequentially, resulting in an output feature map with 64 channels. The deep fusion feature map was obtained by concatenating and fusing the high-level semantic feature map and low-level texture feature map extracted by the shared feature encoder in the improved pixel voting network, resulting in 128 channels. The rendered feature map and the deep fusion feature map were concatenated along the channel dimension to form a joint feature map with 192 channels, which served as the input to the residual network. The residual network adopted a fully convolutional structure, consisting of three 3×3 convolutional layers connected in series. The first two convolutional layers were followed by batch normalization layers and ReLU activation functions, and the last convolutional layer directly output 6 channels of pose residuals. These 6 channels corresponded to the three components of the 3D translation residual and the three Euler angle components of the 3D rotation residual.

[0063] The iterative optimizer repeatedly applies the pose residual to correct the initial pose. In each iteration, the rendered features and deep fusion features at the current pose are input into the residual network to obtain the pose residual. This pose residual is then applied to the current pose to obtain the updated pose. The number of iterations of the iterative optimizer is fixed at 5. The rotation matrix and translation vector output after 5 iterations are the optimal pose.

[0064] See Figure 5 In the figure, the horizontal axis represents the number of iterations during the training process of the improved pixel voting network, ranging from 0 to 500, and the vertical axis represents the total reprojection error at the corresponding number of iterations. The figure contains two curves, namely the "Madara distance weighted error" and the "Euclidean distance reprojection error," which are used to compare the error convergence of the Madara distance weighted error function based on the covariance matrix and the traditional Euclidean distance error function in the uncertainty-driven perspective n-point algorithm during training.

[0065] Looking at the trend of the curves, in the initial stage of 0 to 100 algebraic steps, the Mahalanobis distance weighted error curve rapidly decreased from about 500 to about 50, while the Euclidean distance error curve also decreased from about 540 to about 90, indicating that both have a relatively fast error convergence speed. However, the Mahalanobis distance weighted error is significantly lower than the Euclidean distance error, indicating that the error weighting mechanism combined with the covariance matrix effectively improves the accuracy of error measurement and convergence efficiency.

[0066] In the mid-stage, with 100 to 300 algebraic steps, the weighted error of the Mahalanobis distance continues to decrease steadily to near zero, with relatively small fluctuations in the error value; while the downward trend of the Euclidean distance error curve slows down, but remains higher than that of the Mahalanobis distance error, especially with more significant fluctuations in error between 200 and 300 steps, indicating that the convergence stability of the traditional Euclidean distance metric is poor.

[0067] In the later stages, with 300 to 500 iterations, both curves tend to stabilize, with error values ​​approaching zero and fluctuations being small, indicating that the training process is nearing convergence. Notably, the Mahalanobis distance-weighted error shows a better overall downward trend than the Euclidean distance error, further demonstrating the positive impact of using an uncertainty-driven error function based on the covariance matrix on improving pixel voting network training.

[0068] Example 5

[0069] In practice, the improved pixel voting network is trained using a multi-task loss function. This multi-task loss function consists of a weighted sum of three loss terms: the cross-entropy loss for semantic segmentation, the smoothing L1 loss for vector field regression, and a constraint term for the scale-adaptive normalization layer parameters.

[0070] The cross-entropy loss for semantic segmentation is supervised by the semantic segmentation mask of the billet end face. Each frame of the RGB image in the training data is accompanied by a pixel-by-pixel labeled ground truth mask for semantic segmentation, where foreground pixels are labeled as 1 and background pixels as 0. The pixel-by-pixel foreground probability output by the semantic segmentation decoder is denoted as... ,in and These represent the row and column indices of a pixel in the image, respectively, and their corresponding truth labels are denoted as . , The value is either 1 or 0. Cross-entropy loss for semantic segmentation. The calculation method is to sum the cross-entropy of all pixels and then take the average:

[0071]

[0072] in, This indicates the number of samples in a batch during training. Indicates the height of the semantic segmentation mask. Indicates the width of the semantic segmentation mask. This represents the row index of the pixel, with a value range of 1. to , This represents the column index of the pixel, with a value range of 1. to , Indicates the semantic segmentation decoder at position The output foreground probability value, Indicates position The truth label at the location has a value of 1 indicating the foreground and a value of 0 indicating the background.

[0073] The smoothed L1 loss of vector field regression is supervised for scale-invariant unit vector fields. In the training data, a ground truth orientation of a two-dimensional unit vector is pre-generated for each 3D keypoint. This is achieved by projecting the 3D keypoint onto a two-dimensional image plane using camera calibration parameters and the true pose, obtaining the projected coordinates, calculating the direction vector from the coordinates of each foreground pixel to the projected coordinates, and then normalizing it. The unnormalized prediction vector output by the vector field regression decoder is denoted as... ,in and Still representing the row and column indices of the pixel position, Indicates the first The indices of 3D keypoints, corresponding to the truth unit vector denoted as . .Will The output after processing by the scale-adaptive normalization layer is denoted as . Smoothing L1 loss for vector field regression The calculation method involves summing the smoothed L1 loss over all foreground pixels and all keypoints, and then averaging the sums. The smoothed L1 function is defined as follows: for the difference... ,when Time to take ,when Time to take The time difference is calculated as follows: and The difference at each vector component is calculated by applying smoothed L1 loss to the horizontal and vertical components respectively, and then summed.

[0074] The constraint terms for the scale-adaptive normalization layer parameters are related to the scale adaptation factor. Perform regularization. Constraint terms. The calculation method is to calculate all Individual scale adaptive factor The sum of squares is used to find the mean, i.e. ,in This represents the total number of 3D key points, with a value of 8. Indicates the first The scale adaptation factor corresponding to each key point. Constraint terms. To prevent the scale adaptation factor from growing to an excessively large value during training, thereby maintaining the numerical stability of the scale-invariant unit vector field.

[0075] Multi-task loss function It is composed of the weighted sum of the three loss items mentioned above, that is Weighting coefficients Set to 1.0, weighting coefficient Set to 0.5, weighting coefficient Set to 0.01. Weighting coefficient. The weight is set to 0.5 because vector field regression converges more slowly than semantic segmentation; assigning a lower weight helps maintain a relatively balanced gradient scale between the two tasks in the early stages of training. The reason for setting it to 0.01 is that the regularization term only plays a constraining role, and its magnitude should be much smaller than that of the dominant loss term to avoid overly restricting the learning ability of the scale adaptation factor.

[0076] The specific parameters for training the improved pixel voting network are set as follows: the training dataset consists of 3,000 RGB images of billet end faces collected on the billet production line, each image is equipped with a corresponding semantic segmentation ground truth mask and unit vector ground truth map; the optimizer uses stochastic gradient descent with a momentum coefficient of 0.9; the initial learning rate is set to 0.001, and the learning rate decays to 0.1 times the original value every 50 training epochs; the batch size is set to 8; and the total number of training epochs is set to 200.

[0077] In some embodiments, a 6D pose estimation system for a billet end face based on improved pixel voting includes a memory, a processor, and a computer program stored in the memory and running on the processor. When executing the computer program, the processor implements the steps of the 6D pose estimation method for a billet end face based on improved pixel voting described in all the above embodiments. The memory uses a high-speed double-rate synchronous dynamic random access memory (DRAM) to temporarily store the RGB image of the billet end face acquired by the depth camera, intermediate image data generated during preprocessing, network weights and feature maps of the improved pixel voting network, the 2D candidate point set and covariance matrix generated during the voting process, and intermediate parameters during the pose calculation process. The processor reads the RGB image data by calling the software development kit (SDK) interface of the depth camera, and sequentially performs Gaussian filtering and histogram equalization preprocessing, forward inference of the improved pixel voting network, foreground pixel voting and 2D keypoint extraction, pose calculation using an uncertainty-driven perspective n-point algorithm, and iterative optimization of the pose predictor, finally outputting the optimal pose for use by the billet gripping device.

[0078] See Figure 6 The figure shows the convergence trend of the multi-task loss function based on the improved pixel voting network in Example 5 as the number of training iterations increases. The horizontal axis represents the number of iterations in the training process, ranging from 0 to 500 steps; the vertical axis represents the corresponding loss value, ranging from approximately 0 to 1.2. The curves include the semantic segmentation cross-entropy loss (red solid line), the vector field regression smoothing L1 loss (blue dashed line), and the total multi-task loss (green dotted line).

[0079] As shown in the figure, the initial value of the semantic segmentation cross-entropy loss is approximately 0.9. It exhibits a significant decreasing trend with increasing iteration steps, dropping below approximately 0.1 after 200 steps, and then stabilizing with a slow, continuous decrease. This indicates that the network's learning effect on the semantic segmentation of the billet end face gradually improves. The initial value of the vector field regression smoothing L1 loss is approximately 0.45, decreasing more gradually than the semantic segmentation loss. It drops to approximately 0.07 after 200 steps, entering a stable phase, reflecting the gradual convergence of the direction prediction of the scale-invariant unit vector field. The multi-task total loss starts at approximately 1.15, comprehensively reflecting the weighted effect of the three loss terms. Its overall decreasing trend is similar to that of the semantic segmentation loss, decreasing rapidly in the first 200 steps, then stabilizing, and finally settling at around 0.03. This indicates that the overall training process effectively balances the optimization needs of the semantic segmentation and vector field regression tasks.

[0080] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A 6D pose estimation method for billet end face based on improved pixel voting, characterized in that, Includes the following steps: RGB images of the billet end face are acquired using a depth camera; The RGB image is subjected to Gaussian filtering and histogram equalization. Based on the three-dimensional model of the billet end face, the farthest point sampling is used to obtain the 3D key points distributed on the model surface; The preprocessed RGB image is input into an improved pixel voting network, which simultaneously outputs a semantic segmentation mask for the billet end face and a scale-invariant unit vector field. Based on the semantic segmentation mask and the scale-invariant unit vector field, foreground pixels are selected and voted on. Random sampling consistency is used to generate 2D candidate hypotheses for each 3D key point. The voting score estimates the spatial distribution to obtain the coordinates and covariance matrix of the 2D key points. Based on the camera intrinsic parameters, a one-to-one correspondence is established between the 2D key points and the 3D key points to form 2D-3D matching point pairs; The matching point pair and the covariance matrix are input into the uncertainty-driven perspective n-point algorithm to minimize the Mahalanobis distance to solve the pose and output the initial pose. The initial pose and the deep fusion features output by the improved pixel voting network are input into the pose predictor to obtain the optimal pose.

2. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 1, characterized in that, The improved pixel voting network includes a shared feature encoder, a semantic segmentation decoder, and a vector field regression decoder; The shared feature encoder extracts multi-scale feature maps from the preprocessed RGB image; The semantic segmentation decoder generates the semantic segmentation mask of the billet end face based on the multi-scale feature map; The vector field regression decoder predicts unit vectors pointing to the 3D key points pixel by pixel based on the multi-scale feature map, and normalizes each unit vector through a scale-adaptive normalization layer to output the scale-invariant unit vector field.

3. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 1, characterized in that, The 3D key points distributed on the surface of the model based on the three-dimensional model of the billet end face are obtained by sampling from the farthest point, including: Convert the 3D model of the billet end face into a point cloud and set the number of sampling points K; Randomly select an initial point, iteratively calculate the farthest distance from each point to the selected point set, add the point with the maximum distance to the selected point set, until K points are selected as the 3D key points.

4. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 1, characterized in that, The process of generating 2D candidate hypotheses for each of the 3D keypoints using random sampling consistency, and estimating the spatial distribution of voting scores to obtain the 2D keypoint coordinates and covariance matrix includes: For each of the 3D key points, the voting position of each foreground pixel is calculated based on the scale-invariant unit vector field, and the results are aggregated to generate a 2D candidate point set. A two-dimensional Gaussian distribution is fitted to the 2D candidate point set using random sampling consensus, and the voting score of each candidate hypothesis is calculated. The cluster mean with the highest score is used as the coordinates of the 2D keypoints, and the covariance matrix is ​​calculated based on the inliers.

5. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 4, characterized in that, The calculation of the covariance matrix based on interior points includes: Calculate the sample covariance of the voting inlier set, add a preset isotropic noise covariance matrix, and fuse them to obtain the covariance matrix.

6. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 1, characterized in that, The step of inputting the matching point pair and the covariance matrix into an uncertainty-driven perspective n-point algorithm, minimizing the Mahalanobis distance to solve the pose, and outputting the initial pose includes: Construct a Mahalanobis distance-weighted reprojection error function based on the covariance matrix; The initial pose is output by iteratively solving a nonlinear least squares problem and updating the pose parameters until the convergence condition is met.

7. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 1, characterized in that, The pose predictor includes a residual network and an iterative optimizer; The residual network takes the concatenation of the rendering features corresponding to the initial pose and the deep fusion features as input, and outputs the pose residual. The iterative optimizer repeatedly applies the pose residual to correct the initial pose and generate the optimal pose.

8. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 1, characterized in that, The deep fusion feature is obtained by concatenating and fusing the high-level semantic feature map and low-level texture feature map extracted by the shared feature encoder in the improved pixel voting network channel by channel.

9. The 6D pose estimation method for billet end face based on improved pixel voting according to claim 1 or 2, characterized in that, The improved pixel voting network is trained using a multi-task loss function, which includes cross-entropy loss for semantic segmentation and smooth L1 loss for vector field regression. A constraint term for the scale-adaptive normalization layer parameters is introduced into the vector field regression loss.

10. A 6D pose estimation system for billet end face based on improved pixel voting, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the 6D pose estimation method for billet end face based on improved pixel voting as described in any one of claims 1 to 9.

Images