A 6D pose extraction method and device of a three-dimensional workpiece

Abstract

The application discloses a kind of 6D pose extraction methods and devices of three-dimensional workpiece, it is related to workpiece pose acquisition technical field, including image acquisition module, pre-processing module, multimodal feature extraction module, hierarchical pose inference module, post-processing module, communication interface module and closed loop learning module, in the present application, through multidimensional uncertainty quantification and industrial scene physical consistency verification, effectively filter physically infeasible error results, can greatly improve the reliability of system operation;Through closed loop learning module, self-optimization and automatic expansion of system are realized, without manual intervention can quickly adapt to new workpiece and new scene, fully meet the comprehensive requirements of industrial production to the precision, speed, robustness and safety of three-dimensional workpiece 6D pose extraction.

Description

Technical Field

[0001] This invention relates to the field of workpiece pose acquisition technology, and in particular to a method and apparatus for 6D pose extraction of a three-dimensional workpiece. Background Technology

[0002] In automated production scenarios such as disordered grasping, flexible assembly, and loading / unloading of industrial robots, accurate 6D pose extraction of three-dimensional workpieces is the core foundation for achieving intelligent operation.

[0003] Existing technologies still suffer from the following technical shortcomings: Firstly, existing instance-level pose estimation methods rely on pre-built static 3D models and mask databases, making it difficult to quickly adapt to production line changeover requirements. Furthermore, they lack depth restoration and optimization for special materials widely found in industrial scenarios, such as reflective metals, light-absorbing black surfaces, and transparent plastics, resulting in large depth measurement errors and a sharp decline in pose estimation accuracy. Secondly, existing category-level methods often employ pure geometric features or simple RGB-D feature stitching, failing to achieve deep interaction between texture and geometric features. This results in insufficient differentiation of geometrically similar workpieces, and the single inference architecture cannot simultaneously balance the inference speed for known instances with the generalization ability for unknown workpieces. Thirdly, existing generative model-free methods suffer from slow inference speed and low accuracy, making it difficult to meet industrial real-time requirements. Moreover, all existing technologies lack multi-dimensional uncertainty quantification and physical consistency verification mechanisms specific to industrial scenarios, failing to effectively filter physically infeasible erroneous results and posing serious security risks. Summary of the Invention

[0004] The purpose of this invention is to provide a 6D pose extraction method and apparatus for three-dimensional workpieces to solve the above-mentioned technical problems.

[0005] The technical solution adopted in this invention is as follows: A method for 6D pose extraction of a three-dimensional workpiece, comprising: S1. Obtain the RGB image, depth image, and camera intrinsic parameters of the target scene; S2. Then, multimodal instance segmentation is performed on the RGB image and depth image to obtain the segmentation mask, workpiece category and occlusion rate of each 3D workpiece in the scene. S3. Then, classify the materials of each 3D workpiece to obtain the corresponding material category; S4. Based on the material type of the 3D workpiece, a differentiated depth repair strategy is adopted to repair the depth missing area of ​​the corresponding workpiece in the depth image, and the repaired depth image is obtained. S5. Then, by combining the camera intrinsic parameters, segmentation mask, and repaired depth image, a normalized partial observation point cloud is generated for each 3D workpiece. S6. Then, a dual-stream cross-attention network is used to extract multi-scale texture features from RGB images and multi-scale geometric features from some observed point clouds, and cross-modal feature interaction is performed to obtain multi-modal fusion features for each 3D workpiece. S7. Then, based on the workpiece category, occlusion rate, and dynamic incremental workpiece template library matching results of each 3D workpiece, at least one of the instance-level inference branch, category-level inference branch, or generative adaptation branch is dynamically selected to process the multimodal fusion features and some observation point clouds to obtain preliminary 6D pose and size estimation results. S8. Then, the preliminary 6D pose and size estimation results are subjected to multi-dimensional uncertainty quantification and physical consistency verification in industrial scenarios. S9. Output the final 6D pose, 3D dimensions, and corresponding confidence information that have passed verification. The confidence information includes the overall confidence score and the uncertainty value of each dimension.

[0006] As a preferred option, in S3, the material categories include reflective metallic materials, black metallic light-absorbing materials, transparent plastic materials, and ordinary non-metallic materials.

[0007] As a further preferred option, in S4, a differentiated deep repair strategy is adopted, including: S4.1 When the material type is metallic reflective material, the neighborhood interpolation method is used to repair small areas of depth loss; S4.2 When the material type is metallic reflective material, a specular suppression combined with an RGB-guided depth generation network is used to first remove the specular areas in the RGB image, and then use the remaining texture and contour information to predict the depth value of the missing area. S4.3 When the material category is black metal light-absorbing material, the depth is repaired by combining the prior category of the workpiece and the geometric features of the visible part through a shape completion algorithm based on CAD template. S4.4 When the material type is transparent plastic, a transmission light correction algorithm is used to eliminate the depth offset caused by the transmission effect.

[0008] As a preferred option, in S7, the dynamic selection of instance-level inference branches specifically includes: S7.1 If the 3D workpiece exists in the dynamic incremental workpiece template library and the occlusion rate is ≤30%, then select the instance-level inference branch; S7.2 If the 3D workpiece belongs to a known workpiece category and the occlusion rate is ≤60%, then select the category-level reasoning branch; S7.3 If the 3D workpiece is a brand new workpiece or the occlusion rate is > 60%, then select the generative adaptation branch; When each branch outputs a valid result, the results are weighted and fused according to the priority of instance-level inference branches over category-level inference branches, and category-level inference branches over generative adaptation branches. The weights are proportional to the confidence scores of each branch. When the confidence score of any single branch is lower than a preset threshold, cross-validation of other branches is automatically triggered.

[0009] As a preferred option, in S8: Multidimensional uncertainty quantification includes: based on a Bayesian neural network, several 6D pose and size estimation results are obtained through Monte Carlo dropout sampling, and the variances of rotation parameters, translation parameters and size parameters are calculated respectively as the uncertainty values ​​of the corresponding dimensions; Industrial scenario physical consistency verification includes: verifying whether the three workpieces are located within the preset working area, whether the size is within the machining tolerance range of this type of workpiece, whether there is spatial collision between several workpieces, and whether the pose meets the accessibility requirements of the actuator.

[0010] As a further preferred approach, the method for handling instance-level reasoning branches is as follows: S7.11. Compress the multimodal fusion features of the 3D workpiece into a fixed-length global feature vector, and retrieve the most similar template in the dynamic incremental workpiece template library through approximate nearest neighbor search. S7.12. Obtain the CAD model and initial 6D pose of the matching template; S7.13. Dynamically adjust the ICP registration strategy based on the occlusion rate of the 3D workpiece: When the occlusion rate is ≤30%, point-to-area ICP is adopted; When the occlusion rate is 30%~60%, weighted point-to-area ICP is used to reduce the weight of occluded points; When the occlusion rate is > 60%, feature-point based ICP is used to match only the key geometric features of the workpiece.

[0011] As a further preferred approach, the method for handling category-level reasoning branches is as follows: S7.21. Perform similarity matching between the multimodal fusion features of the 3D workpiece and all prototypes in the pre-built multi-prototype workpiece category library, and select the workpiece prototype with the highest similarity. S7.22. Predict the symmetry type of the workpiece through an adaptive symmetry sensing network. The symmetry types include asymmetry, single-axis symmetry, double-axis symmetry, and rotational symmetry. S7.23. Based on the predicted symmetry type, decide whether to perform symmetry completion on part of the observed point cloud and what completion method to use; S7.24. Predict the 3D rotation matrix of the workpiece through the orientation estimation subnetwork, and predict the center coordinates and three-dimensional dimensions of the workpiece through the dense center voting subnetwork to obtain preliminary 6D pose and size estimation results.

[0012] As a further preferred approach, the generative adaptation branching method is as follows: S7.31. Input the RGB image, the repaired depth image, the multimodal fusion features, and the prototype features of the most similar category into a lightweight NeRF generative network with category prior constraints to generate a high-precision 3D model of the workpiece. S7.32. Extract multimodal feature points from the generated 3D model and observed images, and perform feature matching to obtain the 2D-3D correspondence; S7.33. The initial pose and scale are solved using the PnP algorithm, and then further refined by point cloud registration to obtain preliminary 6D pose and size estimation results.

[0013] A 6D pose extraction device for a three-dimensional workpiece, comprising: The image acquisition module is used to acquire RGB images, depth images, and camera intrinsic parameters of the target scene; The preprocessing module, connected to the image acquisition module, is used to perform multimodal instance segmentation, material classification, differential depth repair and normalized point cloud generation, and outputs the segmentation mask, workpiece category, occlusion rate, material category and normalized partial observation point cloud for each 3D workpiece. A multimodal feature extraction module, connected to the preprocessing module, has a built-in dual-stream cross-attention network for extracting and fusing RGB texture features and depth geometric features, and outputting the multimodal fused features of each 3D workpiece; The hierarchical pose inference module, connected to the multimodal feature extraction module, includes instance-level inference branches, category-level inference branches, generative adaptation branches, and a dynamic branch selector, which are used to dynamically select the optimal inference path based on scene information and output preliminary 6D pose and size estimation results. The post-processing module, connected to the hierarchical pose inference module, is used to perform multi-dimensional uncertainty quantification and industrial scene physical consistency verification, and outputs the final 6D pose, three-dimensional dimensions and confidence information. The communication interface module is connected to the post-processing module and is used to interact with external actuators for data exchange.

[0014] As a further preferred embodiment, a closed-loop learning module is also included, which is connected to the preprocessing module and the hierarchical pose inference module. This module is used to receive the operation results returned by the external actuator. If the operation is successful, the pose estimation result and multimodal features of the workpiece are added to the corresponding template library or category library. If the operation fails, re-inference is triggered and the model parameters are optimized.

[0015] The above technical solution has the following advantages or beneficial effects: (1) In this invention, multimodal instance segmentation can accurately extract the segmentation mask, category, and occlusion rate of 3D workpieces, solving the boundary ambiguity problem when multiple workpieces are stacked, and providing accurate basic data for pose estimation. Moreover, by classifying the material of each 3D workpiece and combining it with a differentiated depth repair strategy, the depth measurement error of workpieces with special materials can be significantly reduced, providing high-quality geometric data for pose estimation. Furthermore, multimodal feature extraction and interactive enhancement can improve the differentiation of workpieces with similar geometric shapes, and hierarchical pose inference can select the optimal inference path according to different situations, which can effectively improve the accuracy of 6D pose estimation of 3D workpieces and meet the needs of industrial production for high-precision pose estimation.

[0016] (2) In this invention, by dynamically selecting the three-layer inference architecture of instance-level, category-level or generative adaptation branches, the inference speed of known instances, the estimation accuracy of unknown instances of known categories and the generalization ability of new workpieces are taken into account. In this way, pose estimation results can be obtained quickly for common known workpieces, improving production efficiency; and it can also have good adaptability to newly emerging workpieces, ensuring that the system can still operate stably in the event of production line changes.

[0017] (3) In this invention, by using multi-dimensional uncertainty quantification and industrial scene physical consistency verification, the erroneous results that are physically infeasible can be effectively filtered out, which can greatly improve the reliability of system operation; by using the closed-loop learning module, the system can achieve self-optimization and automatic expansion, and can quickly adapt to new workpieces and new scenes without manual intervention, fully meeting the comprehensive requirements of industrial production for the accuracy, speed, robustness and safety of 6D pose extraction of three-dimensional workpieces. Attached Figure Description

[0018] Figure 1 This is a flowchart of the 6D pose extraction method for a three-dimensional workpiece in this invention; Figure 2 This is a structural diagram of the 6D pose extraction device for a three-dimensional workpiece in this invention. Detailed Implementation

[0019] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the description of this invention, it should be noted that terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, terms such as "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] Figure 1 This is a flowchart of the 6D pose extraction method for a three-dimensional workpiece in this invention; Figure 2 This is a structural diagram of the 6D pose extraction device for a three-dimensional workpiece in this invention. Please refer to [link / reference]. Figures 1 to 2 The diagram illustrates a preferred embodiment of a 6D pose extraction method for a three-dimensional workpiece, comprising: S1. Acquire the RGB image, depth image, and camera intrinsic parameters of the target scene; In this embodiment, an industrial camera or other image acquisition device can be used to acquire the RGB image, depth image, and camera intrinsic parameters of the target scene. In practical applications, the position, angle, and parameters of the camera can be reasonably adjusted according to factors such as the size of the scene, the position and orientation of the 3D workpiece, to ensure clear and accurate image information is obtained. For large workpieces or workpieces far from the camera, it may be necessary to adjust the camera's focal length and exposure time to ensure image clarity and contrast.

[0023] S2. Then, multimodal instance segmentation is performed on the RGB and depth images to obtain the segmentation mask, object category, and occlusion rate of each 3D object in the scene. In this embodiment, a multimodal instance segmentation algorithm is used to process the RGB and depth images. This algorithm can be based on a deep learning model, such as Mask R-CNN. By analyzing the color, texture, and depth features in the image, each 3D object in the scene is identified, and its segmentation mask, object category, and occlusion rate are obtained. When training the model, a large amount of multimodal image data containing different objects and different occlusion conditions is required to improve the model's segmentation accuracy and generalization ability.

[0024] S3. Then, classify the material of each 3D workpiece to obtain the corresponding material category. The material classification model can be based on deep learning architectures such as convolutional neural networks (CNN), and learn from a large amount of image data of workpieces with different materials to master the feature representation of different materials. The material categories include reflective metallic materials, light-absorbing black metallic materials, transparent plastic materials, and ordinary non-metallic materials.

[0025] S4. Based on the material type of the 3D workpiece, a differentiated depth repair strategy is adopted to repair the depth missing area of ​​the corresponding workpiece in the depth image, and the repaired depth image is obtained. The differentiated deep repair strategy includes: S4.1 The material type is ordinary non-metallic material. The neighborhood interpolation method is used to repair small areas of depth loss. Specifically, the depth value of the missing area can be calculated by linear interpolation or other suitable interpolation methods based on the known depth values ​​around the missing area in the depth image.

[0026] S4.2 When the material type is metallic reflective material, a depth generation network combining specular suppression and RGB guidance is used. First, the specular areas in the RGB image are removed, and then the remaining texture and contour information is used to predict the depth values ​​of the missing areas. Specifically, image processing algorithms can be used to remove specular areas from the RGB image, for example, by detecting and suppressing specular pixels in the image. Then, the processed RGB image and the original depth image are input into the depth generation network, which, based on deep learning technology, uses the remaining texture and contour information to predict the depth values ​​of the missing areas. Training the depth generation network requires a large amount of image data of metallic reflective workpieces with both specular highlights and depth defects, enabling the network to learn how to use RGB information to repair depth defects.

[0027] S4.3 When the material type is a black metallic light-absorbing material, depth is repaired by combining the prior knowledge of the workpiece type and the geometric features of the visible part through a shape completion algorithm based on CAD templates. First, a corresponding CAD template can be selected from a pre-built CAD template library according to the workpiece type. Then, the geometric features of the visible part of the workpiece in the depth image are used to match and adjust the CAD template. Finally, the missing parts in the depth image are filled in according to the adjusted CAD template.

[0028] S4.4 When the material type is transparent plastic, a transmission light correction algorithm is used to eliminate depth shift caused by transmission effects. Depth images can be corrected by analyzing the transmission characteristics of the transparent material and the camera imaging principle.

[0029] S5. Then, combining the camera intrinsic parameters, segmentation mask, and repaired depth image, a normalized partial observation point cloud for each 3D workpiece is generated. In this embodiment, the depth values ​​in the depth image can be converted into 3D spatial coordinates based on the camera intrinsic parameters, and then the point cloud data corresponding to each workpiece is determined by combining the segmentation mask. Then, these point cloud data are normalized to give them a uniform scale and coordinate system, which facilitates subsequent feature extraction and processing.

[0030] S6. Then, the multi-scale texture features of the RGB image and the multi-scale geometric features of some observed point clouds are extracted through the dual-stream cross-attention (Transformer) network, and cross-modal feature interaction is performed to obtain the multi-modal fusion features of each three-dimensional workpiece. In this embodiment, the processing flow of the dual-stream cross-attention (Transformer) network includes: first, extracting multi-scale texture features from the RGB image using a lightweight ViT-Small encoder in the RGB branch, focusing on the edges, corners, and processing texture features of the 3D workpiece; then, extracting multi-scale geometric features from a portion of the observed point cloud using a lightweight PointNet++ encoder in the depth branch, focusing on the planar, curved, and hole features of the workpiece; and within each Transformer block, using a geometry-guided texture attention mechanism, using geometric features to guide texture features to focus on key structural regions of the workpiece; simultaneously, using a texture-guided geometric attention mechanism, using texture features to guide geometric features to focus on semantically related geometric structures; finally, fusing cross-attention features of different scales through a feature pyramid network to obtain multimodal fused features. It is necessary to dynamically adjust the fusion weights of texture features and geometric features according to the material type of the workpiece: for reflective metallic materials, increase the weight of geometric features; for texture-rich non-metallic materials, increase the weight of texture features.

[0031] S7. Then, based on the workpiece category, occlusion rate, and dynamic incremental workpiece template library matching results of each 3D workpiece, at least one of the instance-level inference branch, category-level inference branch, or generative adaptation branch is dynamically selected to process the multimodal fusion features and some observation point clouds to obtain preliminary 6D pose and size estimation results. In S7, dynamically selecting instance-level inference branches specifically includes: S7.1 If the 3D workpiece exists in the dynamic incremental workpiece template library and the occlusion rate is ≤30%, then the instance-level inference branch is selected. In this embodiment, the instance-level inference branch processing flow includes S7.11, compressing the multimodal fusion features of the 3D workpiece into a fixed-length global feature vector, and retrieving the most similar template in the dynamic incremental workpiece template library through approximate nearest neighbor search; S7.12, obtaining the CAD model and initial 6D pose of the matching template; S7.13, dynamically adjusting the ICP registration strategy according to the occlusion rate of the 3D workpiece: when the occlusion rate is ≤30%, point-to-surface ICP is used; when the occlusion rate is 30%~60%, weighted point-to-surface ICP is used to reduce the weight of occluded points; when the occlusion rate is >60%, feature point-based ICP is used to match only the key geometric features of the workpiece.

[0032] S7.2 If the 3D workpiece belongs to a known workpiece category and the occlusion rate is ≤60%, then select the category-level inference branch. In this embodiment, the category-level inference branch processing method is as follows: S7.21, perform similarity matching between the multimodal fusion features of the 3D workpiece and all prototypes in the pre-built multi-prototype workpiece category library, and select the workpiece prototype with the highest similarity; S7.22, predict the symmetry type of the workpiece through an adaptive symmetry perception network. The symmetry type includes asymmetry, single-axis symmetry, bi-axis symmetry, and rotational symmetry; S7.23, based on the predicted symmetry type, decide whether to perform symmetry completion on part of the observation point cloud and what kind of completion method to use; S7.24, predict the 3D rotation matrix of the workpiece through the orientation estimation sub-network, and predict the center coordinates and 3D size of the workpiece through the dense center voting sub-network to obtain the preliminary 6D pose and size estimation results.

[0033] S7.3 If the 3D workpiece is a completely new workpiece or the occlusion rate is > 60%, then the generative adaptation branch is selected. In this embodiment, the generative adaptation branch processing method is as follows: S7.31 Input the RGB image, the repaired depth image, the multimodal fusion features and the prototype features of the most similar category into the lightweight NeRF generative network with category prior constraints to generate a high-precision 3D model of the workpiece; S7.32 Extract multimodal feature points from the generated 3D model and the observed image, and perform feature matching to obtain the 2D-3D correspondence; S7.33 Use the PnP algorithm to solve the initial pose and scale, and then further refine it through point cloud registration to obtain the preliminary 6D pose and size estimation results.

[0034] When each branch outputs a valid result, the results are weighted and fused according to the priority of instance-level inference branches over category-level inference branches, and category-level inference branches over generative adaptation branches. The weights are proportional to the confidence scores of each branch. When the confidence score of any single branch is lower than a preset threshold, cross-validation of other branches is automatically triggered.

[0035] S8. Then, the preliminary 6D pose and size estimation results are subjected to multi-dimensional uncertainty quantification and physical consistency verification in industrial scenarios. The multi-dimensional uncertainty quantification includes: based on a Bayesian neural network, several 6D pose and size estimation results are obtained through Monte Carlo dropout sampling. The variances of rotation, translation, and size parameters are calculated as the uncertainty values ​​for the corresponding dimensions. Specifically, during the training of the Bayesian neural network, multiple random dropout operations are performed to obtain multiple different model parameters. These models with different parameters are used to predict the input data, resulting in several sets of 6D pose and size estimation results. Then, the variances of rotation, translation, and size parameters in each set of results are calculated to evaluate the uncertainty of each dimension. For example, a large variance in the rotation parameter indicates high uncertainty in the estimation of the rotation angle, requiring further analysis and verification.

[0036] Industrial scenario physical consistency verification includes: verifying whether the three workpieces are located within the preset working area, whether the size is within the machining tolerance range of this type of workpiece, whether there is spatial collision between several workpieces, and whether the pose meets the accessibility requirements of the actuator.

[0037] S9. Output the final 6D pose, 3D dimensions, and corresponding confidence information that have passed verification. This information can be used for operations such as grasping and assembly of industrial robots. The confidence information includes the overall confidence score and the uncertainty values ​​of each dimension.

[0038] This invention also discloses a 6D pose extraction device for a three-dimensional workpiece, comprising: The image acquisition module is used to acquire RGB images, depth images, and camera intrinsic parameters of the target scene; The preprocessing module, connected to the image acquisition module, is used to perform multimodal instance segmentation, material classification, differential depth repair and normalized point cloud generation, and outputs the segmentation mask, workpiece category, occlusion rate, material category and normalized partial observation point cloud for each 3D workpiece; The multimodal feature extraction module, connected to the preprocessing module, has a built-in dual-stream cross-attention network for extracting and fusing RGB texture features and depth geometric features, and outputs the multimodal fused features of each 3D workpiece. The hierarchical pose inference module, connected to the multimodal feature extraction module, includes instance-level inference branches, category-level inference branches, generative adaptation branches, and dynamic branch selectors, which are used to dynamically select the optimal inference path based on scene information and output preliminary 6D pose and size estimation results. The post-processing module, connected to the hierarchical pose inference module, is used to perform multi-dimensional uncertainty quantification and industrial scene physical consistency verification, and outputs the final 6D pose, three-dimensional dimensions and confidence information. The communication interface module connects to the post-processing module and is used for data interaction with external actuators.

[0039] Furthermore, as a preferred implementation, a closed-loop learning module is also included, connected to the preprocessing module and the hierarchical pose inference module. This module receives the operation results returned by the external actuator. If the operation is successful, the pose estimation result and multimodal features of the workpiece are added to the corresponding template library or category library. If the operation fails, re-inference is triggered and the model parameters are optimized. In this embodiment, the closed-loop learning module enables rapid adaptation to new workpieces and new scenarios without manual intervention.

[0040] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for 6D pose extraction of a three-dimensional workpiece, characterized in that, include: S1. Obtain the RGB image, depth image, and camera intrinsic parameters of the target scene; S2. Then, multimodal instance segmentation is performed on the RGB image and depth image to obtain the segmentation mask, workpiece category and occlusion rate of each 3D workpiece in the scene. S3. Then, classify the materials of each 3D workpiece to obtain the corresponding material category; S4. Based on the material type of the 3D workpiece, a differentiated depth repair strategy is adopted to repair the depth missing area of ​​the corresponding workpiece in the depth image, and the repaired depth image is obtained. S5. Then, by combining the camera intrinsic parameters, segmentation mask, and repaired depth image, a normalized partial observation point cloud is generated for each 3D workpiece. S6. Then, a dual-stream cross-attention network is used to extract multi-scale texture features from RGB images and multi-scale geometric features from some observed point clouds, and cross-modal feature interaction is performed to obtain multi-modal fusion features for each 3D workpiece. S7. Then, based on the workpiece category, occlusion rate, and dynamic incremental workpiece template library matching results of each 3D workpiece, at least one of the instance-level inference branch, category-level inference branch, or generative adaptation branch is dynamically selected to process the multimodal fusion features and some observation point clouds to obtain preliminary 6D pose and size estimation results. S8. Then, the preliminary 6D pose and size estimation results are subjected to multi-dimensional uncertainty quantification and physical consistency verification in industrial scenarios. S9. Output the final 6D pose, 3D dimensions, and corresponding confidence information that have passed verification. The confidence information includes the overall confidence score and the uncertainty value of each dimension.

2. The 6D pose extraction method for a three-dimensional workpiece as described in claim 1, characterized in that, In S3, the material categories include reflective metallic materials, black metallic light-absorbing materials, transparent plastic materials, and ordinary non-metallic materials.

3. The 6D pose extraction method for a three-dimensional workpiece as described in claim 2, characterized in that, In S4, a differentiated deep repair strategy is employed, including: S4.1 When the material type is metallic reflective material, the neighborhood interpolation method is used to repair small areas of depth loss; S4.2 When the material type is metallic reflective material, a specular suppression combined with an RGB-guided depth generation network is used to first remove the specular areas in the RGB image, and then use the remaining texture and contour information to predict the depth value of the missing area. S4.3 When the material category is black metal light-absorbing material, the depth is repaired by combining the prior category of the workpiece and the geometric features of the visible part through a shape completion algorithm based on CAD template. S4.4 When the material type is transparent plastic, a transmission light correction algorithm is used to eliminate the depth offset caused by the transmission effect.

4. The 6D pose extraction method for a three-dimensional workpiece as described in claim 1, characterized in that, In S7, dynamically selecting instance-level inference branches specifically includes: S7.1 If the 3D workpiece exists in the dynamic incremental workpiece template library and the occlusion rate is ≤30%, then select the instance-level inference branch; S7.2 If the 3D workpiece belongs to a known workpiece category and the occlusion rate is ≤60%, then select the category-level reasoning branch; S7.3 If the 3D workpiece is a brand new workpiece or the occlusion rate is > 60%, then select the generative adaptation branch; When each branch outputs a valid result, the results are weighted and fused according to the priority of instance-level inference branches over category-level inference branches, and category-level inference branches over generative adaptation branches. The weights are proportional to the confidence scores of each branch. When the confidence score of any single branch is lower than a preset threshold, cross-validation of other branches is automatically triggered.

5. The 6D pose extraction method for a three-dimensional workpiece as described in claim 1, characterized in that, In S8: Multidimensional uncertainty quantification includes: based on a Bayesian neural network, several 6D pose and size estimation results are obtained through Monte Carlo dropout sampling, and the variances of rotation parameters, translation parameters and size parameters are calculated respectively as the uncertainty values ​​of the corresponding dimensions; Industrial scenario physical consistency verification includes: verifying whether the three workpieces are located within the preset working area, whether the size is within the machining tolerance range of this type of workpiece, whether there is spatial collision between several workpieces, and whether the pose meets the accessibility requirements of the actuator.

6. The 6D pose extraction method for a three-dimensional workpiece as described in claim 4, characterized in that, The method for handling instance-level reasoning branches is as follows: S7.

11. Compress the multimodal fusion features of the 3D workpiece into a fixed-length global feature vector, and retrieve the most similar template in the dynamic incremental workpiece template library through approximate nearest neighbor search. S7.

12. Obtain the CAD model and initial 6D pose of the matching template; S7.

13. Dynamically adjust the ICP registration strategy based on the occlusion rate of the 3D workpiece: When the occlusion rate is ≤30%, point-to-area ICP is adopted; When the occlusion rate is 30%~60%, weighted point-to-area ICP is used to reduce the weight of occluded points; When the occlusion rate is > 60%, feature-point based ICP is used to match only the key geometric features of the workpiece.

7. The 6D pose extraction method for a three-dimensional workpiece as described in claim 4, characterized in that, The method for handling category-level reasoning branches is as follows: S7.

21. Perform similarity matching between the multimodal fusion features of the 3D workpiece and all prototypes in the pre-built multi-prototype workpiece category library, and select the workpiece prototype with the highest similarity. S7.

22. Predict the symmetry type of the workpiece through an adaptive symmetry sensing network. The symmetry types include asymmetry, single-axis symmetry, double-axis symmetry, and rotational symmetry. S7.

23. Based on the predicted symmetry type, decide whether to perform symmetry completion on part of the observed point cloud and what completion method to use; S7.

24. Predict the 3D rotation matrix of the workpiece through the orientation estimation subnetwork, and predict the center coordinates and three-dimensional dimensions of the workpiece through the dense center voting subnetwork to obtain preliminary 6D pose and size estimation results.

8. The 6D pose extraction method for a three-dimensional workpiece as described in claim 4, characterized in that, The generative adaptation branching method is as follows: S7.

31. Input the RGB image, the repaired depth image, the multimodal fusion features, and the prototype features of the most similar category into a lightweight NeRF generative network with category prior constraints to generate a high-precision 3D model of the workpiece. S7.

32. Extract multimodal feature points from the generated 3D model and observed images, and perform feature matching to obtain the 2D-3D correspondence; S7.

33. The initial pose and scale are solved using the PnP algorithm, and then further refined by point cloud registration to obtain preliminary 6D pose and size estimation results.

9. A 6D pose extraction device for a three-dimensional workpiece, characterized in that, include: The image acquisition module is used to acquire RGB images, depth images, and camera intrinsic parameters of the target scene; The preprocessing module, connected to the image acquisition module, is used to perform multimodal instance segmentation, material classification, differential depth repair and normalized point cloud generation, and outputs the segmentation mask, workpiece category, occlusion rate, material category and normalized partial observation point cloud for each 3D workpiece. A multimodal feature extraction module, connected to the preprocessing module, has a built-in dual-stream cross-attention network for extracting and fusing RGB texture features and depth geometric features, and outputting the multimodal fused features of each 3D workpiece; The hierarchical pose inference module, connected to the multimodal feature extraction module, includes instance-level inference branches, category-level inference branches, generative adaptation branches, and a dynamic branch selector, which are used to dynamically select the optimal inference path based on scene information and output preliminary 6D pose and size estimation results. The post-processing module, connected to the hierarchical pose inference module, is used to perform multi-dimensional uncertainty quantification and industrial scene physical consistency verification, and outputs the final 6D pose, three-dimensional dimensions and confidence information. The communication interface module is connected to the post-processing module and is used to interact with external actuators for data exchange.

10. The 6D pose extraction device for a three-dimensional workpiece as described in claim 9, characterized in that, It also includes a closed-loop learning module, which is connected to the preprocessing module and the hierarchical pose inference module. This module receives the operation results returned by the external actuator. If the operation is successful, the pose estimation results and multimodal features of the workpiece are added to the corresponding template library or category library. If the operation fails, re-inference is triggered and the model parameters are optimized.

Images