Six degree of freedom object pose estimation system, method, and medium in industrial scenarios
By using multimodal data processing and simulation-driven data augmentation, the accuracy and robustness of six-DOF pose estimation are improved, and the problems of occlusion and reflection interference in industrial environments are solved by traditional methods, thus achieving efficient pose estimation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HUICUI INTELLIGENT TECH CO LTD
- Filing Date
- 2025-10-13
- Publication Date
- 2026-08-04
AI Technical Summary
Traditional six-DOF pose estimation methods are susceptible to interference from factors such as occlusion, reflection, and noise in industrial environments, have limited generalization ability, and rely on a large amount of real labeled data, resulting in unstable model performance in practical applications.
Employing multimodal data preprocessing, shared encoder feature extraction, contrastive learning and attention fusion, pose regression and rotation decoupling, and simulation-driven data augmentation modules, high-quality training data is generated through filtering and denoising, pixel-level spatial alignment, occlusion-aware feature fusion, Fresnel equation-based mirror material simulation, and multi-view sampling.
In scenarios with 70% occlusion or reflection, the pose estimation accuracy is improved by 12.8%, the rotation error is reduced by 21.4%, the robustness is improved, the dependence on real-world labeled data is reduced, and the needs of real-time industrial applications are met.
Smart Images
Figure CN121639783B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial automation technology, and more specifically, to a six-degree-of-freedom object pose estimation system, method, and medium in industrial scenarios. Background Technology
[0002] Traditional six-DOF pose estimation methods are mostly based on a single modality (such as RGB or point cloud), which are easily affected by factors such as occlusion, reflection, and noise in industrial environments. Existing methods such as PoseCNN, DenseFusion, and PVN3D, although performing well on public datasets, have limited generalization ability in complex industrial scenarios such as high reflection, strong occlusion, and stacked objects. Furthermore, the cost of annotating real data is high, and it is difficult to cover all extreme cases, resulting in unstable model performance in practical applications.
[0003] Therefore, the disadvantages of the existing technology are as follows:
[0004] 1. Sensitive to occlusion: Traditional methods have difficulty handling partial occlusion.
[0005] 2. Poor robustness to reflective surfaces: Reflective materials such as metals can cause image distortion and affect feature extraction;
[0006] 3. Relies on a large amount of real-world labeled data: Labeling 6-DoF poses is costly and time-consuming;
[0007] 4. Simple multimodal fusion: Early fusion methods failed to fully consider the complementarity and occlusion between modes. Summary of the Invention
[0008] The purpose of this invention is to provide a six-degree-of-freedom object pose estimation system, method, and medium for industrial scenarios, to solve the problems of inaccurate pose estimation and poor adaptability to reflective materials such as metals.
[0009] The first aspect of this invention provides a six-degree-of-freedom object pose estimation system for industrial scenarios, comprising:
[0010] The system comprises a multimodal data preprocessing module, a shared encoder feature extraction module, a contrastive learning and attention fusion module, a pose regression and rotation decoupling module, and a simulation-driven data augmentation module.
[0011] The multimodal data preprocessing module is used to receive and process the original RGB image and depth map, and output pixel-level spatially aligned RGB-D data pairs through filtering and noise reduction and coordinate projection transformation based on camera intrinsic parameters.
[0012] The shared encoder feature extraction module is used to extract features from aligned RGB-D data pairs and perform adaptive weighted fusion through a channel attention mechanism to output a shared feature that integrates appearance and geometric information.
[0013] The contrastive learning and attention fusion module is used to perform occlusion-aware feature fusion on the RGB-D data pair and the shared features through the occlusion probability map generated by the depth gradient, and introduces contrastive learning loss to constrain the feature space to output enhanced features that are robust to occlusion and appearance changes.
[0014] The pose regression and rotation decoupling module is used to obtain the enhanced features, and outputs quaternions and translation vectors representing rotation through the regression network head. After normalizing the quaternions, they are converted into rotation matrices. Finally, the rotation error is calculated using a differentiable loss function based on Lie algebra to complete the accurate estimation of the six-degree-of-freedom pose.
[0015] The simulation-driven data augmentation module is used to simulate object stacking and occlusion in a virtual environment through physical simulation, simulate metallic reflective materials based on Fresnel equations, and perform multi-view sampling and automatic annotation to generate synthetic RGB-D training data with precise pose labels.
[0016] In this scheme, the multimodal data preprocessing module is used to standardize, align, and register the RGB image and the depth map. Specifically, the depth map undergoes bilateral filtering for noise reduction, and pixel space calibration is performed using camera intrinsic parameters. This allows the depth information to be projected pixel-by-pixel onto the RGB image, calculated as follows:
[0017]
[0018] in, It is a three-dimensional point cloud coordinate, (f x ,f y (c) represents the camera's focal length. x ,c y Let be the coordinates of the main point, and D(u,v) be the depth value of that point on the depth image.
[0019] In this scheme, the shared encoder feature extraction module processes the RGB image and depth map through two branches respectively. Each branch consists of a convolutional layer + batch normalization + activation function, adopts a shared weight design, and performs channel attention fusion after feature extraction. The calculation formula is as follows:
[0020] F fused =σ(W c [F rgb ;F depth ]);
[0021] Among them, Ffused To share features, W c For learnable fusion weights, σ is the Sigmoid activation function, and F... rgb For RGB image features, F depth For depth map features, [;] indicates channel splicing.
[0022] In this scheme, the contrastive learning and attention fusion module estimates the occlusion probability map based on the depth gradient of the input depth map. This enhanced feature is then incorporated into feature fusion, and the calculation formula is as follows:
[0023]
[0024] Among them, F occ-aware For enhanced features, For depth gradient estimation of occlusion probability map, F fused For shared features, F rgb Using RGB image features, and combining contrastive learning, the clustering of objects in the feature space under different angles is constrained:
[0025]
[0026] in, For the total loss, To compare the losses, For pose loss, For style loss, α, β, and γ are the hyperparameter balancing factors of the loss term.
[0027] In this scheme, the pose regression and rotation decoupling module represents the rotation parameters as quaternions and performs unit normalization after the network output, as follows:
[0028]
[0029] q=(q w ,q x ,q y ,q z );
[0030] in, To return to the starting position, for L2 norm, The normalized quaternion is represented by Quat2Rot, which is the transformation function. The rotation matrix is then used to optimize the differentiable regression loss through Lie algebraic mapping, and the result is obtained using the logarithmic mapping function:
[0031]
[0032] in, For rotational losses, R * For the true rotation matrix, This represents the difference in the rotation matrix.
[0033] In this solution, the simulation-driven data augmentation module is used for realistic occlusion simulation, specular reflection modeling, and multi-camera angle sampling. Specifically, it includes: physical fall simulation to generate stacked and occlusion data, specular material modeling based on Fresnel model to simulate the reflection phenomenon under the reflective surface of metal, and multi-view sampling and automatic recording of pose labels to construct a training dataset.
[0034] A second aspect of the present invention provides a method for estimating the pose of a six-DOF object in an industrial setting, applicable to any of the six-DOF object pose estimation systems described in the present invention, wherein the method includes the following steps:
[0035] Obtain the original RGB image and depth map, and preprocess them to obtain aligned RGB-D data pairs;
[0036] The RGB-D data pairs are subjected to RGB and depth feature extraction using a weight-sharing dual-branch convolutional neural network, and the RGB and depth features are fused using a channel attention mechanism to obtain shared features.
[0037] Enhanced features are obtained by fusing occlusion-aware features of the RGB-D data pairs and the shared features using an occlusion probability map generated from depth gradients;
[0038] Pose regression and decoupling are performed based on the enhanced features. The regression network head outputs quaternions representing rotation and translation vectors. After normalizing the quaternions, they are converted into rotation matrices. Finally, the rotation error is calculated using a differentiable loss function based on Lie algebras, thus completing the accurate estimation of the six-degree-of-freedom pose.
[0039] In this scheme, the method further includes loss calculation during the training phase, and the calculation formula is as follows:
[0040]
[0041] in, For rotational losses, R * For the true rotation matrix, This represents the difference in the rotation matrix.
[0042] In this scheme, the method further includes generating RGB-D training data before model training. In this process, physical simulation is used in a virtual environment to simulate object stacking and occlusion, the Fresnel equation is used to simulate reflective metallic materials, and multi-view sampling and automatic annotation are performed to generate synthetic RGB-D training data with accurate pose labels.
[0043] A third aspect of the present invention provides a computer-readable storage medium comprising a program for a six-degree-of-freedom object pose estimation method in an industrial setting, wherein when the program is executed by a processor, it implements the steps of the six-degree-of-freedom object pose estimation method in an industrial setting as described in any of the preceding claims.
[0044] This invention discloses a six-degree-of-freedom object pose estimation system, method, and medium for industrial scenarios, with the following beneficial effects:
[0045] 1. Improved accuracy: In scenarios with 70% occlusion or reflection, the ADD index is improved by 12.8%, and the rotation error is reduced by 21.4%.
[0046] 2. Strong generalization ability, with zero-shot generalization performance improved by more than 15% compared to GDR-Net;
[0047] 3. High robustness, especially against deep noise, occlusion, and reflection.
[0048] 4. Highly practical: In actual applications, it has been deployed in the UR5 robotic arm + Zivid camera system, with inference latency <130ms and processing speed of 8FPS, meeting the requirements of real-time industrial applications.
[0049] 5. High data efficiency, especially with simulation data augmentation significantly reducing reliance on real labeled data. Attached Figure Description
[0050] Figure 1 This invention illustrates a schematic diagram of the composition of a six-degree-of-freedom object pose estimation system in an industrial setting.
[0051] Figure 2 The diagram shows the multimodal fusion network structure of a six-degree-of-freedom object pose estimation system in an industrial setting according to the present invention.
[0052] Figure 3 This invention illustrates the comparative learning training mechanism of a six-DOF object pose estimation system in an industrial setting.
[0053] Figure 4 The following is a flowchart illustrating the simulation data generation process of a six-degree-of-freedom object pose estimation system in an industrial setting according to the present invention.
[0054] Figure 5 The following is a flowchart illustrating the estimation process of a six-degree-of-freedom object pose estimation system in an industrial setting according to the present invention.
[0055] Figure 6 A block diagram of a six-degree-of-freedom object pose estimation system for an industrial scenario is shown. Detailed Implementation
[0056] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.
[0057] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0058] Figure 1 A schematic diagram of the composition of a six-degree-of-freedom object pose estimation system in an industrial setting is shown.
[0059] like Figure 1 As shown, this invention discloses a six-DOF object pose estimation system for industrial scenarios, comprising:
[0060] The system comprises a multimodal data preprocessing module, a shared encoder feature extraction module, a contrastive learning and attention fusion module, a pose regression and rotation decoupling module, and a simulation-driven data augmentation module.
[0061] The multimodal data preprocessing module is used to receive and process the original RGB image and depth map, and output pixel-level spatially aligned RGB-D data pairs through filtering and noise reduction and coordinate projection transformation based on camera intrinsic parameters.
[0062] The shared encoder feature extraction module is used to extract features from aligned RGB-D data pairs and perform adaptive weighted fusion through a channel attention mechanism to output a shared feature that integrates appearance and geometric information.
[0063] The contrastive learning and attention fusion module is used to perform occlusion-aware feature fusion on the RGB-D data pair and the shared features through the occlusion probability map generated by the depth gradient, and introduces contrastive learning loss to constrain the feature space to output enhanced features that are robust to occlusion and appearance changes.
[0064] The pose regression and rotation decoupling module is used to obtain the enhanced features, and outputs quaternions and translation vectors representing rotation through the regression network head. After normalizing the quaternions, they are converted into rotation matrices. Finally, the rotation error is calculated using a differentiable loss function based on Lie algebra to complete the accurate estimation of the six-degree-of-freedom pose.
[0065] The simulation-driven data augmentation module is used to simulate object stacking and occlusion in a virtual environment through physical simulation, simulate metallic reflective materials based on Fresnel equations, and perform multi-view sampling and automatic annotation to generate synthetic RGB-D training data with precise pose labels.
[0066] It should be noted that, in this embodiment, the six-DOF object pose estimation system for industrial scenarios of the present invention consists of five core modules, namely: a multimodal data preprocessing module, a shared encoder feature extraction module, a contrastive learning and attention fusion module, a pose regression and rotation decoupling module, and a simulation-driven data augmentation module. Each module is both independent and tightly coupled with the others, together constructing a complete end-to-end pose estimation framework driven by multiple sources of data.
[0067] According to an embodiment of the present invention, the multimodal data preprocessing module is used to standardize, align, and register the RGB image and the depth map. Specifically, the depth map undergoes bilateral filtering for noise reduction, and pixel space calibration is performed using camera intrinsic parameters. This allows for pixel-by-pixel projection of the depth information onto the RGB image, calculated as follows:
[0068]
[0069] in, It is a three-dimensional point cloud coordinate, (f x ,f y (c) represents the camera's focal length. x ,c y Let be the coordinates of the main point, and D(u,v) be the depth value of that point on the depth image.
[0070] It should be noted that, in this embodiment, the multimodal data preprocessing module is responsible for standardizing, aligning, and registering the RGB image and the depth map. Since the two types of images often come from different sensors in industrial environments, their spatial alignment suffers from displacement errors. Therefore, this module first performs bilateral filtering denoising on the depth map, and then performs pixel spatial calibration using camera intrinsic parameters, so that the depth information is projected pixel-by-pixel onto the RGB image, denoted as:
[0071]
[0072] in, It is a three-dimensional point cloud coordinate, (f x fy (c) represents the camera's focal length. x c y Let be the coordinates of the main point, and D(u,v) be the depth value of that point on the depth image.
[0073] According to an embodiment of the present invention, the shared encoder feature extraction module processes the RGB image and the depth map through two branches, respectively. Each branch consists of a convolutional layer + batch normalization + activation function, adopts a shared weight design, and performs channel attention fusion after feature extraction. The calculation formula is as follows:
[0074] F fused =σ(W c [F rgb ;F depth ]);
[0075] Among them, F fused To share features, W c For learnable fusion weights, σ is the Sigmoid activation function, and F... rgb For RGB image features, F depth For depth map features, [;] indicates channel splicing.
[0076] It should be noted that, in this embodiment, as Figure 2 As shown, the diagram illustrates the multimodal fusion network structure. The shared encoder feature extraction module processes the RGB image and depth map through two branches, each consisting of a convolutional layer, batch normalization, and an activation function (such as Swish) to ensure that low-level features retain spatial information while remaining robust to illumination changes. This module employs a shared weight design to enhance modal consistency and performs channel attention fusion after feature extraction, as shown in the following formula.
[0077] F fused =σ(W c [F rgb ;F depth ]);
[0078] Among them, F fused To share features, W c For learnable fusion weights, σ is the Sigmoid activation function, and F... rgb For RGB image features, F depth For depth map features, [;] indicates channel splicing.
[0079] According to an embodiment of the present invention, the contrastive learning and attention fusion module estimates the occlusion probability map based on the depth gradient of the input depth map. This enhanced feature is then incorporated into feature fusion, and the calculation formula is as follows:
[0080]
[0081] Among them, F occ-aware For enhanced features, For depth gradient estimation of occlusion probability map, F fused For shared features, F rgb Using RGB image features, and combining contrastive learning, the clustering of objects in the feature space under different angles is constrained:
[0082]
[0083] in, For the total loss, To compare the losses, For pose loss, Let α, β, and γ be the style loss, and let α, β, and γ be the hyperparameter balancing factors of the loss term.
[0084] It should be noted that, in this embodiment, as Figure 3 As shown in the diagram, which illustrates the contrastive learning training mechanism, this invention further introduces an occlusion-aware attention mechanism, which estimates the occlusion probability map based on the depth gradient of the input depth map. This enhanced feature is then incorporated into feature fusion, and the calculation formula is as follows:
[0085]
[0086] In this way, when images are occluded, the system can adaptively suppress invalid features of the RGB image and strengthen the geometric representation. At the same time, by combining contrastive learning, it can constrain the clustering of objects in the feature space from different angles.
[0087]
[0088] in, For the total loss, To compare the losses, For pose loss, For style loss, α, β, and γ are the hyperparameter balancing factors of the loss term.
[0089] According to an embodiment of the present invention, the pose regression and rotation decoupling module represents the rotation parameters as quaternions and performs unit normalization after the network output, as follows:
[0090]
[0091] q=(q w ,q x ,q y qz );
[0092] in, To return to the starting position, for L2 norm, The normalized quaternion is represented by Quat2Rot, which is the transformation function. The rotation matrix is then used to optimize the differentiable regression loss through Lie algebraic mapping, and the result is obtained using the logarithmic mapping function:
[0093]
[0094] in, For rotational losses, R * For the true rotation matrix, This represents the difference in the rotation matrix.
[0095] It should be noted that, in this embodiment, the present invention innovatively uses the quaternion q = (q w q x q y q z This indicates that unit normalization is performed after the network output:
[0096]
[0097] in, To return to the starting position, for L2 norm, The normalized quaternion is represented by Quat2Rot, which is the transformation function. The rotation matrix is then used to optimize the differentiable regression loss through Lie algebraic mapping, and the result is obtained using the logarithmic mapping function: in, For rotational losses, R * For the true rotation matrix, This represents the difference in the rotation matrix.
[0098] According to an embodiment of the present invention, the simulation-driven data augmentation module is used for realistic occlusion simulation, specular reflection modeling, and multi-camera angle sampling. Specifically, it includes: generating stacked and occlusion data through physical fall simulation, modeling specular materials based on Fresnel models to simulate reflection phenomena under metallic reflective surfaces, and sampling from multiple perspectives and automatically recording pose labels to construct a training dataset.
[0099] It should be noted that, in this embodiment, as Figure 4As shown, the simulation data generation flowchart is displayed. The simulation-driven data augmentation module is also one of the key innovations of this invention. It has functions such as realistic occlusion simulation, specular reflection modeling, and multi-camera angle sampling. The generation process includes three stages: (1) physical fall simulation to generate stacked and occlusion data; (2) specular material modeling based on Fresnel model to simulate the reflection phenomenon under the reflective surface of metal; (3) multi-view sampling and automatic recording of pose labels to build a training dataset.
[0100] Furthermore, in this embodiment, as Figure 5 The diagram shows the estimation flowchart of the system. This invention undergoes extensive experiments on multiple public and self-built datasets to verify its robustness and generalization ability. Training and testing are performed on YCB-Video, T-LESS, and our self-built industrial reflective occlusion dataset (IND-Reflect-OCC), comparing it with several current mainstream methods, including PoseCNN, DenseFusion, GDR-Net, and PVN3D. Evaluation metrics include ADD (Average Distance of Model Points), ADI (Average Distance of Indistinguishable Points), rotation error, and translation error.
[0101]
[0102] in, Using the point set of the object model, experiments show that in scenes with 70% occlusion or metallic reflection, this system improves the ADD accuracy by an average of 12.8% and reduces the rotation error by about 21.4%. It remains stable under deep noise perturbation conditions. After introducing simulation data, the model still maintains high accuracy when trained with a small amount of real data. Its zero-shot generalization performance is more than 15% higher than GDR-Net, proving the effectiveness of the data augmentation module in synthesizing realistic consistency.
[0103] This invention has been engineered and implemented in the following industrial applications: (1) Automatic sorting system: accurately grabs reflective metal parts by estimating the six degrees of freedom pose in real time; (2) Robot assembly guidance: even when the assembly parts are partially obscured, their precise posture can still be calculated for insertion positioning; (3) Vision inspection system: measures the position and orientation of stacked workpieces to improve the false detection rate of the production line. Especially in real factory deployment cases, the system is integrated on the UR5 robotic arm and uses a Zivid RGB-D camera to collect images in real time. The inference latency of the whole system is less than 130ms, supports an 8FPS processing rate, and can be applied to continuous operation environments on the production line.
[0104] Meanwhile, this invention can be extended to more dimensions: (1) Multi-object joint pose estimation: In the future, a graph neural network can be constructed to model the spatial relationship between multiple objects, thereby improving the robustness of densely stacked scenes; (2) Introduction of time series information: The continuity between video frames is integrated, and the problem of instantaneous occlusion misidentification is alleviated by using LSTM or Transformer structure; (3) Self-supervised training mechanism: Through simulated closed-loop control, the robot actually performs pose feedback for error backpropagation, thereby achieving online adaptive correction; (4) Lightweight deployment optimization: Adapt to edge deployment environments, such as ARM architecture or Jetson platform, and further prune and compress network parameters and computational load.
[0105] Figure 6 The flowchart of a six-degree-of-freedom object pose estimation method in an industrial setting according to the present invention is shown.
[0106] like Figure 6 As shown, this invention discloses a method for estimating the pose of a six-DOF object in an industrial setting, applicable to a six-DOF object pose estimation system in an industrial setting as described in any of the above claims. The method includes the following steps:
[0107] S602, acquire the original RGB image and depth map, perform preprocessing to obtain aligned RGB-D data pairs;
[0108] S604, RGB features and depth features are extracted from the RGB-D data pair through a weight-sharing dual-branch convolutional neural network, and the RGB features and depth features are fused through a channel attention mechanism to obtain shared features;
[0109] S606, an enhanced feature is obtained by fusing the occlusion perception features of the RGB-D data pair and the shared features using the occlusion probability map generated by the depth gradient;
[0110] S608, pose regression and decoupling are performed based on the enhanced features. Specifically, the quaternion representing rotation and the translation vector are output by the regression network head. After normalization, the quaternion is converted into a rotation matrix. Finally, the rotation error is calculated using a differentiable loss function based on Lie algebra to complete the accurate estimation of the six-degree-of-freedom pose.
[0111] According to an embodiment of the present invention, the method further includes performing loss calculation during the training phase, the calculation formula being as follows:
[0112]
[0113] in, For rotational losses, R * For the true rotation matrix, This represents the difference in the rotation matrix.
[0114] It should be noted that, in this embodiment, ...
[0115] According to an embodiment of the present invention, the method further includes generating RGB-D training data before model training, wherein object stacking and occlusion are simulated by physical simulation in a virtual environment, metallic reflective materials are simulated based on Fresnel equations, and multi-view sampling and automatic annotation are performed to generate synthetic RGB-D training data with accurate pose labels.
[0116] It should be noted that, in this embodiment, the specific implementation of the six-degree-of-freedom object pose estimation method in an industrial scenario disclosed in this application is consistent with the content of the six-degree-of-freedom object pose estimation system in the industrial scenario described above, and therefore will not be repeated in this embodiment.
[0117] A third aspect of the present invention provides a computer-readable storage medium comprising a program for a six-degree-of-freedom object pose estimation method in an industrial setting. When executed by a processor, the program implements the steps of the six-degree-of-freedom object pose estimation method in an industrial setting as described in any of the preceding claims.
[0118] This invention discloses a six-degree-of-freedom object pose estimation system, method, and medium for industrial scenarios, which can improve estimation accuracy. In scenarios with 70% occlusion or reflection, the ADD index is improved by 12.8%, the rotation error is reduced by 21.4%, and it has strong robustness to depth noise, occlusion, and reflection, making it highly practical.
[0119] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0120] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0121] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0122] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0123] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
Claims
1. A six-degree-of-freedom object pose estimation system for industrial scenarios, characterized in that, include: The system comprises a multimodal data preprocessing module, a shared encoder feature extraction module, a contrastive learning and attention fusion module, a pose regression and rotation decoupling module, and a simulation-driven data augmentation module. The multimodal data preprocessing module is used to receive and process the original RGB image and depth map, and output pixel-level spatially aligned RGB-D data pairs through filtering and noise reduction and coordinate projection transformation based on camera intrinsic parameters. The shared encoder feature extraction module is used to extract features from aligned RGB-D data pairs and perform adaptive weighted fusion through a channel attention mechanism to output a shared feature that integrates appearance and geometric information. The contrastive learning and attention fusion module is used to perform occlusion-aware feature fusion on the RGB-D data pair and the shared features through the occlusion probability map generated by the depth gradient, and introduces contrastive learning loss to constrain the feature space to output enhanced features that are robust to occlusion and appearance changes. The pose regression and rotation decoupling module is used to obtain the enhanced features, and outputs quaternions and translation vectors representing rotation through the regression network head. After normalizing the quaternions, they are converted into rotation matrices. Finally, the rotation error is calculated using a differentiable loss function based on Lie algebra to complete the accurate estimation of the six-degree-of-freedom pose. The simulation-driven data augmentation module is used to simulate object stacking and occlusion in a virtual environment through physical simulation, simulate metallic reflective materials based on Fresnel equations, and perform multi-view sampling and automatic annotation to generate synthetic RGB-D training data with precise pose labels.
2. The six-degree-of-freedom object pose estimation system in an industrial setting according to claim 1, characterized in that, The multimodal data preprocessing module is used to standardize, align, and register the RGB image and the depth map. Specifically, it performs bilateral filtering for noise reduction on the depth map and performs pixel-space calibration using camera intrinsic parameters, enabling pixel-by-pixel projection of the depth information onto the RGB image. The calculation formula is as follows: in, It is a three-dimensional point cloud coordinate, (f x ,f y (c) represents the camera's focal length. x ,c y Let be the coordinates of the main point, and D(u,v) be the depth value of that point on the depth image.
3. The six-degree-of-freedom object pose estimation system in an industrial setting according to claim 2, characterized in that, The shared encoder feature extraction module processes the RGB image and depth map through two branches, respectively. Each branch consists of a convolutional layer, batch normalization, and an activation function. It adopts a shared weight design and performs channel attention fusion after feature extraction. The calculation formula is as follows: F fused = σ(Wc[F rgb ; F depth ]) ; where F fused is the shared feature, W c is the learnable fusion weight, σ is the Sigmoid activation function, F rgb is the RGB image feature, F depth is the depth image feature, and [; ] denotes channel concatenation.
4. The six-degree-of-freedom object pose estimation system in an industrial setting according to claim 3, characterized in that, The contrastive learning and attention fusion module estimates the occlusion probability map based on the depth gradient of the input depth map. This enhanced feature is then incorporated into feature fusion, and the calculation formula is as follows: Among them, F occ-aware For enhanced features, For depth gradient estimation of occlusion probability map, F fused For shared features, F rgb Using RGB image features, and combining contrastive learning, the clustering of objects in the feature space under different angles is constrained: in, For the total loss, To compare the losses, For pose loss, For style loss, α, β, and γ are the hyperparameter balancing factors of the loss term.
5. A six-degree-of-freedom object pose estimation system in an industrial setting according to claim 4, characterized in that, The pose regression and rotation decoupling module represents the rotation parameters as quaternions and performs unit normalization on the network output, as follows: q=(q w ,q x ,q y ,q z ); in, To return to the starting position, for L2 norm, The normalized quaternion is represented by Quat2Rot, which is the transformation function. The rotation matrix is then used to optimize the differentiable regression loss through Lie algebraic mapping, and the result is obtained using the logarithmic mapping function: in, For rotational losses, R * For the true rotation matrix, This represents the difference in the rotation matrix.
6. The six-degree-of-freedom object pose estimation system in an industrial setting according to claim 5, characterized in that, The simulation-driven data augmentation module is used for realistic occlusion simulation, specular reflection modeling, and multi-camera angle sampling. Specifically, it includes: physical fall simulation to generate stacked and occlusion data, specular material modeling based on Fresnel model to simulate reflection phenomena under metallic reflective surfaces, and multi-view sampling and automatic recording of pose labels to construct a training dataset.
7. A method for estimating the pose of a six-degree-of-freedom object in an industrial setting, characterized in that, A six-DOF object pose estimation system for an industrial scenario, as described in any one of claims 1-6, wherein the method comprises the following steps: Obtain the original RGB image and depth map, and preprocess them to obtain aligned RGB-D data pairs; The RGB-D data pairs are subjected to RGB and depth feature extraction using a weight-sharing dual-branch convolutional neural network, and the RGB and depth features are fused using a channel attention mechanism to obtain shared features. Enhanced features are obtained by fusing occlusion-aware features of the RGB-D data pairs and the shared features using an occlusion probability map generated from depth gradients; Pose regression and decoupling are performed based on the enhanced features. The regression network head outputs quaternions representing rotation and translation vectors. After normalizing the quaternions, they are converted into rotation matrices. Finally, the rotation error is calculated using a differentiable loss function based on Lie algebras, thus completing the accurate estimation of the six-degree-of-freedom pose.
8. The method for estimating the pose of a six-degree-of-freedom object in an industrial setting according to claim 7, characterized in that, The method also includes loss calculation during the training phase, as shown in the following formula: in, For rotational losses, R * For the true rotation matrix, This represents the difference in the rotation matrix.
9. The method for estimating the pose of a six-degree-of-freedom object in an industrial setting according to claim 8, characterized in that, The method also includes generating RGB-D training data before model training. In this process, physical simulation is used to simulate object stacking and occlusion in a virtual environment, the reflective material of metal is simulated based on Fresnel equations, and multi-view sampling and automatic annotation are performed to generate synthetic RGB-D training data with accurate pose labels.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a program for a six-degree-of-freedom object pose estimation method in an industrial setting. When the program is executed by a processor, it implements the steps of a six-degree-of-freedom object pose estimation method in an industrial setting as described in any one of claims 7 to 9.