Six-DOF pose determination method, electronic equipment and storage medium

Abstract

This application provides a six-degree-of-freedom pose determination method, electronic device, and storage medium. The method includes: extracting features from a two-dimensional image and a depth image to obtain multi-scale semantic feature maps and multi-scale geometric feature maps; performing cross-modal attention fusion on the multi-scale semantic feature maps and multi-scale geometric feature maps to obtain a fused feature map; a lightweight detection unit outputting the initial poses and uncertainties of each object in the target scene based on the fused feature map, and determining the target pose of each object based on each initial pose, the uncertainty of each initial pose, and a preset threshold; and determining each target object based on the target pose of each object and using a cost function that includes penetration multiplication and the height of the supporting plane. This significantly improves the robustness of pose estimation in complex scenes, achieves a balance between high accuracy and real-time performance, and effectively eliminates pose conflicts in occluded scenes.

Description

Technical Field

[0001] This application relates to the technical field of computer vision, and more specifically, to a six-degree-of-freedom pose determination method, electronic device, and storage medium. Background Technology

[0002] Six-degree-of-freedom (6D) pose estimation refers to the accurate calculation of an object's three-dimensional translation and rotation in three-dimensional space. It is a core perception capability for industrial operations such as autonomous grasping, precision assembly, and intelligent palletizing of robots. Its accuracy, robustness, and real-time performance directly determine the safety, success rate, and production line cycle time of the robotic arm's movements.

[0003] Existing 6D pose estimation methods include corresponding point-based methods, template matching-based methods, and end-to-end deep learning methods. However, these methods are highly dependent on the quality of the input modality and lack robustness in weak texture or highly reflective scenes. Furthermore, they have high computational loads, making it difficult to balance real-time performance with high accuracy. In addition, they cannot provide accurate grab points in complex environments with multiple object occlusions and cluttered scenes. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a six-degree-of-freedom pose determination method, electronic device, and storage medium, thereby improving the accuracy of six-degree-of-freedom pose determination.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a six-degree-of-freedom pose determination method, the method comprising: Acquire 2D and depth images of the target scene; Feature extraction is performed on the two-dimensional image and the depth image respectively to obtain multi-scale semantic feature maps and multi-scale geometric feature maps; Cross-modal attention fusion is performed on the multi-scale semantic feature map and the multi-scale geometric feature map to obtain a fused feature map, which contains both semantic and geometric features. The fused feature map is input to the lightweight detection unit, which outputs the initial pose of each object in the target scene and the uncertainty of each initial pose based on the fused feature map. Based on each initial pose, the uncertainty of each initial pose, and a preset threshold, the target pose of each object is determined. Each target object is determined based on its target pose and using a cost function that includes penetration multiplication and the height of the supporting plane, wherein the target object is at least a portion of the objects.

[0006] Optionally, the step of extracting features from the two-dimensional image and the depth image to obtain multi-scale semantic feature maps and multi-scale geometric feature maps includes: The depth image is converted into a 3D point cloud based on the camera's intrinsic parameter matrix. Background filtering and outlier removal are performed on the 3D point cloud to obtain the removed point cloud. The removed point cloud is then transformed based on the calibration matrix between the robot coordinate system and the camera coordinate system to obtain the target point cloud data in the robot coordinate system. The two-dimensional image is input into a deep encoder, which extracts semantic features from the two-dimensional image to obtain multi-scale semantic feature maps. Each scale semantic feature map includes at least: color, texture, and edge. The target point cloud data is input into a sparse convolutional network, which extracts geometric features from the target point cloud data to obtain multi-scale geometric feature maps. Each scale of the geometric feature map contains the normal vector and curvature information of the local neighborhood.

[0007] Optionally, the step of the lightweight detection unit outputting the initial poses of each object in the target scene and the uncertainty of each initial pose based on the fused feature map includes: The lightweight detection head in the lightweight detection unit performs target detection on the fused feature map, outputs the target detection box, initial pose and category score of each object, and performs pose regression processing on the initial pose of each object based on the heteroscedasticity regression loss function, outputting the uncertainty of each initial pose.

[0008] Optionally, the cross-modal attention fusion of the multi-scale semantic feature maps and the multi-scale geometric feature maps to obtain a fused feature map includes: A cross-attention mechanism is used to obtain the initial fused feature maps at each scale by combining the semantic feature maps and the geometric feature maps at each scale. The initial fused feature maps at each scale are residually connected with the geometric feature maps at each scale to obtain intermediate fused feature maps at each scale. The intermediate fused feature maps at each scale are upsampled and multi-scale stitched together to obtain the fused feature maps.

[0009] Optionally, the semantic feature maps and geometric feature maps at each scale are subjected to a cross-attention mechanism to obtain initial fused feature maps at each scale, including: Using the geometric features in the geometric feature map as query factors and the semantic features in the semantic feature map as keys and values, we calculate attention weights and use the attention weights to perform a weighted summation of the values ​​to obtain the initial fused feature map.

[0010] Optionally, determining the target pose of each object based on each initial pose, the uncertainty of each initial pose, and a preset threshold includes: If the uncertainty of the initial pose is less than the preset threshold, then the initial pose is taken as the target pose; If the uncertainty of the initial pose is greater than or equal to the preset threshold, then a local fine-grained matching process is performed based on the initial pose and the object model of the object to obtain the target pose of the object.

[0011] Optionally, the step of performing local fine-tuning matching based on the initial pose and the object model to obtain the target pose of the object includes: The object model is projected into a three-dimensional space based on the initial pose to obtain the projected point cloud data. The projected point cloud data is then expanded to obtain the point cloud data of the region of interest. An iterative nearest-point algorithm with normal vector constraints is used to iteratively register the point cloud data of the region of interest with the point cloud of the object model. When the iteration ends, the target pose of the object is obtained.

[0012] Optionally, each target object is determined based on its target pose and using a cost function that includes penetration multiplication and the height of the supporting plane. The target object is at least a subset of the objects, including: Based on the target pose of the current object and the target poses of other objects, determine the distance between the current object and other objects, as well as the coordinates of the lowest point of the bottom surface of the current object, wherein the current object is any object; Based on the distance, the coordinates of the lowest point of the bottom surface of the current object, and the height of the supporting plane of the target scene, determine the pose rationality between the current object and the other objects; Based on the pose rationality and whether there is embedding between the current object and the other objects, it is determined whether the current object and the other objects are culled; If not, then the current object and all other objects are considered as target objects; if yes, then the current object and all other objects are discarded.

[0013] Secondly, embodiments of this application also provide a six-degree-of-freedom pose determination device, the device comprising: The acquisition module is used to acquire two-dimensional images and depth images of the target scene; The feature extraction module is used to extract features from the two-dimensional image and the depth image respectively, to obtain multi-scale semantic feature maps and multi-scale geometric feature maps; The fusion module is used to perform cross-modal attention fusion on the multi-scale semantic feature map and the multi-scale geometric feature map to obtain a fused feature map, which contains semantic features and geometric features. The detection module is used to input the fused feature map to the lightweight detection unit, which outputs the initial pose of each object in the target scene and the uncertainty of each initial pose based on the fused feature map, and determines the target pose of each object based on each initial pose, the uncertainty of each initial pose, and a preset threshold. A determination module is used to determine each target object based on the target pose of each of the objects and using a cost function that includes penetration multiplication and the height of the supporting plane, wherein the target object is at least a portion of the objects.

[0014] Thirdly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the steps of the six-degree-of-freedom pose determination method described in the first aspect.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the six-degree-of-freedom pose determination method described in the first aspect.

[0016] The beneficial effects of this application are: This application provides a six-degree-of-freedom pose determination method, electronic device, and storage medium. It extracts features from two-dimensional and depth images to obtain multi-scale semantic and geometric feature maps. Then, cross-modal attention fusion is used to fuse these multi-scale semantic and geometric feature maps, generating a fused feature map representing the structure and semantics of the target scene. This ensures that the fused feature map no longer relies solely on texture information, allowing pose estimation to be performed based on geometric features even in industrial scenarios with metallic reflections, weak textures, or no texture. This effectively solves the problem of traditional vision technologies failing on special objects. A lightweight detection unit then outputs the initial poses and uncertainties of each object based on the fused feature map. The initial poses are then corrected based on the uncertainties to obtain the target poses of each object. This is a coarse-to-fine two-level cascaded architecture based on dynamic confidence evaluation. The lightweight detection unit achieves rapid initial localization, and further fine-grained matching is performed when uncertainty is insufficient to obtain the target pose. This avoids high global computational load, ensuring the overall processing frame rate meets real-time requirements while improving accuracy for key areas. After determining the target pose, by embedding resolvable physical constraint priors, the robustness of target discrimination and the physical authenticity of pose are significantly improved in challenging scenarios such as severe occlusion, missing textures or sensor noise interference, thus avoiding grasping collisions or empty grasping problems caused by pose conflicts. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a six-degree-of-freedom pose determination method provided in an embodiment of this application; Figure 2 A flowchart illustrating the second six-degree-of-freedom pose determination method provided in this application embodiment; Figure 3 A flowchart illustrating the third six-degree-of-freedom pose determination method provided in this application embodiment; Figure 4 A flowchart illustrating the fourth six-degree-of-freedom pose determination method provided in this application embodiment; Figure 5 A flowchart illustrating the fifth six-degree-of-freedom pose determination method provided in this application embodiment; Figure 6 A schematic diagram of a visualization result provided for an embodiment of this application; Figure 7 A schematic diagram of an apparatus for a six-degree-of-freedom pose determination method provided in an embodiment of this application; Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0020] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] Optionally, the six-degree-of-freedom pose determination method provided in this application embodiment can be applied to an electronic device, such as a mobile phone, tablet computer, laptop computer, handheld computer, desktop computer, or other terminal device with computing power and display function, or it can be a server. The electronic device can also be an electronic device embedded in a robot. Specifically, it can be applied to applications in terminal devices, such as mobile phone applications (APPs) and computer application systems.

[0023] Figure 1 This is a flowchart illustrating a six-degree-of-freedom pose determination method provided in an embodiment of this application. The execution subject of this method is as described above: electronic device. Figure 2As shown, the method includes: S101. Obtain a two-dimensional image and a depth image of the target scene.

[0024] Specifically, the 2D image and depth image are simultaneously acquired using a Red-Green-Blue (RGB) camera triggered at the same timestamp. The RGB camera has a built-in synchronization mechanism to ensure strict alignment of the 2D and depth images on the timeline, avoiding misalignment caused by motion blur or dynamic scene changes. The 2D image can be a color RGB image, and the depth image is a corresponding viewpoint depth map simultaneously acquired by the RGB camera. Furthermore, both the 2D and depth images are acquired using the robot's coordinate system as a reference.

[0025] The target scenario can be, for example, a material basket stacking scenario, a shelf placement scenario, or a dynamic operation scenario, in which an object is placed that can be grasped by a robot.

[0026] S102. Perform feature extraction on the two-dimensional image and the depth image respectively to obtain multi-scale semantic feature maps and multi-scale geometric feature maps.

[0027] Among them, semantic feature maps at each scale can be used To represent, geometric feature maps at various scales can be used To indicate, among which, This represents the network layer. Semantic feature maps and geometric feature maps have a one-to-one correspondence in the scale dimension, such as the first... Layer semantic feature map and the first The layer set feature maps have the same spatial resolution and channel number mapping relationship, which provides a foundation for subsequent cross-modal attention fusion.

[0028] Optionally, semantic features can be extracted from the 2D image to obtain multi-scale semantic feature maps. These maps maintain semantic consistency through cross-scale connections, ensuring comparable representational strength for the same semantic feature across different scales. Each scale's semantic feature map can contain different semantic features, such as color, texture, and edges. Geometric features can also be extracted from the depth image to obtain multi-scale geometric feature maps. Each scale's geometric feature map can include different geometric features, such as normal vectors, curvature, and point cloud density. The geometric feature maps at each scale explicitly preserve the relative geometric relationships in 3D Euclidean space, avoiding structural ambiguity caused by projection distortion.

[0029] S103. Perform cross-modal attention fusion on the multi-scale semantic feature map and the multi-scale geometric feature map to obtain the fused feature map.

[0030] This fused feature map contains both semantic and geometric features, and can maintain discriminative power even in textureless regions by relying on the aggregated features. For example, the fused feature map can use... To express.

[0031] Optionally, a geometry-guided semantic modulation fusion method can be used to perform cross-modal attention fusion on multi-scale semantic feature maps and multi-scale geometric feature maps. Specifically, geometric feature maps can be used as the decision backbone for space and structure, and semantic feature maps can be used as content modifiers for appearance and category, through learnable cross-modal attention fusion. The geometric structure can provide an immutable physical skeleton, while semantic features serve only as flexible modifications, and the modification magnitude of semantic features is gated by geometric confidence. While maintaining the rigidity of three-dimensional geometric constraints, robust semantic priors are injected to generate fused feature maps that combine structural fidelity and semantic discriminative power.

[0032] S104. Input the fused feature map into the lightweight detection unit. The lightweight detection unit outputs the initial pose of each object in the target scene and the uncertainty of each initial pose based on the fused feature map. Based on each initial pose, the uncertainty information of each initial pose and the preset threshold, the target pose of each object is determined.

[0033] Optionally, the lightweight detection unit can first perform a coarse estimation based primarily on the geometric features in the fused features to determine the initial pose of each object, and then quickly regress each initial pose to obtain the uncertainty of each initial pose. When the uncertainty of the initial pose obtained by the coarse estimation is less than a preset threshold, local refinement is activated, and the initial pose of the object is corrected using a local ICP registration method based on the point cloud to obtain the target pose of the object.

[0034] S105. Determine each target object and its target pose based on the target pose of each object and using a cost function that includes penetration multiplication and the height of the supporting plane.

[0035] The target object is at least a portion of the objects.

[0036] Optionally, after obtaining the target pose set of each object, a joint geometric constraint model based on multidimensional physical prior knowledge is introduced to perform scene-level rationality verification of the target poses. Based on the verification results, each object is then selected to obtain the target objects. After obtaining each target object, the pose calculated in step S104 is used as the target pose of that target object.

[0037] The cost function, which includes penetration multiplication and the height of the supporting plane, contains two types of coupled constraint terms. One type is spatial penetration avoidance constraint, which specifically models the non-penetrating relationship between the object's 3D envelope and known static structures in the scene, and quantifies geometric conflicts in continuous space by constructing a differentiable penetration penalty term. The other type is support relationship adaptation constraint, which characterizes the physical support compatibility between the object's bottom geometry and the supporting surface below it. By introducing a pose conflict resolution algorithm based on physical constraints, combined with prior knowledge of the scene such as the table plane and the frame boundary, and a collision detection model, the penetration or floating phenomena between objects are corrected, ensuring that the target poses of each output target object are strictly consistent in physical space and can be grasped.

[0038] In this embodiment, feature extraction is performed on 2D and depth images respectively to obtain multi-scale semantic feature maps and multi-scale geometric feature maps. Then, cross-modal attention fusion is used to fuse the multi-scale semantic feature maps and multi-scale geometric feature maps to generate a fused feature map representing the structure and semantics of the target scene. This makes the obtained fused feature map no longer solely dependent on texture information, so that pose estimation can still be performed based on geometric features in industrial scenes with metal reflections, weak textures, or no textures, effectively solving the problem that traditional vision technology is prone to failure on special objects. Then, the lightweight detection unit outputs the initial pose of each object and the uncertainty of each initial pose based on the fused feature map, and corrects the initial pose based on the uncertainty to obtain the target pose of each object. That is, a coarse and fine two-level cascaded architecture based on dynamic confidence evaluation is used to achieve rapid initial localization through the lightweight detection unit. When the uncertainty is insufficient, fine matching is further performed to obtain the target pose, avoiding high global load computation. While ensuring that the overall processing frame rate meets the real-time requirements, accuracy is improved for key parts. After determining the target pose, by embedding resolvable physical constraint priors, the robustness of target discrimination and the physical authenticity of pose are significantly improved in challenging scenarios such as severe occlusion, missing textures or sensor noise interference, thus avoiding grasping collisions or empty grasping problems caused by pose conflicts.

[0039] Figure 2 A flowchart illustrating the second six-degree-of-freedom pose determination method provided in this application embodiment is shown below. Figure 2 As shown, in S102 above, feature extraction is performed on the two-dimensional image and the depth image respectively to obtain multi-scale semantic feature maps and multi-scale geometric feature maps, which may include: S201. Convert the depth image into a 3D point cloud based on the camera's intrinsic parameter matrix. Perform background filtering and outlier removal on the 3D point cloud to obtain the removed point cloud. Then, transform the removed point cloud based on the calibration matrix between the robot coordinate system and the camera coordinate system to obtain the target point cloud data in the robot coordinate system.

[0040] Background filtering can be performed using methods such as the Euclidean distance thresholding method to remove point clouds with distances greater than a preset distance. Outlier removal can be achieved through statistical filtering to remove outliers from the point cloud after background filtering, resulting in a point cloud with outliers. This dual filtering significantly suppresses false points caused by depth noise, motion blur, and reflection interference, while avoiding excessive smoothing that leads to distortion of target edges. The point cloud with outliers is in the camera coordinate system; therefore, it is transformed into a point cloud in the robot coordinate system according to the calibration matrix.

[0041] S202. Input the two-dimensional image into the deep encoder, and the deep encoder extracts semantic features from the two-dimensional image to obtain a multi-scale semantic feature map.

[0042] The semantic feature maps at each scale include at least the following semantic features: color, texture, and edges.

[0043] Optionally, the deep encoder is an encoder with an explicit multi-branch decoupling structure, such as a Swing encoder. The Transformer can simultaneously capture local textures, component-level structures, and global scene semantics, thereby generating multi-scale semantic feature maps with spatial alignment and semantic discriminative power. The resulting semantic feature maps at each scale are pixel-aligned with the 2D image, supporting sub-pixel-level cross-modal matching, and encoding fine textures and global layouts. Fine textures, such as screw patterns, and global layouts, such as object orientation and occlusion relationships, form complementary semantics.

[0044] S203. Input the target point cloud data into a sparse convolutional network. The sparse convolutional network extracts geometric features from the target point cloud data to obtain several feature maps at multiple scales.

[0045] The geometric feature maps at each scale contain the normal vectors of the local neighborhood and curvature information.

[0046] Optionally, the sparse convolutional network is a shallow network. For example, the target point cloud data can be voxelized first, and then the sparse convolutional network can extract multi-scale geometric features from the voxelized point cloud.

[0047] In this embodiment, in weak-texture or reflective industrial scenes, the RGB information in a 2D image may degenerate into noise, while the geometric information of the point cloud, though sparse, is stable. Therefore, during feature extraction, it is not necessary to make the depths of the two symmetrical. Thus, a shallow sparse convolutional network is used to extract the target point cloud data, and a deep encoder is used to extract features from the 2D image. In subsequent feature fusion, the shallow geometric features can provide reliable geometric priors, guiding the deep semantic features to focus more on the effective area, which is more suitable for weak-texture, reflective, and occluded scenes.

[0048] Optionally, the step S104 above, in which the lightweight detection unit outputs the initial poses of each object in the target scene and the uncertainties of each initial pose based on the fused feature map, may include: Specifically, the lightweight detection head in the lightweight detection unit performs target detection on the fused feature map, outputs the target detection box, initial pose and category score of each object, and performs pose regression processing on the initial pose of each object based on the heteroscedasticity regression loss function, outputting the uncertainty of each initial pose.

[0049] The lightweight detection head can consist of three convolutional layers and a fully connected layer. The lightweight detection head fuses feature maps. The algorithm performs object detection and outputs the bounding box, initial pose, and class score for each object. The initial pose is represented as a quaternion and a translation vector. Then, a pose regression process is performed on the initial pose of each object using a heteroscedasticity regression loss function, outputting the uncertainty of each initial pose.

[0050] The heteroscedasticity regression loss function is shown in the following formula (1).

[0051] Formula (1); Where N is the total number of samples, Let be the error between the predicted initial pose (i) and the true pose, where i refers to the i-th sample. Let represent the uncertainty of the i-th initial pose. This is an uncertain loss.

[0052] Figure 3 A flowchart illustrating the third six-degree-of-freedom pose determination method provided in this application embodiment is shown below. Figure 3 As shown, in S103 above, cross-modal attention fusion is performed on multi-scale semantic feature maps and multi-scale geometric feature maps to obtain a fused feature map, which may include: S301. A cross-attention mechanism is used to obtain the initial fused feature maps at each scale by combining the semantic feature maps and the geometric feature maps at each scale.

[0053] Semantic feature maps possess high spatial fidelity and strong discriminative power, but lack three-dimensional geometric constraints, making them susceptible to issues such as illumination variations, texture loss, and specular reflection interference, leading to feature drift. Geometric feature maps, on the other hand, exhibit a three-dimensional Euclidean structure and are insensitive to texture, but are semantically sparse, unable to distinguish between structures like screws and nut holes that appear similar but have vastly different functions, and their low-density nature results in the loss of detailed semantics. Therefore, a cross-attention mechanism is employed to obtain initial fused feature maps at various scales.

[0054] Specifically, the first semantic feature map of the layer With the Geometric feature map of the layer Using a cross-attention mechanism, we obtain the first... Initial fusion feature map of the layer .

[0055] S302. Perform residual connection between the initial fused feature maps at each scale and the geometric feature maps at each scale to obtain intermediate fused feature maps at each scale.

[0056] Specifically, it can be done through formulas Intermediate fused feature maps at each scale are obtained, where, For the first The initial fusion feature map of the layer, For the first Geometric feature map of the layer For the first Intermediate fusion feature map of the layer.

[0057] Optionally, residual modulation is performed on the geometric feature maps at each scale. That is, while preserving the original three-dimensional spatial relationships and differential properties of the geometric feature maps, the geometric structural responses that are strongly correlated with the current object category are selectively enhanced. For example, the curvature response of the hexagonal contour region is enhanced for bolt-like objects; the response of the edge normal vector abrupt region is enhanced for thin-walled sheet metal parts, etc. This enables the geometric features to have task-driven semantic sensitivity and overcomes the ambiguity of pure geometric methods on axisymmetric or planar objects.

[0058] S303. Upsample and multi-scale stitching are performed on the intermediate fused feature maps at each scale to obtain the fused feature maps.

[0059] Optionally, the intermediate feature maps at each scale obtained through the above S302 processing are reconstructed into a high-resolution fused feature map through scale-adaptive upsampling and cross-scale feature stitching. The upsampling process introduces geometric continuity constraints to ensure that the reconstructed fused feature map satisfies the local smoothness of the three-dimensional surface in the spatial dimension, avoiding artifacts and structural breaks caused by scale jumps.

[0060] In this embodiment, by adaptively fusing image semantics and point cloud geometric features at different network depths through an attention mechanism, and then outputting a robust fused feature map via residual connections and multi-scale stitching, discriminativeness can still be maintained by relying on geometric information even in weakly textured scenes. That is, when the object surface lacks texture, it can maintain high-precision pose estimation solely by relying on geometric features such as the local curvature and normal vectors of the point cloud, significantly improving robustness in industrial reflective and textureless part scenarios.

[0061] Optionally, the cross-attention mechanism used in S301 above to obtain the initial fused feature maps at each scale for the semantic feature maps and geometric feature maps at each scale may include: Optionally, for each scale of the initial fused feature map, the geometric features in the geometric feature map can be used as query factors, and the semantic features in the semantic feature map can be used as keys and values ​​to calculate attention weights. The attention weights are then used to perform a weighted summation of the values ​​to obtain the initial fused feature map.

[0062] In this embodiment, a one-way dominant attention mechanism is used to enhance geometric features by actively querying semantic features. However, semantic features do not query set features in the reverse direction. This asymmetric one-way information flow prioritizes geometry and supplements semantics, thus avoiding the interference of erroneous semantic information with geometric features when the RGB information quality is poor.

[0063] Optionally, determining the target pose of each object in S104 based on each initial pose, the uncertainty information of each initial pose, and a preset threshold may include: Specifically, if the uncertainty of the initial pose is less than a preset threshold, the initial pose is used as the target pose. If the uncertainty of the initial pose is greater than or equal to the preset threshold, local fine-grained matching is performed based on the initial pose and the object model to obtain the target pose of the object. That is, when the uncertainty of the object's initial pose is less than the preset threshold, it means that the initial pose estimation of the object is more accurate. In this case, the initial pose is directly used as the target pose of the object, ensuring that the required accuracy of the target pose is met with the lowest computational cost when the perception quality is high and the scene constraints are strong. When the initial pose is greater than or equal to the preset threshold, it means that the estimation of the initial pose has significant ambiguity, that is, the estimation of the initial pose is unreliable. In this case, local fine-grained matching is activated.

[0064] In this embodiment, by determining whether to activate subsequent local fine-grained matching based on uncertainty, high-load iteration across the entire scene and all objects is avoided. While ensuring the overall processing frame rate, sub-millimeter-level precision improvement is only performed at necessary locations.

[0065] Figure 4 A flowchart illustrating the fourth six-degree-of-freedom pose determination method provided in this application embodiment is shown below. Figure 4 As shown, the above-mentioned local fine-tuning matching process based on the initial pose and the object model to obtain the target pose of the object can include: S401. Project the object model into three-dimensional space according to the initial pose to obtain the projected point cloud data, and perform expansion processing on the projected point cloud data to obtain the point cloud data of the region of interest.

[0066] In this process, the point cloud in the object model is a point cloud in the object coordinate system. The initial pose includes a rotation matrix and a translation vector. Therefore, the point cloud on the object model is transformed to the camera coordinate system based on the initial pose, resulting in projected point cloud data. This projected point cloud data is then expanded, for example, by using a bounding box centered on the centroid of the object model with a radius larger than the object model's size as the region of interest. The point cloud data within this region of interest is the actual point cloud data from the scene.

[0067] S402. The iterative nearest point algorithm with normal vector constraints is used to iteratively register the point cloud data of the region of interest with the point cloud of the object model. When the iteration ends, the target pose of the object is obtained.

[0068] Optionally, the objective function minimized during the iterative registration process is: ,in, Let be the normal vector of the i-th point cloud data in the region of interest. Let be the normal vector of the i-th point cloud of the object model. These are the weighting coefficients. Let i be the i-th point cloud of the object model. For regions of interest and The corresponding nearest point, R is the rotation matrix, and t is the translation vector. When the object model rotates around a certain axis, the normal vector of the point cloud on the object model also rotates. Therefore, we need to find the normal vector of the point cloud on the rotated object model. It should be consistent with the normal vector actually observed in the scene. As consistent as possible, that is, ensuring the consistency of the set.

[0069] Optionally, in each iteration, for each point cloud on the object model, the nearest point corresponding to that point cloud is found in the point cloud data of the region of interest, such that the Euclidean distance between the point cloud and its nearest point is minimized. In the first iteration, the point cloud of the object model is transformed using the initial pose to obtain the point cloud data of the region of interest. Then, the point cloud of the region of interest and the point cloud of the object model are registered, and a new pose is obtained based on minimizing the objective function. The next iteration is then performed based on the new pose, that is, the point cloud of the object model is transformed based on the new pose to obtain the point cloud data of the new region of interest. Then, the point cloud of the new region of interest and the point cloud of the object model are registered, and the pose is updated based on minimizing the objective function. After a preset number of iterations, the new pose obtained is the target pose of the object. The preset number of iterations is, for example, 3-5 iterations, and can also be set according to actual needs.

[0070] Figure 5 A flowchart illustrating the fifth six-degree-of-freedom pose determination method provided in this application embodiment is shown below. Figure 5 As shown, in step S105 above, determining each target object based on its target pose and using a cost function that includes penetration multiplication and the height of the supporting plane may include: S501. Based on the target pose of the current object and the target poses of other objects, determine the distance between the current object and other objects, as well as the coordinates of the lowest point of the bottom surface of the current object.

[0071] Here, "current object" refers to any single object, and "other objects" refers to any object in the target scene other than the current object. The distance between the current object and other objects is the distance with secondary height; for example, for the current object k and other object j, the distance between them is... The coordinates of the lowest point of the current object's bottom surface can be obtained, for example, using... To express.

[0072] S502. Determine the pose reasonableness between the current object and other objects based on the distance, the coordinates of the lowest point of the current object's bottom surface, and the height of the supporting plane of the target scene.

[0073] Specifically, the rationality of the pose can be determined by formula (2).

[0074] Formula (2); in, The coordinates of the lowest point of the current object's bottom surface. The height of the supporting plane of the target scene. This represents the distance between the current object and other objects. and These are the preset balance coefficients. For the rationality of the position and posture.

[0075] S503. Based on the pose rationality and whether the current object is embedded with other objects, determine whether the current object and other objects are removed.

[0076] Specifically, if the pose reasonableness between the current object and other objects is greater than or equal to the reasonableness threshold, it is determined whether there is embedding between the current object and other objects. If there is no embedding between the current object and other objects, the current object and the other object are both taken as target objects. If there is embedding between the current object and other objects, the current object and the other object are both removed.

[0077] If the pose reasonableness between the current object and other objects is less than the reasonableness threshold, then the current object and other objects will be directly removed.

[0078] Optionally, if after translating the current object k by a preset distance, the current object k is not embedded with other objects j, then it is determined that there is no embedding between the current object k and other objects j; if after translating the current object k by a preset distance, the current object k is still embedded with other objects j, then it is determined that there is embedding between the current object k and other objects j.

[0079] In this embodiment, the pose conflict resolution algorithm based on physical constraints, combined with prior knowledge of the scene such as the desktop plane and the material frame boundary, as well as the collision detection model, ensures that the output target pose is strictly consistent in physical space and can be grasped, thus avoiding grasping collisions or empty grasping problems caused by pose conflicts.

[0080] The hardware platform configuration of this application includes: a processor and an RGB-D camera for real-time acquisition of 2D and depth images. On the software side, a network model is built based on the PyTorch deep learning framework, inference is optimized using CUDA acceleration, and it is encapsulated as an independent node through the ROS robot operating system, achieving seamless integration with the robot control system. During deployment, TensorRT is used to quantize and accelerate the model, keeping the forward inference time within 30ms to meet real-time requirements.

[0081] Figure 6 This is a schematic diagram illustrating a visualization result provided in an embodiment of this application. Figure 6 As can be seen from the embodiments of this application, the method can achieve accurate capture.

[0082] Figure 7 A schematic diagram of an apparatus for a six-degree-of-freedom pose determination method provided in an embodiment of this application is shown below. Figure 7 As shown, the device includes: The acquisition module 601 is used to acquire two-dimensional images and depth images of the target scene; The feature extraction module 602 is used to extract features from the two-dimensional image and the depth image respectively to obtain multi-scale semantic feature maps and multi-scale geometric feature maps. The fusion module 603 is used to perform cross-modal attention fusion on the multi-scale semantic feature map and the multi-scale geometric feature map to obtain a fused feature map, which contains semantic features and geometric features. The detection module 604 is used to input the fused feature map to the lightweight detection unit, which outputs the initial pose of each object in the target scene and the uncertainty of each initial pose based on the fused feature map, and determines the target pose of each object based on each initial pose, the uncertainty of each initial pose, and a preset threshold. The determination module 605 is used to determine each target object based on the target pose of each of the objects and using a cost function that includes penetration multiplication and the height of the supporting plane, wherein the target object is a part of the objects.

[0083] Optionally, the feature extraction module 602 is specifically used for: The depth image is converted into a 3D point cloud based on the camera's intrinsic parameter matrix. Background filtering and outlier removal are performed on the 3D point cloud to obtain the removed point cloud. The removed point cloud is then transformed based on the calibration matrix between the robot coordinate system and the camera coordinate system to obtain the target point cloud data in the robot coordinate system. The two-dimensional image is input into a deep encoder, which extracts semantic features from the two-dimensional image to obtain multi-scale semantic feature maps. Each scale semantic feature map includes at least: color, texture, and edge. The target point cloud data is input into a sparse convolutional network, which extracts geometric features from the target point cloud data to obtain multi-scale geometric feature maps. Each scale of the geometric feature map contains the normal vector and curvature information of the local neighborhood.

[0084] Optionally, the detection module 604 is specifically used for: The lightweight detection head in the lightweight detection unit performs target detection on the fused feature map, outputs the target detection box, initial pose and category score of each object, and performs pose regression processing on the initial pose of each object based on the heteroscedasticity regression loss function, outputting the uncertainty of each initial pose.

[0085] Optionally, the fusion module 603 is specifically used for: A cross-attention mechanism is used to obtain the initial fused feature maps at each scale by combining the semantic feature maps and the geometric feature maps at each scale. The initial fused feature maps at each scale are residually connected with the geometric feature maps at each scale to obtain intermediate fused feature maps at each scale. The intermediate fused feature maps at each scale are upsampled and multi-scale stitched together to obtain the fused feature maps.

[0086] Optionally, the fusion module 603 is specifically used for: Using the geometric features in the geometric feature map as query factors and the semantic features in the semantic feature map as keys and values, we calculate attention weights and use the attention weights to perform a weighted summation of the values ​​to obtain the initial fused feature map.

[0087] Optionally, the detection module 604 is specifically used for: If the uncertainty of the initial pose is less than the preset threshold, then the initial pose is taken as the target pose; If the uncertainty of the initial pose is greater than or equal to the preset threshold, then a local fine-grained matching process is performed based on the initial pose and the object model of the object to obtain the target pose of the object.

[0088] Optionally, the detection module 604 is specifically used for: The object model is projected into a three-dimensional space based on the initial pose to obtain the projected point cloud data. The projected point cloud data is then expanded to obtain the point cloud data of the region of interest. An iterative nearest-point algorithm with normal vector constraints is used to iteratively register the point cloud data of the region of interest with the point cloud of the object model. When the iteration ends, the target pose of the object is obtained.

[0089] Optionally, the determining module 605 is specifically used for: Based on the target pose of the current object and the target poses of other objects, determine the distance between the current object and other objects, as well as the coordinates of the lowest point of the bottom surface of the current object, wherein the current object is any object; Based on the distance, the coordinates of the lowest point of the bottom surface of the current object, and the height of the supporting plane of the target scene, determine the pose rationality between the current object and the other objects; Based on the pose rationality and whether there is embedding between the current object and the other objects, it is determined whether the current object and the other objects are culled; If not, then the current object and all other objects are considered as target objects; if yes, then the current object and all other objects are discarded.

[0090] Figure 8 This is a structural block diagram of an electronic device 700 provided in an embodiment of this application. (See diagram below.) Figure 8 As shown, the electronic device may include: a processor 701 and a memory 702.

[0091] Optionally, a bus 703 may also be included, wherein the memory 702 is used to store machine-readable instructions executable by the processor 701. When the electronic device 700 is running, the processor 701 communicates with the memory 702 via the bus 703, and the processor 701 executes the machine-readable instructions to perform the method steps in the above method embodiments.

[0092] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the method steps described in the above-described six-degree-of-freedom pose determination method embodiment.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0095] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for determining a six-degree-of-freedom pose, characterized in that, The method includes: Acquire 2D and depth images of the target scene; Feature extraction is performed on the two-dimensional image and the depth image respectively to obtain multi-scale semantic feature maps and multi-scale geometric feature maps; Cross-modal attention fusion is performed on the multi-scale semantic feature map and the multi-scale geometric feature map to obtain a fused feature map, which contains both semantic and geometric features. The fused feature map is input to the lightweight detection unit, which outputs the initial pose of each object in the target scene and the uncertainty of each initial pose based on the fused feature map. Based on each initial pose, the uncertainty of each initial pose, and a preset threshold, the target pose of each object is determined. Each target object is determined based on its target pose and using a cost function that includes penetration multiplication and the height of the supporting plane, wherein the target object is at least a portion of the objects.

2. The six-degree-of-freedom pose determination method according to claim 1, characterized in that, The step of extracting features from the two-dimensional image and the depth image to obtain multi-scale semantic feature maps and multi-scale geometric feature maps includes: The depth image is converted into a 3D point cloud based on the camera's intrinsic parameter matrix. Background filtering and outlier removal are performed on the 3D point cloud to obtain the removed point cloud. The removed point cloud is then transformed based on the calibration matrix between the robot coordinate system and the camera coordinate system to obtain the target point cloud data in the robot coordinate system. The two-dimensional image is input into a deep encoder, which extracts semantic features from the two-dimensional image to obtain multi-scale semantic feature maps. Each scale semantic feature map includes at least: color, texture, and edge. The target point cloud data is input into a sparse convolutional network, which extracts geometric features from the target point cloud data to obtain multi-scale geometric feature maps. Each scale of the geometric feature map contains the normal vector and curvature information of the local neighborhood.

3. The six-degree-of-freedom pose determination method according to claim 1, characterized in that, The step of the lightweight detection unit outputting the initial poses of each object in the target scene and the uncertainty of each initial pose based on the fused feature map includes: The lightweight detection head in the lightweight detection unit performs target detection on the fused feature map, outputs the target detection box, initial pose and category score of each object, and performs pose regression processing on the initial pose of each object based on the heteroscedasticity regression loss function, outputting the uncertainty of each initial pose.

4. The six-degree-of-freedom pose determination method according to any one of claims 1-3, characterized in that, The cross-modal attention fusion of the multi-scale semantic feature map and the multi-scale geometric feature map to obtain a fused feature map includes: A cross-attention mechanism is used to obtain the initial fused feature maps at each scale by combining the semantic feature maps and the geometric feature maps at each scale. The initial fused feature maps at each scale are residually connected with the geometric feature maps at each scale to obtain intermediate fused feature maps at each scale. The intermediate fused feature maps at each scale are upsampled and multi-scale stitched together to obtain the fused feature maps.

5. The six-degree-of-freedom pose determination method according to claim 4, characterized in that, The semantic feature maps and geometric feature maps at each scale are processed using a cross-attention mechanism to obtain initial fused feature maps at each scale, including: Using the geometric features in the geometric feature map as query factors and the semantic features in the semantic feature map as keys and values, we calculate attention weights and use the attention weights to perform a weighted summation of the values ​​to obtain the initial fused feature map.

6. The six-degree-of-freedom pose determination method according to any one of claims 1-3, characterized in that, The step of determining the target pose of each object based on each initial pose, the uncertainty of each initial pose, and a preset threshold includes: If the uncertainty of the initial pose is less than the preset threshold, then the initial pose is taken as the target pose; If the uncertainty of the initial pose is greater than or equal to the preset threshold, then a local fine-grained matching process is performed based on the initial pose and the object model of the object to obtain the target pose of the object.

7. The six-degree-of-freedom pose determination method according to claim 6, characterized in that, The step of performing local fine-grained matching based on the initial pose and the object model to obtain the target pose of the object includes: The object model is projected into a three-dimensional space based on the initial pose to obtain the projected point cloud data. The projected point cloud data is then expanded to obtain the point cloud data of the region of interest. An iterative nearest-point algorithm with normal vector constraints is used to iteratively register the point cloud data of the region of interest with the point cloud of the object model. When the iteration ends, the target pose of the object is obtained.

8. The six-degree-of-freedom pose determination method according to any one of claims 1-3, characterized in that, Each target object is determined based on its target pose and using a cost function that includes penetration multiplication and the height of the supporting plane. The target objects are a subset of the objects in the given set, including: Based on the target pose of the current object and the target poses of other objects, determine the distance between the current object and other objects, as well as the coordinates of the lowest point of the bottom surface of the current object, wherein the current object is any object; Based on the distance, the coordinates of the lowest point of the bottom surface of the current object, and the height of the supporting plane of the target scene, determine the pose rationality between the current object and the other objects; Based on the pose rationality and whether there is embedding between the current object and the other objects, it is determined whether the current object and the other objects are culled; If not, then the current object and all other objects are considered as target objects; if yes, then the current object and all other objects are discarded.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the steps of the six-degree-of-freedom pose determination method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the six-degree-of-freedom pose determination method as described in any one of claims 1-8.

Images