A pose determination method, apparatus and related products

Abstract

This application provides a pose determination method, apparatus, and related products. The method first segments a two-dimensional image (target image) including the target object whose pose needs to be determined, obtaining a target instance mask of the target object. However, the initial 6D pose of the target object is determined based on this target instance mask as the target region. The 6D pose determined using a two-dimensional image lacks true three-dimensional geometric constraints, resulting in significant errors in depth direction, rotation angle, and local surface fit. Therefore, this application introduces true three-dimensional geometric constraints. By adjusting the model source point cloud generated in the camera coordinate system from the three-dimensional CAD model of the target object and obtaining the relative pose of the scene target point cloud based on the target instance mask and the real scene of the target image, the initial pose is optimized in the backend. This reduces the deficiency of relying solely on two-dimensional images and improving the accuracy of 6D pose determination.

Description

Technical Field

[0001] This application relates to the field of industrial automation technology, and in particular to a pose determination method, apparatus and related products. Background Technology

[0002] In the field of industrial automation, robots are increasingly used in scenarios such as disordered grasping, workpiece loading, flexible assembly, and quality inspection. These scenarios all require robots to accurately estimate the 6-degree-of-freedom (6D) pose of the target object from a single red-green-blue (RGB) image that includes the target object.

[0003] FoundPose, a model-driven 6D pose determination method, establishes a correspondence between images and models using basic visual features and template matching. It can determine the 6D pose of a target object without retraining the pose network for each specific object. However, in real-world industrial scenarios, especially when the target object has weak or no surface texture, local occlusion, cluttered backgrounds, or drastic changes in lighting conditions, the 6D pose determined by FoundPose suffers from poor accuracy, failing to meet the technical requirements for pose estimation accuracy in industrial automation. Summary of the Invention

[0004] Based on the above problems, this application provides a pose determination method, apparatus and related products to improve the 6D pose determination accuracy of a target object, so that the 6D pose of the target object meets the requirements of pose estimation accuracy in the field of industrial automation.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, embodiments of this application provide a pose determination method, the method comprising: Acquire a target image, wherein the target image is a two-dimensional image and includes a target object whose pose is to be determined; The target image is segmented to obtain the target instance mask of the target object; Using the target instance mask as the target region, determine the initial 6D pose of the target object; Obtain the model source point cloud and the scene target point cloud; the model source point cloud is generated by converting the 3D CAD model of the target object to camera coordinates, and the scene target point cloud is obtained based on the target instance mask and the real scene depth information of the target image; Based on the initial 6D pose, the final 6D pose of the target object is obtained by adjusting the pose of the model source point cloud relative to the scene target point cloud.

[0006] Optionally, obtaining the final 6D pose of the target object based on the initial 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud includes: By iteratively optimizing, the pose of the model source point cloud relative to the scene target point cloud is adjusted so that the points in the model source point cloud converge to the corresponding points in the scene target point cloud in three-dimensional space, thereby obtaining the final 6D pose of the target object.

[0007] Optionally, the convergence condition is one or more of the following conditions: The spatial distance between the points in the model source point cloud and the corresponding points in the scene target point cloud is minimized; The normal error between the points in the model source point cloud and the corresponding points in the scene target point cloud is minimized; The number of iterations meets the maximum number of iterations. The average spatial distance between points in the model source point cloud and corresponding points in the scene target point cloud is less than or equal to a preset threshold. The change in pose of the model source point cloud relative to the scene target point cloud in two adjacent iterations is less than or equal to the change threshold.

[0008] Optionally, obtaining the final 6D pose of the target object based on the initial 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud includes: By iteratively optimizing, the pose of the model source point cloud relative to the scene target point cloud is adjusted to minimize the spatial distance between the points in the model source point cloud and the corresponding points in the scene target point cloud, thereby obtaining the intermediate 6D pose of the target object. Determine the quality of the target point cloud in the scene; If the quality of the scene target point cloud is higher than a preset quality threshold, based on the intermediate 6D pose, the pose of the model source point cloud relative to the scene target point cloud is adjusted through iterative optimization to minimize the normal error between the points in the model source point cloud and the corresponding points in the scene target point cloud, thereby obtaining the final 6D pose of the target object.

[0009] Optionally, the method further includes: If the quality of the target point cloud in the scene is lower than or equal to a preset quality threshold, the intermediate 6D pose is taken as the final 6D pose of the target object.

[0010] Optionally, the step of segmenting the target image to obtain the target instance mask of the target object includes: The target image is processed based on an image segmentation model to obtain a target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, which includes sample data of the target object under various imaging conditions.

[0011] Optionally, the image segmentation model is the YOLO instance segmentation model.

[0012] Optionally, the training sample set is an industrial parts dataset conforming to the BOP evaluation specification, and the method for obtaining the training sample set includes: Acquire multimodal image data of the target object and camera intrinsic parameters at the time of acquisition, wherein the multimodal image data includes images of the target object under different imaging conditions; Based on the 3D CAD model of the target object, determine the true 6D pose of the target object in the multimodal image data; The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set. The hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data item is used to store the 3D CAD model of the target object, the image data item is used to store the multimodal image data, and the annotation data item is used to store the camera intrinsic parameters and the 6D pose ground truth value corresponding to each frame of the multimodal image data.

[0013] Optionally, determining the initial 6D pose of the target object using the target instance mask as the target region includes: Using the target instance mask as the target region, a target template image matching the target image is selected from a pre-established template library; A two-dimensional optical flow algorithm is used to perform pixel-level matching between the target image and the target template image to obtain the 2D-3D correspondence of the target object. The PnP algorithm is used to process the 2D-3D correspondence of the target object to obtain the initial 6D pose of the target object.

[0014] Optionally, determining the 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence of the target object includes: The following target operation is performed on the 2D-3D object of the target object after a preset number of iterations to obtain the 6D pose of the target object; The target operation includes: randomly selecting a minimum sampling set from the 2D-3D correspondence of the target object, the minimum sampling set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; and determining the number of points whose reprojection error is less than a preset threshold as the target number of points. Among them, the candidate 6D pose with the most target points is taken as the 6D pose of the target object.

[0015] Secondly, embodiments of this application provide a pose determination device, the device comprising: The first acquisition unit is used to acquire a target image, wherein the target image is a two-dimensional image and includes a target object whose pose is to be determined; A segmentation unit is used to segment the target image to obtain a target instance mask of the target object; The first determining unit is used to determine the initial 6D pose of the target object using the target instance mask as the target region; The second acquisition unit is used to acquire the model source point cloud and the scene target point cloud; the model source point cloud is generated by converting the 3D CAD model of the target object to camera coordinates, and the scene target point cloud is acquired based on the target instance mask and the real scene depth information of the target image; The second determining unit is used to obtain the final 6D pose of the target object by adjusting the pose of the model source point cloud relative to the scene target point cloud based on the initial 6D pose.

[0016] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the pose determination method as described in any of the first aspects.

[0017] Fourthly, embodiments of this application provide a computer program product, characterized in that, when it is run on a computer, it executes the pose determination method as described in any of the first aspects.

[0018] Fifthly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores a computer program; and the processor is coupled to the memory for calling and executing the computer program to implement the pose determination method as described in any of the first aspects.

[0019] Compared with the prior art, this application has the following beneficial effects: This application provides a pose determination method, apparatus, and related products. The method first segments a two-dimensional image (target image) including the target object whose pose needs to be determined, obtaining a target instance mask of the target object. However, the initial 6D pose of the target object is determined based on this target instance mask as the target region. The 6D pose determined using a two-dimensional image lacks true three-dimensional geometric constraints, resulting in significant errors in depth direction, rotation angle, and local surface fit. Therefore, this application introduces true three-dimensional geometric constraints. By adjusting the model source point cloud generated in the camera coordinate system from the three-dimensional CAD model of the target object and obtaining the relative pose of the scene target point cloud based on the target instance mask and the real scene of the target image, the initial pose is optimized in the backend. This reduces the deficiency of relying solely on two-dimensional images and improving the accuracy of 6D pose determination. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a pose determination method provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for obtaining a training sample set, as provided in an embodiment of this application; Figure 3 A flowchart illustrating another iterative method for obtaining the final 6D pose of a target object, as provided in this application embodiment; Figure 4 This is a schematic diagram of a pose determination device provided in an embodiment of this application. Detailed Implementation

[0022] As described earlier, in real industrial scenarios, especially in harsh industrial environments such as weak or no surface texture of the target object, partial occlusion of the target object, cluttered background, or drastic changes in lighting conditions, using FoundPose to determine the 6D pose of the target object will result in poor determination accuracy, making it difficult to meet the technical requirements for pose estimation accuracy in the field of industrial automation.

[0023] It should be noted that the 6D pose of the target object refers to a set of parameters representing a rigid transformation between the target object's own 3D coordinate system and the camera coordinate system. The 6D pose includes 3 rotational degrees of freedom and 3 translational degrees of freedom. Through this 6D pose, the point cloud of the target object's 3D computer-aided design (CAD) model can be transformed from the object coordinate system to the camera coordinate system, thereby uniquely determining the target object's position and orientation in space. For example, in the field of robotics, a robot's control system can use the target object's 6D pose to align the gripping point or assembly point with the target object's actual spatial position and orientation, thereby achieving high-precision automation.

[0024] Research has revealed that FoundPose is a pose estimation scheme based on fundamental visual features. It utilizes template retrieval and feature matching to establish a 2D-3D correspondence, achieving generalized recognition without requiring retraining for new objects. However, FoundPose is essentially a 2D image-based pose estimation method, lacking realistic 3D geometric constraints. This leads to issues such as depth direction errors, rotation angle deviations, and local surface misalignment in industrial scenarios. These misalignment problems are further exacerbated when the target object has complex structures and special material characteristics, resulting in poor 6D pose accuracy for the determined target object.

[0025] For example, in a scenario where a robotic arm grasps a metal part, the target object is an industrial part with a complex geometry and special material characteristics, whose surface has highly reflective areas and includes multiple holes. When the target image is processed by CNOS, the high reflectivity causes the boundaries of the industrial part to be blurred, resulting in the incorrect inclusion of the background area of ​​the workbench in the instance mask, while the hole area is regarded as a valid part of the industrial part.

[0026] In view of this, the present application provides a pose determination method that uses a 6D pose determined based on a 2D image as an initial value. The method adjusts the 3D CAD model of the target object to be converted to the camera coordinate system to generate a model source point cloud and obtains the relative pose of the scene target point cloud based on the target instance mask and the real scene of the target image. The initial pose is then optimized in the backend to reduce the deficiency of relying solely on 2D images and lacking depth constraints, thereby improving the accuracy of 6D pose determination.

[0027] The pose determination method provided in this application can be applied to the fields of computer vision, 3D machine vision, intelligent perception of industrial robots, and other pose estimation fields. This application is not limited to these fields. To facilitate understanding by those skilled in the art, the following description uses intelligent perception of industrial robots, specifically their applications in scenarios such as disordered grasping, workpiece loading, flexible assembly, and quality inspection, as examples.

[0028] It should be noted that the execution subject of the pose determination method provided in this application embodiment can be an electronic device with computing capabilities. Specifically, the electronic device includes, but is not limited to, personal computers, embedded vision processing systems, robot control systems, etc. It can also be a hardware module or software platform deployed in the above-mentioned electronic device, and this application embodiment is not limited. For ease of explanation, the execution subject of this application embodiment is described using a robot control system as an example.

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0030] Appendix Figure 1 This is a flowchart illustrating a pose determination method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps: S10, acquire the target image, wherein the target image includes the target object whose pose is to be determined, and the target image is a two-dimensional image.

[0031] The target object can be an industrial part with a complex geometry and / or specific material characteristics. For example, the target object is an industrial valve, which has a highly reflective metallic surface and a complex structure.

[0032] The target image is a two-dimensional image that includes the target object, such as an RGB image.

[0033] In this embodiment of the application, the control system needs to acquire a target image including the target object in order to determine the 6D pose of the target object. For example, if the control system needs to determine the 6D pose of a valve so that the robot's robotic arm can accurately grasp and assemble it, it can acquire RGB and / or RGBD images of the target valve in real time through a camera mounted at the end of the robot's robotic arm or a camera mounted at a fixed position on the robot.

[0034] It should be noted that the camera in this application embodiment can be an RGB camera, an RGB-D depth camera, etc., and this application embodiment is not limited to any particular type.

[0035] S20, perform segmentation processing on the target image to obtain the target instance mask of the target object.

[0036] In this embodiment, CNOS in FoundPose can be used to segment the target image to obtain the target object instance mask.

[0037] In another example, after the control system acquires the target image, it performs segmentation processing on the target image based on the trained image segmentation model to obtain the instance mask of the target object, i.e., the target instance mask.

[0038] In this embodiment, an image segmentation model trained using a pre-constructed training sample set is used. The training sample set includes sample data of the target object under various imaging conditions. Imaging conditions refer to external variables and / or acquisition parameters of the target object when acquiring images including the target object, such as the target object's illumination intensity, shooting angle, shooting distance, occlusion, or surface reflectivity.

[0039] In practical applications, the training sample set can be an industrial parts dataset that conforms to the Benchmark for 6DObject Pose Estimation (BOP) evaluation specification. Based on this, this application also provides an automated method for obtaining the training sample set. (Appendix) Figure 2 A flowchart illustrating a method for obtaining a training sample set, as provided in this application embodiment, includes the following steps: S210, acquire the multimodal image data of the target object and the camera intrinsic parameters at the time of acquisition.

[0040] The multimodal image data includes images of the target object under different imaging conditions. These different imaging conditions can include changes in the light source, adjustment of light intensity, adjustment of shooting distance and angle, introduction of partial occlusion, and changes in the background area. In this embodiment, the multimodal image data should comprehensively reflect the various imaging conditions that the target object may encounter in practical applications, thereby improving the robustness of the trained image segmentation model to complex environments.

[0041] Camera intrinsics are key parameters for pose estimation, used to convert 2D image coordinates into 3D camera coordinates. Camera intrinsics can be obtained offline through camera calibration tools (such as the calibration module provided by OpenCV), or in real-time via sensors integrated into the camera during each acquisition, or through other methods; this application does not limit the methods used.

[0042] S220 determines the true 6D pose of the target object in each frame of the image based on the 3D CAD model of the target object.

[0043] Each frame of the image data consists of a single frame of multimodal image data.

[0044] This application embodiment can determine the 6D pose truth value of the target object in each frame of an image based on a 3D CAD model of the target object. For example, the 3D CAD model of the target object is rendered into an image under known camera intrinsics and a preset pose to obtain a projected image. The projected image is compared with the actual acquired image, and the optimal pose parameters that minimize the difference between the projected image and the actual image are found through manual adjustment or a preset algorithm. The optimal pose parameters are the 6D pose truth value of the target object in that frame of the image. This application embodiment can also determine the 6D pose truth value of the target object in each frame of the image through other methods, such as structured light scanning reconstruction. This application embodiment is not limited to these methods.

[0045] S230: The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set.

[0046] In this embodiment, the hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data items store the 3D CAD model of the target object. It can be stored in a standard CAD model file format, such as Standard for the Exchange of Product Data (STEP), Initial Graphics Exchange Specification (IGES), or Stereolithography (STL), which can completely describe the geometric topology and surface information of the target object. The image data items store multimodal image data. Image data can be stored at its original resolution and format (e.g., PNG, JPG, TIFF), or it can be stored according to imaging conditions such as acquisition time and viewing angle. The annotation data items store the camera intrinsic parameters and 6D pose ground truth values ​​corresponding to each frame of the multimodal image data. In this embodiment, the annotation data item can be a directory containing multiple JSON or YAML files, each file corresponding to a frame image and recording the camera intrinsic parameter matrix (e.g., focal length, principal point, distortion coefficients) and the true 6D pose of the target object in the camera coordinate system for that frame image. In practical use, the hierarchical data structure can be organized using a file system directory structure. For example, the root directory contains multiple object folders, and each object folder contains subfolders such as model data items, image data items, and annotation data items.

[0047] For example, a training sample set is constructed by using the joint parts in a robotic arm operation scenario as the target object.

[0048] First, an RGBD camera is used to acquire multimodal image data of the part on the industrial production line. During the acquisition process, the shooting angle, shooting distance, lighting conditions, and occlusion can be intentionally changed to simulate the diversity of the actual working environment. Simultaneously, camera intrinsic parameters are acquired and recorded for each acquisition using a camera calibration tool. Next, the 3D CAD model of the part is rendered onto the acquired images, and the precise 6D pose of each frame is determined through manual alignment or feature matching optimization using interactive software. Finally, the directory structure is organized according to the BOP specification: a folder named "Part Name" is created in the root directory. This folder includes model data items, image data items, and annotation data items.

[0049] The above approach collects multimodal image data under various imaging conditions and combines it with CAD models to determine the true pose value, making the training data closer to real industrial scenarios. The BOP hierarchical data structure is adopted to improve the standardization of data and provide a unified benchmark for subsequent segmentation model training and pose estimation.

[0050] In this embodiment, the image segmentation model can be a general image segmentation model, such as a Mask Region-Based Convolutional Neural Network (Mask R-CNN) or a U-Shaped Convolutional Neural Network (U-Net). Alternatively, the image segmentation model can be the YOLO instance segmentation model. The YOLO instance segmentation model is a graph segmentation model capable of simultaneously performing object detection and instance segmentation. This model can directly predict the bounding box, category, and pixel-level instance mask of the target within a unified network. For example, models such as YOLOv5, YOLOv7, or YOLOv8 can be used, achieving a good balance between speed and accuracy. The YOLO instance segmentation model can directly output the instance mask of each object in the input image in an end-to-end manner. This approach avoids the complex region proposal generation and feature extraction processes of traditional two-stage segmentation methods, significantly improving processing efficiency.

[0051] For example, let's take the YOLOv6 instance segmentation model as an example. First, the control system constructs a training sample set using sample data containing the target object under various imaging conditions, and trains the YOLOv6 instance segmentation model. During training, the model learns the visual features and boundary information of the target object, enabling it to accurately identify and segment the target object. In practical applications, when the control system acquires a target image containing the target object, this image is input into the trained YOLOv6 instance segmentation model. The YOLOv6 model processes the image in real time and outputs an instance mask corresponding to the target object. This mask clearly outlines the contour of the target object and distinguishes it from the background or other irrelevant areas. Then, this target instance mask can be used for template matching and pixel-level matching, and based on the matching results, the 6D pose of the target object can be generated.

[0052] In some of the solutions described above in this application, the YOLO instance segmentation model is proposed as the image segmentation model for obtaining target instance masks, in order to obtain more accurate target regions and provide reliable region constraints for subsequent template matching and pose determination. However, in this process, the conventional YOLO instance segmentation model pays consistent attention to the loss of all regions of the target object. However, the geometric structure region of the target object is the key to determining the accuracy of the mask boundary contour. If the error in this region is not given enough attention, inaccurate mask boundaries, missed or incorrect segmentation, or even the introduction of background or neighboring regions into the target region can easily occur, which will interfere with the accuracy of subsequent template retrieval, pixel matching, and pose determination.

[0053] In view of this, this application further proposes an improved YOLO instance segmentation model. This YOLO instance segmentation model is an instance segmentation model where the detection head includes edge-sensitive loss weights; wherein the edge-sensitive loss weights include a first weight set for the geometric structural regions of the target object, and a second weight set for the non-geometric structural regions of the target object. The first weight is greater than the second weight; for example, the first weight is 0.6 and the second weight is 0.4.

[0054] The detection head of the YOLO instance segmentation model is responsible for the final prediction output. Its function is to generate an instance mask of the target object based on the features extracted by the model. The detection head includes edge-sensitive loss weights, meaning that when calculating the difference between the model's prediction and the true label, different importance is assigned to the region (geometric or non-geometric region) where the pixel is located. The introduction of these loss weights aims to guide the model to pay more attention to regions that are critical to the segmentation quality during training.

[0055] Edge-sensitive loss weights are a mechanism for adjusting the contribution of a loss function. They apply differentiated penalties to prediction errors based on the characteristics of different regions of the target object. For example, during training, the weight of a pixel in the total loss can be dynamically adjusted based on whether it is located within the geometric structure region of the target object. For instance, a weight map can be introduced into the loss calculation of the mask branch in the YOLO instance segmentation model, dynamically adjusting the loss contribution based on whether a pixel is located within a geometric structure region. Alternatively, the loss function can be modified to apply a higher penalty factor to pixels closer to the edge of the target object when calculating the mask loss.

[0056] As a specific implementation method, when training the YOLO instance segmentation model, in addition to providing the original image and the target instance mask, a corresponding geometric structure region mask can be provided for the target object in the training sample set. This geometric structure region mask can be obtained by edge extraction from the 3D CAD model of the target object or by performing morphological operations on the labeled instance mask. During the training process of the YOLO instance segmentation model, its detector head applies different weights based on this geometric structure region mask when calculating the loss of the mask branch. Specifically, for pixels marked as geometric features in the geometric structure region mask, their corresponding loss value is multiplied by a larger first weight, such as 0.6; while for pixels marked as non-geometric features in the geometric structure region mask, their corresponding loss value is multiplied by a smaller second weight, such as 0.4. In this way, the YOLO instance segmentation model will focus more on reducing the segmentation error of the geometric structure region during backpropagation and parameter updates, thereby enabling the trained YOLO instance segmentation model to output a target instance mask with more accurate boundaries and clearer geometric details during the inference stage.

[0057] By introducing differentiated weights into the detection head of the YOLO instance segmentation model, the segmentation accuracy of the model for key geometric regions of the target object can be improved without significantly altering the model structure or increasing the inference burden, thereby obtaining a higher quality target instance mask.

[0058] S30, use the target instance mask as the target region to determine the initial 6D pose of the target object.

[0059] Specifically, the control system first uses the target instance mask as the target region, matches the target template image that matches the target image from the pre-established template library, and performs pixel-level matching between the target template image and the target image.

[0060] In this embodiment, the template library is a pre-established database or data structure collection that stores template images corresponding to multiple target objects. It provides reference templates with known geometric shapes, texture features, and scale information when matching with target objects in target images, thereby assisting in achieving high-precision pixel-level matching and alignment within the target area. These template images typically represent the standard appearance of the target object under different viewing angles and lighting conditions. The template library is constructed to provide a reference benchmark for target object recognition and pose estimation.

[0061] The control system can use the target instance mask obtained in step S20 as the target region and match the target template image that matches the target image from the template library. In one specific implementation, the target template images can be the K template images that are determined to best match the target region of the current target image after similarity calculation and filtering. Here, K is a positive integer, for example, K is 5.

[0062] In one example, the control system can use a target instance mask as the target region and extract the visual features of the target region from the target image. The control system then calculates the similarity between the visual features and template images in a template library, and uses the template images with the highest similarity (top K) as the target template image.

[0063] Visual features are attributes in an image that can be used to describe and distinguish target objects, including but not limited to edges, corners, textures, and color distribution. Acquiring visual features of a target region aims to extract visual information that represents the essential attributes of the target object from the target region defined by a target instance mask. By limiting feature extraction to the target region, interference from background noise and non-target objects can be effectively eliminated, ensuring that the acquired features are a pure representation of the target object itself. For example, a control system can first crop an image patch of the target region based on the target instance mask, and then perform feature extraction on that patch; alternatively, features can be extracted from the entire image, and then feature points or feature descriptors falling within the target region can be selected based on the target instance mask.

[0064] Similarity calculation is the process of quantifying the degree of similarity between two visual features. Its purpose is to establish a comparable matching score between each template image in the template library and the target region of the current target image. Embodiments of this application can employ various similarity calculation methods. For example, Euclidean distance, cosine similarity, or Hamming distance can be used to compare the distance between feature descriptors; a smaller distance generally indicates a higher similarity.

[0065] The following is a concrete example to illustrate this. Suppose the target object whose pose needs to be determined is an industrial gear, and a target instance mask of the gear in the target image has already been obtained through an image segmentation model. First, the robot's control system uses this target instance mask to precisely delineate the region where the gear is located in the target image, i.e., the target region. Next, visual features are extracted from this target region. For example, the Scale-Invariant Feature Transform (SIFT) algorithm can be used to detect and describe key points on the gear surface, generating a series of SIFT feature descriptors. Simultaneously, a template library pre-stores multiple template images of the gear under different viewpoints and lighting conditions, and each template image also has its corresponding SIFT feature descriptor pre-extracted and stored. Then, the system compares the gear SIFT feature descriptor extracted from the target image with the SIFT feature descriptors of all template images in the template library, calculating their similarity. After calculating the similarity of all template images, the robot's control system sorts them from highest to lowest similarity. Assuming a preset K value of 5, the system will select the top 5 template images with the highest similarity as the target template images for matching with the target image. These 5 selected template images will serve as candidates for subsequent pixel-level matching to further refine the matching process.

[0066] The control system uses the target instance mask obtained in step S20 as the target region. After matching the target template image that matches the target image from the template library, it can perform pixel-level matching between the target template image and the target image.

[0067] In one possible implementation, the control system can accurately capture pixel-level motion information by calculating the displacement vector field between corresponding pixels in the target template image and the target image. Here, the displacement vector field refers to the relative offset of each pixel between the target template image and the target image. The displacement vector field can precisely quantify the local deformation or positional differences of the target object's surface between the two images, thereby meticulously capturing pixel-level motion information.

[0068] Then, based on the displacement vector field, the pixels in the target region of the target template image are mapped to their corresponding positions in the target image to establish pixel-level matching between the target template image and the target image. This pixel-level matching ensures that each feature point on the target template image can find its accurate corresponding point in the target image, thus forming a high-precision 2D-2D correspondence. In one specific implementation, the coordinates of each pixel in the target template image can be directly added to its corresponding displacement vector to obtain its new coordinates in the target image. For example, for each pixel (xt, yt) in the target region of the target template image, its corresponding displacement vector (dx, dy) in the displacement vector field is queried. Then, the pixel is mapped to its new position (xt + dx, yt + dy) in the target image. Furthermore, since the displacement vector is usually a floating-point number, the new coordinates may not be integers. Therefore, interpolation algorithms (such as bilinear interpolation or cubic spline interpolation) can be used to determine the pixel value of the target image at the non-integer coordinates, thereby completing the pixel mapping and matching.

[0069] In another specific implementation, the control system can perform pixel-level matching between the target template image and the target image based on a two-dimensional optical flow method. Specifically, the two-dimensional optical flow method uses the target template image as the reference frame and the target region in the target image defined by the target instance mask as the current frame. Assuming that the brightness of the same pixel remains basically unchanged between two frames, the two-dimensional displacement vector of each pixel in the image plane is obtained by solving the optical flow constraint equation, forming an optical flow field. The target template image is then registered and aligned to the target image based on this optical flow field, achieving pixel-level or even sub-pixel-level template matching. The two-dimensional optical flow method can employ a sparse optical flow method based on local windows or a dense optical flow method to obtain the full-pixel displacement field, which can be selected according to computational resources and accuracy requirements.

[0070] Through the above technical solution, this application can effectively achieve high-precision pixel-level matching between the target template image and the target image, avoiding misalignment, thus providing a more accurate and reliable foundation for the subsequent establishment of 2D-3D correspondence and 6D pose solving.

[0071] Next, the control system can establish a 2D-3D correspondence between the target object and the pixel-level matching results and the depth information of the target object. Then, the control system determines the initial 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence.

[0072] The embodiments of this application can establish the 2D-3D correspondence of the target object in a variety of ways.

[0073] In one possible approach, the control system can acquire a depth map corresponding to the target image. For each pixel in the target image from the pixel-level matching results, its depth value in the depth map, combined with known camera intrinsic parameters, is back-projected into 3D space to obtain the pixel's 3D coordinates in the camera coordinate system. Simultaneously, the pixel-level matching results also provide the association between the target image pixel and its corresponding pixel in the template image. Each pixel in the template image typically has its 3D coordinates pre-stored on the 3D CAD model of the target object. In this way, a correspondence can be established between the pixels of the target object in the target image and the points in the 3D CAD model of the target object—that is, the 2D-3D correspondence of the target object.

[0074] In another possible approach, after pixel-level matching is completed, the control system, for each matched pixel in the target image, directly obtains its corresponding 3D point on the 3D CAD model by querying the pre-established correspondence between the template image and the 3D CAD model, thereby establishing a 2D-3D object of the target object.

[0075] Furthermore, if the target image itself is acquired by an RGBD camera, the depth information of the target image pixels can be directly obtained from the depth map, and combined with the camera intrinsic parameters to convert it into three-dimensional points in the camera coordinate system, thereby establishing a 2D-3D correspondence between the target object and the target.

[0076] For example, while acquiring the target image, the control system can use a structured light camera or LiDAR to obtain point cloud data of the target object. The target image pixels in the pixel-level matching results can be directly matched with corresponding 3D points in the point cloud data. These 3D points can serve as 3D information of the target object, establishing a correspondence between them and the 2D pixels in the target image.

[0077] It should be noted that the 2D-3D correspondence of the target object can also be established in various ways in the embodiments of this application, and this application is not limited to any particular method.

[0078] In this embodiment, the control system can directly utilize the Perspective-n-Point Algorithm (PnP) to determine the 6D pose of the target object by combining the 2D-3D object relationship. For example, the control system directly inputs the established 2D-3D correspondence into the PnP algorithm, which can solve for the rotation matrix and translation vector of the target object, thus obtaining the initial 6D pose.

[0079] In another example, the control system can iterate the following target operations on the 2D-3D correspondence of the target object and iterate a preset number of times to obtain the initial 6D pose of the target object.

[0080] The target operation includes: randomly selecting a minimum sampling set from the 2D-3D correspondence of the target object, the minimum sampling set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; determining the number of points with a reprojection error less than a preset threshold as the target point number; wherein, the candidate 6D pose with the largest number of target points is taken as the initial 6D pose of the target object.

[0081] The preset number of iterations refers to the number of times the target operation is repeatedly executed. Its purpose is to find the optimal solution or a result that satisfies specific conditions through multiple attempts. In the pose determination process, the preset number of iterations ensures that the algorithm has sufficient opportunities to explore different sampling combinations, thereby increasing the probability of finding the accurate pose. This preset number of iterations can be a fixed value, such as hundreds or thousands, to balance computational efficiency and robustness of the results; or it can be dynamically adjusted based on the algorithm's convergence, for example, stopping when multiple consecutive iterations fail to significantly improve the results.

[0082] Randomly selecting the minimum sample set from the 2D-3D correspondence of the target object means randomly selecting a subset containing the fewest number of point pairs from all established 2D-3D correspondence point pairs. This randomness helps avoid sampling bias and ensures that the algorithm can handle various possible mismatches. The sampling method can be uniform random sampling, where each point pair has an equal probability of being selected; or weighted random sampling, for example, assigning different weights based on the feature quality or confidence level of the point pairs, prioritizing the selection of higher-quality point pairs.

[0083] In this embodiment, the minimum sampling set includes at least four sets of corresponding points, which ensures the computational efficiency of the PnP algorithm while satisfying the computational requirements for solving the problem. In practical applications, four sets of points can be precisely selected, or slightly more than four sets of points, such as five or six sets, can be selected to increase redundancy and robustness.

[0084] Calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm means using the PnP algorithm to quickly calculate a candidate 6D pose from a randomly selected minimum sampling set.

[0085] Reprojection error refers to the distance between the projected 3D point onto the 2D image plane using the currently calculated candidate 6D pose, and the actual observed 2D image point. This distance reflects the consistency between the current candidate pose and all corresponding points. The error is usually calculated using Euclidean distance, which is the straight-line distance between two 2D points on the image plane, or its square; however, this embodiment does not limit the calculation.

[0086] Determining the number of points with reprojection errors less than a preset threshold as the target number means that if the reprojection error of a point is less than the preset threshold, then that point is considered compatible with the current candidate pose, i.e., it is an interior point. The target number of points refers to the total number of points with reprojection errors less than the preset threshold in all 2D-3D correspondences. This threshold can be set according to factors such as image noise level and feature matching accuracy, for example, it can be set to 2 pixels.

[0087] In addition, embodiments of this application may also employ the PnP algorithm, which combines the Random Sample Consensus (RANSAC) algorithm, to process the 2D-3D correspondence of the target object and obtain the initial 6D pose of the target object.

[0088] S40, acquire the source point cloud of the model and the target point cloud of the scene.

[0089] The model source point cloud is generated by converting the 3D CAD model of the target object into the camera coordinate system, representing the ideal geometric shape of the target object. The scene target point cloud is constructed using the depth information of the target image corresponding to the real scene and the target instance mask. Specifically, the control system converts the pixels in the depth image corresponding to the target instance mask region into 3D spatial points, thus forming the scene target point cloud. This scene target point cloud reflects the actual geometric shape of the target object in the real scene.

[0090] S50, based on the initial 6D pose, obtains the final 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud.

[0091] Based on the model source point cloud and scene target point cloud obtained in step S40, the control system can obtain the final 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud, based on the initial 6D pose.

[0092] In one specific implementation, the control system can calculate the spatial difference between the model source point cloud and the scene target point cloud, and reduce this difference through a series of small translation and rotation transformations. For example, the control system can directly calculate a rigid body transformation based on a non-iterative least-squares method, minimizing the distance between corresponding points in the two point clouds. This yields a corrected final 6D pose that is geometrically closer to the real scene point cloud than the initial pose.

[0093] In one specific implementation, the control system can adjust the pose of the model source point cloud relative to the scene target point cloud through iterative optimization, so that the points in the model source point cloud meet the preset convergence conditions to the corresponding points in the scene target point cloud in three-dimensional space, thereby obtaining the final 6D pose of the target object.

[0094] Iterative optimization is a mathematical method that approximates the optimal solution by repeatedly executing a series of computational steps. In this application, its role is to gradually improve the matching degree between the model source point cloud and the scene target point cloud by fine-tuning the pose parameters multiple times.

[0095] The preset convergence criteria are the standards for determining whether the iterative optimization process has reached the expected accuracy or whether it needs to stop. Their function is to ensure that the optimization process stops after achieving a sufficiently good matching effect, avoiding unnecessary computation or overfitting. The convergence criteria provided in this application embodiment are one or more of the following conditions: the spatial distance between points in the model source point cloud and corresponding points in the scene target point cloud is minimized; the normal error between points in the model source point cloud and corresponding points in the scene target point cloud is minimized; the number of iterations meets the maximum number of iterations; the average spatial distance between points in the model source point cloud and corresponding points in the scene target point cloud is less than or equal to a preset threshold; the change in pose of the model source point cloud relative to the scene target point cloud in two adjacent iterations is less than or equal to a change threshold.

[0096] Minimizing spatial distance refers to minimizing the Euclidean distance between each point in the source point cloud of the model and its corresponding point in the target point cloud of the scene. This can be achieved by calculating the sum of squared distances between all corresponding point pairs and finding its minimum, or by using other distance metrics to quantify the degree of matching between point clouds.

[0097] Minimizing normal error means minimizing the difference between the surface normal vectors of points in the source point cloud of the model and their corresponding points in the target point cloud of the scene. This is typically measured by calculating the dot product or angle of the normal vectors at the corresponding points and seeking their maximum or minimum values ​​to ensure that the point clouds are aligned not only in position but also in surface orientation.

[0098] These conditions can be used individually or in combination. For example, the iterative optimization process terminates when any one of the following conditions is met: maximum number of iterations, average spatial distance less than a threshold, or pose change less than a threshold. The current pose is then output as the final 6D pose of the target object. For instance, firstly, in each iteration, the Euclidean distance between each point in the model's source point cloud and its nearest neighbor in the scene's target point cloud is calculated, and the average of these distances is calculated. Simultaneously, the control system can calculate the angle or dot product between the surface normal vectors of corresponding point pairs to evaluate normal consistency. During the iteration process, a maximum number of iterations can be set, such as 100. Once this limit is reached, the optimization process stops. Furthermore, an average spatial distance threshold can be set, such as 0.001 meters. When the calculated average spatial distance is less than or equal to this threshold, the point clouds are considered sufficiently aligned, and the optimization terminates.

[0099] In one specific implementation, the ICP algorithm can be used for iterative optimization to adjust the pose of the model source point cloud relative to the scene target point cloud, thus obtaining the final 6D pose.

[0100] By iteratively optimizing and adjusting the pose of the model's source point cloud relative to the scene's target point cloud, a more accurate 6D pose can be obtained. However, in practical applications, the quality of the scene's target point cloud can be affected by various factors such as ambient lighting, sensor noise, and occlusion, leading to uncertainties or localized gaps in the point cloud data. This quality variation may affect the stability and final accuracy of the pose optimization process.

[0101] To address this, this application further proposes another iterative method for obtaining the final 6D pose of the target object. (See attached document.) Figure 3 A flowchart illustrating another iterative method for obtaining the final 6D pose of a target object, as provided in this application embodiment, includes the following steps: S310 adjusts the pose of the model source point cloud relative to the scene target point cloud through iterative optimization, so as to minimize the spatial distance between the points in the model source point cloud and the corresponding points in the scene target point cloud, and obtain the intermediate 6D pose of the target object.

[0102] The intermediate 6D pose of the target object represents the initial alignment state between the model source point cloud and the scene target point cloud under the criterion of minimizing spatial distance. This intermediate pose provides a good initial value for subsequent fine-grained optimization, avoiding the problem of getting trapped in local optima when optimizing from scratch.

[0103] S320 determines the quality of the target point cloud in the scene.

[0104] The quality of a scene target point cloud refers to the degree of excellence of the scene target point cloud data in terms of completeness, density, noise level, number of outliers, and / or its fit with the real geometry. Scene target point clouds typically have high density, low noise and outliers, and can accurately reflect the surface geometric features of the target object.

[0105] In the embodiments of this application, the quality of the target scene point cloud can be determined in various ways. For example, the density of the point cloud, that is, the number of points per unit volume or unit area, can be calculated as the quality of the target scene point cloud; the proportion of effective depth values ​​within the target instance mask region can also be calculated as the quality of the target scene point cloud; and the dispersion of local point distribution in the target image can also be calculated as the quality of the target scene point cloud.

[0106] S330, if the quality of the scene target point cloud is higher than the preset quality threshold, based on the intermediate 6D pose, the pose of the model source point cloud relative to the scene target point cloud is adjusted through iterative optimization so that the normal error between the points in the model source point cloud and the corresponding points in the scene target point cloud is minimized, and the final 6D pose of the target object is obtained.

[0107] A preset quality threshold is a standard used to determine whether the quality of the target point cloud in a scene meets the requirements for more refined optimization. This threshold can be set according to the actual application scenario, sensor characteristics, and requirements for pose accuracy. For example, it can be set to a minimum point cloud density or a certain percentage of effective depth points.

[0108] In the iterative optimization process, in addition to considering the spatial distance between points, the normal direction of the point cloud surface is also considered.

[0109] Normal error is typically measured by calculating the angle or projected distance between the normal of a point in the model point cloud and the normal of the corresponding point in the scene point cloud. Minimizing normal error helps achieve fine alignment of the point cloud, especially in terms of surface details and local fit, enabling better capture of the geometry of the target object.

[0110] S340, if the quality of the scene target point cloud is lower than or equal to a preset quality threshold, the intermediate 6D pose is taken as the final 6D pose of the target object.

[0111] In practical applications, when the quality of the scene target point cloud is poor—for example, due to sensor noise, occlusion, reflection, etc., resulting in a sparse, incomplete, or large number of outliers—forcing fine optimization based on normal error may introduce new errors or even cause the optimization process to diverge, thereby reducing the accuracy and robustness of pose determination. Therefore, when the quality of the scene target point cloud is lower than or equal to a preset quality threshold, the intermediate 6D pose can be used as the final 6D pose of the target object.

[0112] Furthermore, this application embodiment also provides other iterative methods. Specifically, the control system can first employ an Iterative Closest Point (ICP) algorithm based on point-to-point distance. Starting from the initial 6D pose, it iteratively adjusts the pose of the model source point cloud to minimize the sum of squared Euclidean distances between each point in the model source point cloud and its nearest point in the scene target point cloud, until convergence, thereby obtaining the intermediate 6D pose of the target object. In each iteration, the transformation of the model point cloud under the current pose is calculated, and the pose parameters are updated. Subsequently, the quality of the scene target point cloud is determined. For example, the proportion of effective depth pixels within the target instance mask region can be calculated. If this proportion is higher than a preset 80% threshold, the point cloud quality is considered high. If the quality is higher than the threshold, the second stage of optimization is entered. In the second stage, the intermediate 6D pose is used as the new initial pose, and iterative optimization is performed using the ICP algorithm based on point-to-plane distance. In each iteration, the algorithm calculates the distance between the local tangent planes of points in the source point cloud of the model and the corresponding points in the target point cloud of the scene. The sum of the squares of these distances is used as the optimization objective. The pose transformation is solved by minimizing the objective function until convergence, and finally the final 6D pose of the target object is obtained.

[0113] This application provides a pose determination method. First, a two-dimensional image (target image) including the target object whose pose needs to be determined is segmented to obtain a target instance mask of the target object. However, the initial 6D pose of the target object is determined based on this target instance mask as the target region. The 6D pose determined using the two-dimensional image lacks true three-dimensional geometric constraints, resulting in significant errors in depth direction, rotation angle, and local surface fit. To address this, this application introduces true three-dimensional geometric constraints. By adjusting the model source point cloud generated by transforming the three-dimensional CAD model of the target object into the camera coordinate system and obtaining the relative pose of the scene target point cloud based on the target instance mask and the real scene of the target image, the initial pose is optimized in the backend. This reduces the deficiency of relying solely on the two-dimensional image and lacking depth constraints, thus improving the accuracy of 6D pose determination.

[0114] The above describes a pose determination method. This application also provides a pose determination device for performing the above pose determination method.

[0115] Appendix Figure 4 This is a schematic diagram of a pose determination device provided in an embodiment of this application. Figure 4 As shown, the device 400 includes: The first acquisition unit 401 is used to acquire a target image, wherein the target image is a two-dimensional image and includes a target object whose pose is to be determined. Segmentation unit 402 is used to segment the target image to obtain the target instance mask of the target object; The first determining unit 403 is used to determine the initial 6D pose of the target object using the target instance mask as the target region; The second acquisition unit 404 is used to acquire the model source point cloud and the scene target point cloud; the model source point cloud is generated by converting the three-dimensional CAD model of the target object to camera coordinates, and the scene target point cloud is acquired based on the target instance mask and the real scene depth information of the target image; The second determining unit 405 is used to obtain the final 6D pose of the target object by adjusting the pose of the model source point cloud relative to the scene target point cloud based on the initial 6D pose.

[0116] Optionally, obtaining the final 6D pose of the target object based on the initial 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud includes: By iteratively optimizing, the pose of the model source point cloud relative to the scene target point cloud is adjusted so that the points in the model source point cloud converge to the corresponding points in the scene target point cloud in three-dimensional space, thereby obtaining the final 6D pose of the target object.

[0117] The convergence condition is one or more of the following conditions: The spatial distance between the points in the model source point cloud and the corresponding points in the scene target point cloud is minimized; The normal error between the points in the model source point cloud and the corresponding points in the scene target point cloud is minimized; The number of iterations meets the maximum number of iterations. The average spatial distance between points in the model source point cloud and corresponding points in the scene target point cloud is less than or equal to a preset threshold. The change in pose of the model source point cloud relative to the scene target point cloud in two adjacent iterations is less than or equal to the change threshold.

[0118] Optionally, obtaining the final 6D pose of the target object based on the initial 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud includes: By iteratively optimizing, the pose of the model source point cloud relative to the scene target point cloud is adjusted to minimize the spatial distance between the points in the model source point cloud and the corresponding points in the scene target point cloud, thereby obtaining the intermediate 6D pose of the target object. Determine the quality of the target point cloud in the scene; If the quality of the scene target point cloud is higher than a preset quality threshold, based on the intermediate 6D pose, the pose of the model source point cloud relative to the scene target point cloud is adjusted through iterative optimization to minimize the normal error between the points in the model source point cloud and the corresponding points in the scene target point cloud, thus obtaining the final 6D pose of the target object. Further, if the quality of the scene target point cloud is lower than or equal to the preset quality threshold, the intermediate 6D pose is used as the final 6D pose of the target object.

[0119] Optionally, the step of segmenting the target image to obtain the target instance mask of the target object includes: The target image is processed based on an image segmentation model to obtain a target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, which includes sample data of the target object under various imaging conditions.

[0120] Optionally, the image segmentation model is the YOLO instance segmentation model.

[0121] Optionally, the training sample set is an industrial parts dataset conforming to the BOP evaluation specification, and the method for obtaining the training sample set includes: Acquire multimodal image data of the target object and camera intrinsic parameters at the time of acquisition, wherein the multimodal image data includes images of the target object under different imaging conditions; Based on the 3D CAD model of the target object, determine the true 6D pose of the target object in the multimodal image data; The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set. The hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data item is used to store the 3D CAD model of the target object, the image data item is used to store the multimodal image data, and the annotation data item is used to store the camera intrinsic parameters and the 6D pose ground truth value corresponding to each frame of the multimodal image data.

[0122] Optionally, determining the initial 6D pose of the target object using the target instance mask as the target region includes: Using the target instance mask as the target region, a target template image matching the target image is selected from a pre-established template library; A two-dimensional optical flow algorithm is used to perform pixel-level matching between the target image and the target template image to obtain the 2D-3D correspondence of the target object. The PnP algorithm is used to process the 2D-3D correspondence of the target object to obtain the initial 6D pose of the target object.

[0123] Optionally, determining the 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence of the target object includes: The following target operation is performed on the 2D-3D object of the target object after a preset number of iterations to obtain the 6D pose of the target object; The target operation includes: randomly selecting a minimum sampling set from the 2D-3D correspondence of the target object, the minimum sampling set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; and determining the number of points whose reprojection error is less than a preset threshold as the target number of points. Among them, the candidate 6D pose with the most target points is taken as the 6D pose of the target object.

[0124] This application provides a pose determination device. First, a two-dimensional image (target image) including the pose of a target object to be determined is segmented to obtain a target instance mask of the target object. Then, the initial 6D pose of the target object is determined based on the target instance mask as the target region. The 6D pose determined using a two-dimensional image lacks true three-dimensional geometric constraints, resulting in significant errors in depth direction, rotation angle, and local surface fit. Therefore, this application introduces true three-dimensional geometric constraints. By adjusting the model source point cloud generated by transforming the three-dimensional CAD model of the target object into the camera coordinate system and obtaining the relative pose of the scene target point cloud based on the target instance mask and the real scene of the target image, the initial pose is optimized in the backend. This reduces the deficiency of relying solely on two-dimensional images and improving the accuracy of 6D pose determination.

[0125] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the various steps or processes performed in any of the foregoing method embodiments.

[0126] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the various steps or processes performed in any of the foregoing method embodiments.

[0127] The computer-readable storage medium may be the aforementioned volatile memory or non-volatile memory, or it may include both volatile memory and non-volatile memory.

[0128] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0129] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated.

[0130] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and equipment embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0131] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of pose determination, the method comprising: The method includes: Acquire a target image, wherein the target image is a two-dimensional image and includes a target object whose pose is to be determined; The target image is segmented to obtain the target instance mask of the target object; Using the target instance mask as the target region, determine the initial 6D pose of the target object; Obtain the model source point cloud and the scene target point cloud; the model source point cloud is generated by converting the three-dimensional computer-aided design CAD model of the target object into camera coordinates, and the scene target point cloud is obtained based on the target instance mask and the real scene depth information of the target image; Based on the initial 6D pose, the final 6D pose of the target object is obtained by adjusting the pose of the model source point cloud relative to the scene target point cloud.

2. The method according to claim 1, characterized in that, The step of obtaining the final 6D pose of the target object based on the initial 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud includes: By iteratively optimizing, the pose of the model source point cloud relative to the scene target point cloud is adjusted so that the points in the model source point cloud converge to the corresponding points in the scene target point cloud in three-dimensional space, thereby obtaining the final 6D pose of the target object.

3. The method according to claim 2, characterized in that, The convergence condition is one or more of the following conditions: The spatial distance between the points in the model source point cloud and the corresponding points in the scene target point cloud is minimized; The normal error between the points in the model source point cloud and the corresponding points in the scene target point cloud is minimized; The number of iterations meets the maximum number of iterations. The average spatial distance between points in the model source point cloud and corresponding points in the scene target point cloud is less than or equal to a preset threshold. The change in pose of the model source point cloud relative to the scene target point cloud in two adjacent iterations is less than or equal to the change threshold.

4. The method of claim 1, wherein, The step of obtaining the final 6D pose of the target object based on the initial 6D pose by adjusting the pose of the model source point cloud relative to the scene target point cloud includes: By iteratively optimizing, the pose of the model source point cloud relative to the scene target point cloud is adjusted to minimize the spatial distance between the points in the model source point cloud and the corresponding points in the scene target point cloud, thereby obtaining the intermediate 6D pose of the target object. Determine the quality of the target point cloud in the scene; If the quality of the scene target point cloud is higher than a preset quality threshold, based on the intermediate 6D pose, the pose of the model source point cloud relative to the scene target point cloud is adjusted through iterative optimization to minimize the normal error between the points in the model source point cloud and the corresponding points in the scene target point cloud, thereby obtaining the final 6D pose of the target object.

5. The method of claim 4, wherein, The method further includes: If the quality of the target point cloud in the scene is lower than or equal to a preset quality threshold, the intermediate 6D pose is taken as the final 6D pose of the target object.

6. The method of claim 1, wherein, The segmentation process of the target image to obtain the target instance mask of the target object includes: The target image is processed based on an image segmentation model to obtain a target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, which includes sample data of the target object under various imaging conditions.

7. The method of claim 6, wherein, The image segmentation model is the YOLO instance segmentation model.

8. The pose determination method of claim 6, wherein, The training sample set is an industrial parts dataset that conforms to the BOP evaluation specification. The method for obtaining the training sample set includes: Acquire multimodal image data of the target object and camera intrinsic parameters at the time of acquisition, wherein the multimodal image data includes images of the target object under different imaging conditions; Based on the 3D CAD model of the target object, determine the true 6D pose of the target object in the multimodal image data; The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set. The hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data item is used to store the 3D CAD model of the target object, the image data item is used to store the multimodal image data, and the annotation data item is used to store the camera intrinsic parameters and the 6D pose ground truth value corresponding to each frame of the multimodal image data.

9. The method according to any one of claims 1 to 8, characterized in that, The step of determining the initial 6D pose of the target object using the target instance mask as the target region includes: Using the target instance mask as the target region, a target template image matching the target image is selected from a pre-established template library; A two-dimensional optical flow algorithm is used to perform pixel-level matching between the target image and the target template image to obtain the 2D-3D correspondence of the target object. The PnP algorithm is used to process the 2D-3D correspondence of the target object to obtain the initial 6D pose of the target object.

10. The pose determination method of claim 9, wherein, The determination of the 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence of the target object includes: The following target operation is performed on the 2D-3D object of the target object after a preset number of iterations to obtain the 6D pose of the target object; The target operation includes: randomly selecting a minimum sampling set from the 2D-3D correspondence of the target object, the minimum sampling set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; and determining the number of points whose reprojection error is less than a preset threshold as the target number of points. Among them, the candidate 6D pose with the most target points is taken as the 6D pose of the target object.

11. A pose determination apparatus, characterized in that The device includes: The first acquisition unit is used to acquire a target image, wherein the target image is a two-dimensional image and includes a target object whose pose is to be determined; A segmentation unit is used to segment the target image to obtain a target instance mask of the target object; The first determining unit is used to determine the initial 6D pose of the target object using the target instance mask as the target region; The second acquisition unit is used to acquire the model source point cloud and the scene target point cloud; the model source point cloud is generated by converting the 3D CAD model of the target object to camera coordinates, and the scene target point cloud is acquired based on the target instance mask and the real scene depth information of the target image; The second determining unit is used to obtain the final 6D pose of the target object by adjusting the pose of the model source point cloud relative to the scene target point cloud based on the initial 6D pose.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when run on a computer, causes the computer to execute the pose determination method according to any one of claims 1-10.

13. A computer program product, characterised in that, When it is run on a computer, it performs the pose determination method according to any one of claims 1-10.

Images