A pose determination method and apparatus, and related products

Abstract

This application provides a pose determination method, apparatus, and related products. The technical solution first uses an image segmentation model trained with sample data including various imaging conditions to process a target image containing the target object. Then, using the instance mask as the target region, a target template image matching the target image is matched from a pre-established template library. Pixel-level matching of the target template image and the target image effectively reduces the impact of background noise, noise from neighboring parts, or highly reflective areas on template retrieval and feature matching, thereby improving matching accuracy. Furthermore, since pixel-level matching is based on the assumption of local grayscale consistency, even when the target object has weak surface texture, no texture, or high reflectivity, it can still obtain a sufficient number of evenly distributed matching point pairs. Therefore, compared to sparse point feature matching, it can reduce the probability of feature point failure and the false matching rate.

Description

Technical Field

[0001] This application relates to the field of industrial automation technology, and in particular to a pose determination method, apparatus and related products. Background Technology

[0002] In the field of industrial automation, robots are increasingly used in scenarios such as disordered grasping, workpiece loading, flexible assembly, and quality inspection. These scenarios all require robots to accurately estimate the 6-degree-of-freedom (6D) pose of the target object from a single red-green-blue (RGB) image or RGB depth image.

[0003] FoundPose, a model-driven 6D pose determination method, establishes a correspondence between images and models using basic visual features and template matching. It can determine the 6D pose of a target object without retraining the pose network for each specific object. However, in real-world industrial scenarios, especially when the target object has weak or no surface texture, local occlusion, cluttered backgrounds, or drastic changes in lighting conditions, the 6D pose determined by FoundPose suffers from poor accuracy, failing to meet the technical requirements for pose estimation accuracy in industrial automation. Summary of the Invention

[0004] Based on the above problems, this application provides a pose determination method, apparatus and related products to improve the 6D pose determination accuracy of a target object, so that the 6D pose of the target object meets the requirements of pose estimation accuracy in the field of industrial automation.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, embodiments of this application provide a pose determination method, the method comprising: Acquire a target image, wherein the target image includes a target object whose pose is to be determined; The target image is processed based on an image segmentation model to obtain a target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, which includes sample data of the target object under various imaging conditions; Using the target instance mask as the target region, a target template image matching the target image is selected from a pre-established template library, and pixel-level matching is performed between the target template image and the target image; wherein, the template library includes the target template image; Based on pixel-level matching results, the 6D pose of the target object is determined.

[0006] Optionally, the pixel-level matching of the target template image and the target image includes: Calculate the displacement vector field between the target template image and the corresponding pixels of the target image, wherein the displacement vector field characterizes the pixel-level offset information of the surface of the target object; Based on the displacement vector field, the pixels of the target template image in the target region are mapped to the corresponding positions of the target image to establish pixel-level matching between the target template image and the target image.

[0007] Optionally, the image segmentation model is the YOLO instance segmentation model.

[0008] Optionally, the YOLO instance segmentation model is an instance segmentation model in which the detection head includes edge-sensitive loss weights; The edge-sensitive loss weight includes a first weight set for the geometric structure region of the target object and a second weight set for the non-geometric structure region of the target object; the first weight is greater than the second weight.

[0009] Optionally, the training sample set is an industrial parts dataset conforming to the BOP evaluation specification, and the method for obtaining the training sample set includes: Acquire multimodal image data of the target object and camera intrinsic parameters at the time of acquisition, wherein the multimodal image data includes images of the target object under different imaging conditions; Based on the 3D CAD model of the target object, determine the true 6D pose of the target object in the multimodal image data; The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set. The hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data item is used to store the 3D CAD model of the target object, the image data item is used to store the multimodal image data, and the annotation data item is used to store the camera intrinsic parameters and the 6D pose ground truth value corresponding to each frame of the multimodal image data.

[0010] Optionally, the imaging conditions include one or more of the following: Light intensity, shooting angle, shooting distance, occlusion information, and surface emissivity.

[0011] Optionally, the step of using the target instance mask as the target region and matching the target template image that matches the target image from a pre-established template library includes: Using the target instance mask as the target region, the visual features of the target region are obtained from the target image; Calculate the similarity between the visual feature and the template images in the template library, and take the template images with the top K similarity as the target template image; where K is a positive integer.

[0012] Optionally, determining the 6D pose of the target object based on pixel-level matching results includes: Based on the pixel-level matching results and the depth information of the target object, a 2D-3D correspondence relationship of the target object is established; Based on the PnP algorithm and the 2D-3D correspondence of the target object, the 6D pose of the target object is determined.

[0013] Optionally, determining the 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence of the target object includes: The following target operation is performed on the 2D-3D object of the target object after a preset number of iterations to obtain the 6D pose of the target object; The target operation includes: randomly selecting a minimum sampling set from the 2D-3D correspondence of the target object, the minimum sampling set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; and determining the number of points whose reprojection error is less than a preset threshold as the target number of points. Among them, the candidate 6D pose with the most target points is taken as the 6D pose of the target object.

[0014] Secondly, embodiments of this application provide a pose determination device, the device comprising: An acquisition unit is used to acquire a target image, wherein the target image includes a target object whose pose is to be determined; The segmentation unit is used to process the target image based on the image segmentation model to obtain the target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, and the training sample set includes sample data of the target object under various imaging conditions; A matching unit is configured to use the target instance mask as the target region, match a target template image that matches the target image from a pre-established template library, and perform pixel-level matching between the target template image and the target image; wherein, the template library includes the target template image; The pose determination unit is used to determine the 6-DOF 6D pose of the target object based on pixel-level matching results.

[0015] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the pose determination method as described in any of the first aspects.

[0016] Fourthly, embodiments of this application provide a computer program product, characterized in that, when it is run on a computer, it executes the pose determination method as described in any of the first aspects.

[0017] Fifthly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores a computer program; and the processor is coupled to the memory for calling and executing the computer program to implement the pose determination method as described in any of the first aspects.

[0018] Compared with the prior art, this application has the following beneficial effects: This application provides a pose determination method, apparatus, and related products. The method first processes a target image including the target object using an image segmentation model trained with sample data from various imaging conditions. The generated instance mask, compared to the instance mask generated by the Zero-Shot Instance Segmentation Model (CNOS), exhibits higher boundary accuracy, lower background noise, and a relatively complete structure. Using this instance mask as the target region, a target template image matching the target image is selected from a pre-established template library. Pixel-level matching between the target template image and the target image effectively reduces the impact of background noise, noise from adjacent parts, or highly reflective areas on template retrieval and feature matching, thereby improving matching accuracy. Furthermore, since pixel-level matching is based on the assumption of local grayscale consistency, even when the target object has weak surface texture, no texture, or high reflectivity, it can still obtain a sufficient number of evenly distributed matching point pairs. Therefore, compared to sparse point feature matching, it can reduce the probability of feature point failure and the false matching rate. Therefore, the 6D pose of the target object determined based on this pixel-level matching result has higher accuracy than FoundPose. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a pose determination method provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for obtaining a training sample set, as provided in an embodiment of this application; Figure 3 This is a schematic diagram of a pose determination device provided in an embodiment of this application. Detailed Implementation

[0021] As described earlier, in real industrial scenarios, especially in harsh industrial environments such as weak or no surface texture of the target object, partial occlusion of the target object, cluttered background, or drastic changes in lighting conditions, using FoundPose to determine the 6D pose of the target object will result in poor determination accuracy, making it difficult to meet the technical requirements for pose estimation accuracy in the field of industrial automation.

[0022] It should be noted that the 6D pose of the target object refers to a set of parameters representing a rigid transformation between the target object's own 3D coordinate system and the camera coordinate system. The 6D pose includes 3 rotational degrees of freedom and 3 translational degrees of freedom. Through this 6D pose, the point cloud of the target object's 3D computer-aided design (CAD) model can be transformed from the object coordinate system to the camera coordinate system, thereby uniquely determining the target object's position and orientation in space. For example, in the field of robotics, a robot's control system can use the target object's 6D pose to align the gripping point or assembly point with the target object's actual spatial position and orientation, thereby achieving high-precision automation.

[0023] Research has revealed that FoundPose's determination of the 6D pose of a target object primarily relies on instance masks generated by CNOS. However, CNOS is a zero-shot instance segmentation model that works by utilizing pre-trained basic visual features on large-scale general datasets to achieve generalized recognition of unseen categories. However, when the target object has complex geometric structures (e.g., porous, deeply recessed, or thin-walled objects), special material characteristics (e.g., highly reflective metals), or variable imaging conditions (e.g., uneven lighting, background occlusion), CNOS suffers from the following limitations: CNOS relies on pre-trained features derived from natural images, where object boundaries are typically defined at the semantic or contour level. However, the boundaries of target objects with complex geometric structures are often composed of chamfers, edges, or assembly gaps. These subtle geometric structures are difficult for CNOS to capture accurately, leading to blurred or inaccurate instance mask boundaries. Furthermore, material features or varying imaging conditions can produce virtual images resembling the background or other objects, or create shadow areas. CNOS can easily missegment these areas as background or adjacent parts, resulting in missed segmentation, or incorrectly include shadow areas in the mask, introducing background noise. Using this instance mask for template retrieval and feature matching can lead to inaccurate 2D-3D correspondences of the target object, thus affecting the accuracy of 6D pose determination.

[0024] It should be noted that the 2D-3D correspondence refers to the matching relationship established between the two-dimensional pixels of an image and the three-dimensional point cloud of the three-dimensional CAD model of the target object.

[0025] For example, in a scenario where a robotic arm grasps a metal part, the target object is an industrial part with a complex geometry and special material characteristics, whose surface has highly reflective areas and includes multiple holes. When the target image is processed by CNOS, the high reflectivity causes the boundaries of the industrial part to be blurred, resulting in the incorrect inclusion of the background area of ​​the workbench in the instance mask, while the hole area is regarded as a valid part of the industrial part.

[0026] Furthermore, when establishing 2D-3D correspondences, FoundPose mainly relies on sparse feature point matching. On the surface of industrial parts with weak or no texture, the inability to extract a sufficient number of stable feature points can lead to insufficient matching point pairs or a large number of mismatches, which further reduces the accuracy of 6D pose determination.

[0027] In view of this, the pose determination method provided in this application first uses an image segmentation model trained with sample data including various imaging conditions to process a target image including the target object. The generated instance mask has higher boundary accuracy, lower background noise, and a relatively complete structure compared to the instance mask generated by CNOS. Using this instance mask as the target region, a target template image matching the target image is matched from a pre-established template library. Pixel-level matching between the target template image and the target image can effectively reduce the impact of background noise, noise from adjacent parts, or highly reflective areas on template retrieval and feature matching, thereby improving matching accuracy. Furthermore, since pixel-level matching is based on the assumption of local grayscale consistency, even when the target object has weak surface texture, no texture, or high reflectivity, it can still obtain a sufficient number of evenly distributed matching point pairs. Therefore, compared to sparse point feature matching, it can reduce the probability of feature point failure and the false matching rate. Therefore, based on this pixel-level matching result, the determined 6D pose of the target object has higher accuracy than FoundPose.

[0028] The pose determination method provided in this application can be applied to the fields of computer vision, 3D machine vision, intelligent perception of industrial robots, and other pose estimation fields. This application is not limited to these fields. To facilitate understanding by those skilled in the art, the following description uses intelligent perception of industrial robots, specifically their applications in scenarios such as disordered grasping, workpiece loading, flexible assembly, and quality inspection, as examples.

[0029] It should be noted that the execution subject of the pose determination method provided in this application embodiment can be an electronic device with computing capabilities. Specifically, the electronic device includes, but is not limited to, personal computers, embedded vision processing systems, robot control systems, etc. It can also be a hardware module or software platform deployed in the above-mentioned electronic device, and this application embodiment is not limited. For ease of explanation, the execution subject of this application embodiment is described using a robot control system as an example.

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0031] Appendix Figure 1 This is a flowchart illustrating a pose determination method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps: S10, acquire the target image, wherein the target image includes the target object whose pose is to be determined.

[0032] The target object can be an industrial part with complex geometry and / or specific material characteristics. For example, the target object is an industrial valve, which has a highly reflective metallic surface and a complex structure. The target image is an RGB image that includes the target object or a Red-Green-Blue-Depth Image (RGBD) that includes depth information.

[0033] In this embodiment of the application, the control system needs to acquire a target image including the target object in order to determine the 6D pose of the target object. For example, if the control system needs to determine the 6D pose of a valve so that the robot's robotic arm can accurately grasp and assemble it, it can acquire RGB and / or RGBD images of the target valve in real time through a camera mounted at the end of the robot's robotic arm or a camera mounted at a fixed position on the robot.

[0034] It should be noted that the camera in this application embodiment can be an RGB camera, a depth camera, or an infrared camera, etc., and this application embodiment is not limited to any particular type.

[0035] S20: Process the target image based on the image segmentation model to obtain the target instance mask of the target object.

[0036] In this embodiment of the application, after the control system acquires the target image, it performs segmentation processing on the target image based on the trained image segmentation model to obtain the instance mask of the target object, i.e., the target instance mask.

[0037] In this embodiment, an image segmentation model trained using a pre-constructed training sample set is used. The training sample set includes sample data of the target object under various imaging conditions. Imaging conditions refer to external variables and / or acquisition parameters of the target object when acquiring images including the target object, such as the target object's illumination intensity, shooting angle, shooting distance, occlusion, or surface reflectivity.

[0038] In practical applications, the training sample set can be an industrial parts dataset that conforms to the Benchmark for 6DObject Pose Estimation (BOP) evaluation specification. Based on this, this application also provides an automated method for obtaining the training sample set. (Appendix) Figure 2 A flowchart illustrating a method for obtaining a training sample set, as provided in this application embodiment, includes the following steps: S210, acquire the multimodal image data of the target object and the camera intrinsic parameters at the time of acquisition.

[0039] The multimodal image data includes images of the target object under different imaging conditions. These different imaging conditions can include changes in the light source, adjustment of light intensity, adjustment of shooting distance and angle, introduction of partial occlusion, and changes in the background area. In this embodiment, the multimodal image data should comprehensively reflect the various imaging conditions that the target object may encounter in practical applications, thereby improving the robustness of the trained image segmentation model to complex environments.

[0040] Camera intrinsics are key parameters for pose estimation, used to convert 2D image coordinates into 3D camera coordinates. Camera intrinsics can be obtained offline through camera calibration tools (such as the calibration module provided by OpenCV), or in real-time via sensors integrated into the camera during each acquisition, or through other methods; this application does not limit the methods used.

[0041] S220 determines the true 6D pose of the target object in each frame of the image based on the 3D CAD model of the target object.

[0042] Each frame of the image data consists of a single frame of multimodal image data.

[0043] This application embodiment can determine the 6D pose truth value of the target object in each frame of an image based on a 3D CAD model of the target object. For example, the 3D CAD model of the target object is rendered into an image under known camera intrinsics and a preset pose to obtain a projected image. The projected image is compared with the actual acquired image, and the optimal pose parameters that minimize the difference between the projected image and the actual image are found through manual adjustment or a preset algorithm. The optimal pose parameters are the 6D pose truth value of the target object in that frame of the image. This application embodiment can also determine the 6D pose truth value of the target object in each frame of the image through other methods, such as structured light scanning reconstruction. This application embodiment is not limited to these methods.

[0044] S230: The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set.

[0045] In this embodiment, the hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data items store the 3D CAD model of the target object. It can be stored in a standard CAD model file format, such as Standard for the Exchange of Product Data (STEP), Initial Graphics Exchange Specification (IGES), or Stereolithography (STL), which can completely describe the geometric topology and surface information of the target object. The image data items store multimodal image data. Image data can be stored at its original resolution and format (e.g., PNG, JPG, TIFF), or it can be stored according to imaging conditions such as acquisition time and viewing angle. The annotation data items store the camera intrinsic parameters and 6D pose ground truth values ​​corresponding to each frame of the multimodal image data. In this embodiment, the annotation data item can be a directory containing multiple JSON or YAML files, each file corresponding to a frame image and recording the camera intrinsic parameter matrix (e.g., focal length, principal point, distortion coefficients) and the true 6D pose of the target object in the camera coordinate system for that frame image. In practical use, the hierarchical data structure can be organized using a file system directory structure. For example, the root directory contains multiple object folders, and each object folder contains subfolders such as model data items, image data items, and annotation data items.

[0046] For example, the joint parts in the robotic arm operation scenario are used as the target objects to construct the training sample set.

[0047] First, an RGBD camera is used to acquire multimodal image data of the part on the industrial production line. During the acquisition process, the shooting angle, shooting distance, lighting conditions, and occlusion can be intentionally changed to simulate the diversity of the actual working environment. Simultaneously, camera intrinsic parameters are acquired and recorded for each acquisition using a camera calibration tool. Next, the 3D CAD model of the part is rendered onto the acquired images, and the precise 6D pose of each frame is determined through manual alignment or feature matching optimization using interactive software. Finally, the directory structure is organized according to the BOP specification: a folder named "Part Name" is created in the root directory. This folder includes model data items, image data items, and annotation data items.

[0048] The above approach collects multimodal image data under various imaging conditions and combines it with CAD models to determine the true pose value, making the training data closer to real industrial scenarios. The BOP hierarchical data structure is adopted to improve the standardization of data and provide a unified benchmark for subsequent segmentation model training and pose estimation.

[0049] In this embodiment, the image segmentation model can be a general image segmentation model, such as a Mask Region-Based Convolutional Neural Network (Mask R-CNN) or a U-Shaped Convolutional Neural Network (U-Net). Alternatively, the image segmentation model can be the YOLO instance segmentation model. The YOLO instance segmentation model is a graph segmentation model capable of simultaneously performing object detection and instance segmentation. This model can directly predict the bounding box, category, and pixel-level instance mask of the target within a unified network. For example, models such as YOLOv5, YOLOv7, or YOLOv8 can be used, achieving a good balance between speed and accuracy. The YOLO instance segmentation model can directly output the instance mask of each object in the input image in an end-to-end manner. This approach avoids the complex region proposal generation and feature extraction processes of traditional two-stage segmentation methods, significantly improving processing efficiency.

[0050] For example, let's take the YOLOv6 instance segmentation model as an example. First, the control system constructs a training sample set using sample data containing the target object under various imaging conditions, and trains the YOLOv6 instance segmentation model. During training, the model learns the visual features and boundary information of the target object, enabling it to accurately identify and segment the target object. In practical applications, when the control system acquires a target image containing the target object, this image is input into the trained YOLOv6 instance segmentation model. The YOLOv6 model processes the image in real time and outputs an instance mask corresponding to the target object. This mask clearly outlines the contour of the target object and distinguishes it from the background or other irrelevant areas. Then, this target instance mask can be used for template matching and pixel-level matching, and based on the matching results, the 6D pose of the target object can be generated.

[0051] In some of the solutions described above in this application, the YOLO instance segmentation model is proposed as the image segmentation model for obtaining target instance masks, in order to obtain more accurate target regions and provide reliable region constraints for subsequent template matching and pose determination. However, in this process, the conventional YOLO instance segmentation model pays consistent attention to the loss of all regions of the target object. However, the geometric structure region of the target object is the key to determining the accuracy of the mask boundary contour. If the error in this region is not given enough attention, inaccurate mask boundaries, missed or incorrect segmentation, or even the introduction of background or neighboring regions into the target region can easily occur, which will interfere with the accuracy of subsequent template retrieval, pixel matching, and pose determination.

[0052] In view of this, this application further proposes an improved YOLO instance segmentation model. This YOLO instance segmentation model is an instance segmentation model where the detection head includes edge-sensitive loss weights; wherein the edge-sensitive loss weights include a first weight set for the geometric structural regions of the target object, and a second weight set for the non-geometric structural regions of the target object. The first weight is greater than the second weight; for example, the first weight is 0.6 and the second weight is 0.4.

[0053] The detection head of the YOLO instance segmentation model is responsible for the final prediction output. Its function is to generate an instance mask of the target object based on the features extracted by the model. The detection head includes edge-sensitive loss weights, meaning that when calculating the difference between the model's prediction and the true label, different importance is assigned to the region (geometric or non-geometric region) where the pixel is located. The introduction of these loss weights aims to guide the model to pay more attention to regions that are critical to the segmentation quality during training.

[0054] Edge-sensitive loss weights are a mechanism for adjusting the contribution of a loss function. They apply differentiated penalties to prediction errors based on the characteristics of different regions of the target object. For example, during training, the weight of a pixel in the total loss can be dynamically adjusted based on whether it is located within the geometric structure region of the target object. For instance, a weight map can be introduced into the loss calculation of the mask branch in the YOLO instance segmentation model, dynamically adjusting the loss contribution based on whether a pixel is located within a geometric structure region. Alternatively, the loss function can be modified to apply a higher penalty factor to pixels closer to the edge of the target object when calculating the mask loss.

[0055] As a specific implementation method, when training the YOLO instance segmentation model, in addition to providing the original image and the target instance mask, a corresponding geometric structure region mask can be provided for the target object in the training sample set. This geometric structure region mask can be obtained by edge extraction from the 3D CAD model of the target object or by performing morphological operations on the labeled instance mask. During the training process of the YOLO instance segmentation model, its detector head applies different weights based on this geometric structure region mask when calculating the loss of the mask branch. Specifically, for pixels marked as geometric features in the geometric structure region mask, their corresponding loss value is multiplied by a larger first weight, such as 0.6; while for pixels marked as non-geometric features in the geometric structure region mask, their corresponding loss value is multiplied by a smaller second weight, such as 0.4. In this way, the YOLO instance segmentation model will focus more on reducing the segmentation error of the geometric structure region during backpropagation and parameter updates, thereby enabling the trained YOLO instance segmentation model to output a target instance mask with more accurate boundaries and clearer geometric details during the inference stage.

[0056] By introducing differentiated weights into the detection head of the YOLO instance segmentation model, the segmentation accuracy of the model for key geometric regions of the target object can be improved without significantly altering the model structure or increasing the inference burden, thereby obtaining a higher quality target instance mask.

[0057] S30: Using the target instance mask as the target region, a target template image matching the target image is selected from the pre-established template library, and pixel-level matching is performed between the target template image and the target image.

[0058] In this embodiment, the template library is a pre-established database or data structure collection that stores template images corresponding to multiple target objects. It provides reference templates with known geometric shapes, texture features, and scale information when matching with target objects in target images, thereby assisting in achieving high-precision pixel-level matching and alignment within the target area. These template images typically represent the standard appearance of the target object under different viewing angles and lighting conditions. The template library is constructed to provide a reference benchmark for target object recognition and pose estimation.

[0059] The control system can use the target instance mask obtained in step S20 as the target region and match the target template image that matches the target image from the template library. In one specific implementation, the target template images can be the K template images that are determined to best match the target region of the current target image after similarity calculation and filtering. Here, K is a positive integer, for example, K is 5.

[0060] In one example, the control system can use a target instance mask as the target region and extract the visual features of the target region from the target image. The control system then calculates the similarity between the visual features and template images in a template library, and uses the template images with the highest similarity (top K) as the target template image.

[0061] Visual features are attributes in an image that can be used to describe and distinguish target objects, including but not limited to edges, corners, textures, and color distribution. Acquiring visual features of a target region aims to extract visual information that represents the essential attributes of the target object from the target region defined by a target instance mask. By limiting feature extraction to the target region, interference from background noise and non-target objects can be effectively eliminated, ensuring that the acquired features are a pure representation of the target object itself. For example, a control system can first crop an image patch of the target region based on the target instance mask, and then perform feature extraction on that patch; alternatively, features can be extracted from the entire image, and then feature points or feature descriptors falling within the target region can be selected based on the target instance mask.

[0062] Similarity calculation is the process of quantifying the degree of similarity between two visual features. Its purpose is to establish a comparable matching score between each template image in the template library and the target region of the current target image. Embodiments of this application can employ various similarity calculation methods. For example, Euclidean distance, cosine similarity, or Hamming distance can be used to compare the distance between feature descriptors; a smaller distance generally indicates a higher similarity.

[0063] The following is a concrete example to illustrate this. Suppose the target object whose pose needs to be determined is an industrial gear, and a target instance mask of the gear in the target image has already been obtained through an image segmentation model. First, the robot's control system uses this target instance mask to precisely delineate the region where the gear is located in the target image, i.e., the target region. Next, visual features are extracted from this target region. For example, the Scale-Invariant Feature Transform (SIFT) algorithm can be used to detect and describe key points on the gear surface, generating a series of SIFT feature descriptors. Simultaneously, a template library pre-stores multiple template images of the gear under different viewpoints and lighting conditions, and each template image also has its corresponding SIFT feature descriptor pre-extracted and stored. Then, the system compares the gear SIFT feature descriptor extracted from the target image with the SIFT feature descriptors of all template images in the template library, calculating their similarity. After calculating the similarity of all template images, the robot's control system sorts them from highest to lowest similarity. Assuming a preset K value of 5, the system will select the top 5 template images with the highest similarity as the target template images for matching with the target image. These 5 selected template images will serve as candidates for subsequent pixel-level matching to further refine the matching process.

[0064] The control system uses the target instance mask obtained in step S20 as the target region. After matching the target template image that matches the target image from the template library, it can perform pixel-level matching between the target template image and the target image.

[0065] In one possible implementation, the control system can accurately capture pixel-level motion information by calculating the displacement vector field between corresponding pixels in the target template image and the target image. Here, the displacement vector field refers to the relative offset of each pixel between the target template image and the target image. The displacement vector field can precisely quantify the local deformation or positional differences of the target object's surface between the two images, thereby meticulously capturing pixel-level motion information.

[0066] Then, based on the displacement vector field, the pixels in the target region of the target template image are mapped to their corresponding positions in the target image to establish pixel-level matching between the target template image and the target image. This pixel-level matching ensures that each feature point on the target template image can find its accurate corresponding point in the target image, thus forming a high-precision 2D-2D correspondence. In one specific implementation, the coordinates of each pixel in the target template image can be directly added to its corresponding displacement vector to obtain its new coordinates in the target image. For example, for each pixel (xt, yt) in the target region of the target template image, its corresponding displacement vector (dx, dy) in the displacement vector field is queried. Then, the pixel is mapped to its new position (xt + dx, yt + dy) in the target image. Furthermore, since the displacement vector is usually a floating-point number, the new coordinates may not be integers. Therefore, interpolation algorithms (such as bilinear interpolation or cubic spline interpolation) can be used to determine the pixel value of the target image at the non-integer coordinates, thereby completing the pixel mapping and matching.

[0067] In another specific implementation, the control system can perform pixel-level matching between the target template image and the target image based on a two-dimensional optical flow method. Specifically, the two-dimensional optical flow method uses the target template image as the reference frame and the target region in the target image defined by the target instance mask as the current frame. Assuming that the brightness of the same pixel remains basically unchanged between two frames, the two-dimensional displacement vector of each pixel in the image plane is obtained by solving the optical flow constraint equation, forming an optical flow field. The target template image is then registered and aligned to the target image based on this optical flow field, achieving pixel-level or even sub-pixel-level template matching. The two-dimensional optical flow method can employ a sparse optical flow method based on local windows or a dense optical flow method to obtain the full-pixel displacement field, which can be selected according to computational resources and accuracy requirements.

[0068] Through the above technical solution, this application can effectively achieve high-precision pixel-level matching between the target template image and the target image, avoiding misalignment, thus providing a more accurate and reliable foundation for the subsequent establishment of 2D-3D correspondence and 6D pose solving.

[0069] S40 determines the 6D pose of the target object based on pixel-level matching results.

[0070] In one specific implementation, the control system can establish a 2D-3D correspondence between the target object and the pixel-level matching results and the depth information of the target object. Then, the control system determines the 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence.

[0071] The embodiments of this application can establish the 2D-3D correspondence of the target object in a variety of ways.

[0072] In one possible approach, the control system can acquire a depth map corresponding to the target image. For each pixel in the target image from the pixel-level matching results, its depth value in the depth map, combined with known camera intrinsic parameters, is back-projected into 3D space to obtain the pixel's 3D coordinates in the camera coordinate system. Simultaneously, the pixel-level matching results also provide the association between the target image pixel and its corresponding pixel in the template image. Each pixel in the template image typically has its 3D coordinates pre-stored on the 3D CAD model of the target object. In this way, a correspondence can be established between the pixels of the target object in the target image and the points in the 3D CAD model of the target object—that is, the 2D-3D correspondence of the target object.

[0073] In another possible approach, after pixel-level matching is completed, the control system, for each matched pixel in the target image, directly obtains its corresponding 3D point on the 3D CAD model by querying the pre-established correspondence between the template image and the 3D CAD model, thereby establishing a 2D-3D object of the target object.

[0074] Furthermore, if the target image itself is acquired by an RGBD camera, the depth information of the target image pixels can be directly obtained from the depth map, and combined with the camera intrinsic parameters to convert it into three-dimensional points in the camera coordinate system, thereby establishing a 2D-3D correspondence between the target object and the target.

[0075] For example, while acquiring the target image, the control system can use a structured light camera or LiDAR to obtain point cloud data of the target object. The target image pixels in the pixel-level matching results can be directly matched with corresponding 3D points in the point cloud data. These 3D points can serve as 3D information of the target object, establishing a correspondence between them and the 2D pixels in the target image.

[0076] It should be noted that the 2D-3D correspondence of the target object can also be established in various ways in the embodiments of this application, and this application is not limited to any particular method.

[0077] In this embodiment, the control system can directly utilize the Perspective-n-Point Algorithm (PnP) to determine the 6D pose of the target object by combining the 2D-3D object relationship. For example, the control system directly inputs the established 2D-3D correspondence into the PnP algorithm, which can solve for the rotation matrix and translation vector of the target object, i.e., the 6D pose.

[0078] In another example, the control system can iterate the following target operations on the 2D-3D correspondence of the target object and iterate a preset number of times to obtain the 6D pose of the target object.

[0079] The target operation includes: randomly sampling a minimum sample set from the 2D-3D correspondence of the target object, the minimum sample set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sample set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; determining the number of points with a reprojection error less than a preset threshold as the target point number; wherein, the candidate 6D pose with the largest number of target points is taken as the 6D pose of the target object.

[0080] The preset number of iterations refers to the number of times the target operation is repeatedly executed. Its purpose is to find the optimal solution or a result that satisfies specific conditions through multiple attempts. In the pose determination process, the preset number of iterations ensures that the algorithm has sufficient opportunities to explore different sampling combinations, thereby increasing the probability of finding the accurate pose. This preset number of iterations can be a fixed value, such as hundreds or thousands, to balance computational efficiency and robustness of the results; or it can be dynamically adjusted based on the algorithm's convergence, for example, stopping when multiple consecutive iterations fail to significantly improve the results.

[0081] Randomly selecting the minimum sample set from the 2D-3D correspondence of the target object means randomly selecting a subset containing the fewest number of point pairs from all established 2D-3D correspondence point pairs. This randomness helps avoid sampling bias and ensures that the algorithm can handle various possible mismatches. The sampling method can be uniform random sampling, where each point pair has an equal probability of being selected; or weighted random sampling, for example, assigning different weights based on the feature quality or confidence level of the point pairs, prioritizing the selection of higher-quality point pairs.

[0082] In this embodiment, the minimum sampling set includes at least four sets of corresponding points, which ensures the computational efficiency of the PnP algorithm while satisfying the computational requirements for solving the problem. In practical applications, four sets of points can be precisely selected, or slightly more than four sets of points, such as five or six sets, can be selected to increase redundancy and robustness.

[0083] Calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm means using the PnP algorithm to quickly calculate a candidate 6D pose from a randomly selected minimum sampling set.

[0084] Reprojection error refers to the distance between the projected 3D point onto the 2D image plane using the currently calculated candidate 6D pose, and the actual observed 2D image point. This distance reflects the consistency between the current candidate pose and all corresponding points. The error is usually calculated using Euclidean distance, which is the straight-line distance between two 2D points on the image plane, or its square; however, this embodiment does not limit the calculation.

[0085] Determining the number of points with reprojection errors less than a preset threshold as the target number means that if the reprojection error of a point is less than the preset threshold, then that point is considered compatible with the current candidate pose, i.e., it is an interior point. The target number of points refers to the total number of points with reprojection errors less than the preset threshold in all 2D-3D correspondences. This threshold can be set according to factors such as image noise level and feature matching accuracy, for example, it can be set to 2 pixels.

[0086] In addition, embodiments of this application may also employ the PnP algorithm, which combines the Random Sample Consensus (RANSAC) algorithm, to process the 2D-3D correspondence of the target object and obtain the 6D pose of the target object.

[0087] Furthermore, in this embodiment, the 6D pose of the target object obtained by the above method can be referred to as the initial pose. In this embodiment, the initial 6D pose can also be further optimized to improve the determination accuracy.

[0088] Specifically, the control system uses the initial 6D pose as the initial value for iteration, transforms the 3D CAD model of the target object into the camera coordinate system according to the initial 6D pose, and generates a model source point cloud aligned with the real scene; at the same time, based on the target instance mask obtained in step S20 and the depth information of the real scene, it extracts the effective point set belonging to the target object from the depth map and constructs the scene target point cloud.

[0089] In one specific implementation, the control system can use the Iterative Closest Point (ICP) algorithm to perform geometric registration between the model source point cloud and the scene target point cloud. The ICP algorithm establishes a nearest neighbor correspondence between the two point clouds, constructs a rigid body transformation optimization target, and iteratively updates the rotation matrix and translation vector, thereby refining the initial 6D pose until one or more of the preset maximum number of iterations, registration error convergence threshold, or pose increment threshold are met.

[0090] To further enhance the local surface fitting accuracy, a point-to-surface ICP algorithm based on normal constraints can be used for optimization. This algorithm introduces the distance constraint between the source point and the local tangent plane of the target point into the registration cost, so that the optimization process simultaneously considers the spatial position error of the point cloud and the consistency of the surface orientation, thereby achieving higher pose accuracy in terms of rotation direction and local geometric fitting degree.

[0091] By using the point cloud-level geometric consistency constraints mentioned above, this application can further eliminate rotational deviations and translational drifts in the initial pose based on image-level pose solving, significantly improving the usability of the final 6D pose results in robotic arm grasping, positioning and assembly tasks.

[0092] This application provides a pose determination method. First, an image segmentation model trained with sample data including various imaging conditions is used to process a target image containing the target object. The generated instance mask has higher boundary accuracy, lower background noise, and a relatively complete structure compared to the instance mask generated by CNOS. Using this instance mask as the target region, a target template image matching the target image is selected from a pre-established template library. Pixel-level matching between the target template image and the target image effectively reduces the impact of background noise, noise from neighboring parts, or highly reflective areas on template retrieval and feature matching, thereby improving matching accuracy. Furthermore, since pixel-level matching is based on the assumption of local grayscale consistency, it can still obtain a sufficient number of evenly distributed matching point pairs even when the target object has weak surface texture, no texture, or high reflectivity. Therefore, compared to sparse point feature matching, it can reduce the probability of feature point failure and the false matching rate. Thus, based on this pixel-level matching result, the determined 6D pose of the target object has higher accuracy than FoundPose.

[0093] The above describes a pose determination method. This application also provides a pose determination device for performing the above pose determination method.

[0094] Appendix Figure 3 This is a schematic diagram of a pose determination device provided in an embodiment of this application. Figure 3 As shown, the device includes: Acquisition unit 301 is used to acquire a target image, wherein the target image includes a target object whose pose is to be determined; The segmentation unit 302 is used to process the target image based on an image segmentation model to obtain a target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, and the training sample set includes sample data of the target object under various imaging conditions; The matching unit 303 is used to use the target instance mask as the target region, match the target template image that matches the target image from the pre-established template library, and perform pixel-level matching between the target template image and the target image; wherein, the template library includes the target template image; The pose determination unit 304 is used to determine the 6-DOF 6D pose of the target object based on pixel-level matching results.

[0095] Optionally, the pixel-level matching of the target template image and the target image includes: Calculate the displacement vector field between the target template image and the corresponding pixels of the target image, wherein the displacement vector field characterizes the pixel-level offset information of the surface of the target object; Based on the displacement vector field, the pixels of the target template image in the target region are mapped to the corresponding positions of the target image to establish pixel-level matching between the target template image and the target image.

[0096] Optionally, the image segmentation model is the YOLO instance segmentation model.

[0097] Optionally, the YOLO instance segmentation model is an instance segmentation model in which the detection head includes edge-sensitive loss weights; The edge-sensitive loss weight includes a first weight set for the geometric structure region of the target object and a second weight set for the non-geometric structure region of the target object; the first weight is greater than the second weight.

[0098] Optionally, the training sample set is an industrial parts dataset conforming to the BOP evaluation specification, and the method for obtaining the training sample set includes: Acquire multimodal image data of the target object and camera intrinsic parameters at the time of acquisition, wherein the multimodal image data includes images of the target object under different imaging conditions; Based on the 3D CAD model of the target object, determine the true 6D pose of the target object in the multimodal image data; The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set. The hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data item is used to store the 3D CAD model of the target object, the image data item is used to store the multimodal image data, and the annotation data item is used to store the camera intrinsic parameters and the 6D pose ground truth value corresponding to each frame of the multimodal image data.

[0099] Optionally, the imaging conditions include one or more of the following: Light intensity, shooting angle, shooting distance, occlusion information, and surface emissivity.

[0100] Optionally, the step of using the target instance mask as the target region and matching the target template image that matches the target image from a pre-established template library includes: Using the target instance mask as the target region, the visual features of the target region are obtained from the target image; Calculate the similarity between the visual feature and the template images in the template library, and take the template images with the top K similarity as the target template image; where K is a positive integer.

[0101] Optionally, determining the 6D pose of the target object based on pixel-level matching results includes: Based on the pixel-level matching results and the depth information of the target object, a 2D-3D correspondence relationship of the target object is established; Based on the PnP algorithm and the 2D-3D correspondence of the target object, the 6D pose of the target object is determined.

[0102] Optionally, determining the 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence of the target object includes: The following target operation is performed on the 2D-3D object of the target object after a preset number of iterations to obtain the 6D pose of the target object; The target operation includes: randomly selecting a minimum sampling set from the 2D-3D correspondence of the target object, the minimum sampling set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; and determining the number of points whose reprojection error is less than a preset threshold as the target number of points. Among them, the candidate 6D pose with the most target points is taken as the 6D pose of the target object.

[0103] This application provides a pose determination device. First, it uses an image segmentation model trained with sample data from various imaging conditions to process a target image including the target object. The generated instance mask, compared to the instance mask generated by CNOS, exhibits higher boundary accuracy, lower background noise, and a relatively complete structure. Using this instance mask as the target region, a target template image matching the target image is selected from a pre-established template library. Pixel-level matching between the target template image and the target image effectively reduces the impact of background noise, noise from neighboring parts, or highly reflective areas on template retrieval and feature matching, thereby improving matching accuracy. Furthermore, since pixel-level matching is based on the assumption of local grayscale consistency, it can still obtain a sufficient number of evenly distributed matching point pairs even when the target object has weak surface texture, no texture, or high reflectivity. Therefore, compared to sparse point feature matching, it can reduce the probability of feature point failure and the false matching rate. Thus, based on this pixel-level matching result, the determined 6D pose of the target object has higher accuracy than FoundPose.

[0104] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the various steps or processes performed in any of the foregoing method embodiments.

[0105] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the various steps or processes performed in any of the foregoing method embodiments.

[0106] The computer-readable storage medium may be the aforementioned volatile memory or non-volatile memory, or it may include both volatile memory and non-volatile memory.

[0107] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0108] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated.

[0109] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and equipment embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0110] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A pose determination method, characterized in that, The method includes: Acquire a target image, wherein the target image includes a target object whose pose is to be determined; The target image is processed based on an image segmentation model to obtain a target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, which includes sample data of the target object under various imaging conditions; Using the target instance mask as the target region, a target template image matching the target image is selected from a pre-established template library, and pixel-level matching is performed between the target template image and the target image; wherein, the template library includes the target template image; Based on pixel-level matching results, the 6D pose of the target object is determined.

2. The pose determination method according to claim 1, characterized in that, The pixel-level matching of the target template image and the target image includes: Calculate the displacement vector field between the target template image and the corresponding pixels of the target image, wherein the displacement vector field characterizes the pixel-level offset information of the surface of the target object; Based on the displacement vector field, the pixels of the target template image in the target region are mapped to the corresponding positions of the target image to establish pixel-level matching between the target template image and the target image.

3. The pose determination method according to claim 1, characterized in that, The image segmentation model is the YOLO instance segmentation model.

4. The pose determination method according to claim 3, characterized in that, The YOLO instance segmentation model is an instance segmentation model in which the detection head includes edge-sensitive loss weights; The edge-sensitive loss weight includes a first weight set for the geometric structure region of the target object and a second weight set for the non-geometric structure region of the target object; the first weight is greater than the second weight.

5. The pose determination method according to claim 1, characterized in that, The training sample set is an industrial parts dataset that conforms to the BOP evaluation specification. The method for obtaining the training sample set includes: Acquire multimodal image data of the target object and camera intrinsic parameters at the time of acquisition, wherein the multimodal image data includes images of the target object under different imaging conditions; Based on the 3D CAD model of the target object, determine the true 6D pose of the target object in the multimodal image data; The multimodal image data, camera intrinsic parameters, and the 6D pose ground truth of each frame in the multimodal image data are organized into a hierarchical data structure according to the BOP evaluation specification to obtain the training sample set. The hierarchical data structure includes at least: model data items, image data items, and annotation data items. The model data item is used to store the 3D CAD model of the target object, the image data item is used to store the multimodal image data, and the annotation data item is used to store the camera intrinsic parameters and the 6D pose ground truth value corresponding to each frame of the multimodal image data.

6. The pose determination method according to any one of claims 1-5, characterized in that, The imaging conditions include one or more of the following: Light intensity, shooting angle, shooting distance, occlusion information, and surface emissivity.

7. The pose determination method according to claim 1, characterized in that, The step of using the target instance mask as the target region and matching the target template image that matches the target image from a pre-established template library includes: Using the target instance mask as the target region, the visual features of the target region are obtained from the target image; Calculate the similarity between the visual feature and the template images in the template library, and take the template images with the top K similarity as the target template image; where K is a positive integer.

8. The pose determination method according to claim 1, characterized in that, The determination of the 6D pose of the target object based on pixel-level matching results includes: Based on the pixel-level matching results and the depth information of the target object, a 2D-3D correspondence relationship of the target object is established; Based on the PnP algorithm and the 2D-3D correspondence of the target object, the 6D pose of the target object is determined.

9. The pose determination method according to claim 8, characterized in that, The determination of the 6D pose of the target object based on the PnP algorithm and the 2D-3D correspondence of the target object includes: The following target operation is performed on the 2D-3D object of the target object after a preset number of iterations to obtain the 6D pose of the target object; The target operation includes: randomly selecting a minimum sampling set from the 2D-3D correspondence of the target object, the minimum sampling set including at least 4 sets of corresponding points; calculating the candidate 6D pose of the target object based on the minimum sampling set and the PNP algorithm; calculating the reprojection error of all other points in the 2D-3D correspondence of the target object based on the candidate 6D pose; and determining the number of points whose reprojection error is less than a preset threshold as the target number of points. Among them, the candidate 6D pose with the most target points is taken as the 6D pose of the target object.

10. A pose determination device, characterized in that, The device includes: An acquisition unit is used to acquire a target image, wherein the target image includes a target object whose pose is to be determined; The segmentation unit is used to process the target image based on the image segmentation model to obtain the target instance mask of the target object; the image segmentation model is an image segmentation model trained based on a training sample set, and the training sample set includes sample data of the target object under various imaging conditions; A matching unit is configured to use the target instance mask as the target region, match a target template image that matches the target image from a pre-established template library, and perform pixel-level matching between the target template image and the target image; wherein, the template library includes the target template image; The pose determination unit is used to determine the 6-DOF 6D pose of the target object based on pixel-level matching results.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when run on a computer, causes the computer to execute the pose determination method according to any one of claims 1-9.

12. A computer program product, characterized in that, When it is run on a computer, it performs the pose determination method according to any one of claims 1-9.

Images