Object 6d pose estimation method, system, electronic device, and storage medium

Abstract

Embodiments of the present application provide an object 6D pose estimation method, system, electronic device and storage medium, belonging to the technical field of computer vision. The method analyzes the best matching template in the 3D template set and the target observation image in the first stage, and determines the rough correspondence relationship between the best matching template and the target observation image; in the second stage, the affine transformation matrix between the target observation image and the best matching template is determined according to the rough correspondence relationship, and the smooth correspondence relationship between the target observation image and the best matching template is determined based on the affine transformation matrix, so as to filter out the correspondence relationship noise and outliers; in the third stage, the coordinate offset is determined according to the regression offset block between the feature maps of the target observation image and the best matching template, and the smooth correspondence relationship is updated according to the coordinate offset to obtain the accurate correspondence relationship; and the object 6D pose is determined according to the accurate correspondence relationship. Through the step-by-step pixel-to-pixel correspondence learning, the accuracy of pose estimation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to an object 6D pose estimation method and system, an electronic device and a storage medium. BACKGROUND

[0002] The pose of an object is usually represented by six degrees of freedom (6DoF) parameters, including three-dimensional rotation and three-dimensional translation, to define the transformation from the standard object space to the camera space. Object pose estimation is of great concern in practical applications such as robot operation and augmented reality, and therefore has been extensively explored in research.

[0003] Early object pose estimation research mainly focused on using the same object CAD model for pose estimation in the training and testing stages, but lacked flexibility for objects not seen in training. Subsequent research solved the pose estimation problem for unseen objects in known categories by defining a category-normalized object coordinate space, but still had difficulty with completely new categories. With the progress of the basic model, more recent research has increasingly focused on handling completely new objects to achieve zero-shot 6D pose estimation, which poses a major challenge to the generalization ability of the model.

[0004] In the task of zero-shot pose estimation for new objects, related technologies use methods of RGB-D images to perform pose calculation through template matching combined with pose updating or point cloud registration, and have achieved remarkable performance. The success of these methods is largely due to the geometric support provided by the depth map, which not only provides key features for matching, but also enhances the localization ability of the object in three-dimensional space through geometric priors. However, the high cost of depth sensors often limits their popularity in practical applications, making methods that rely only on RGB images a more attractive choice. Nevertheless, RGB-based methods have not been fully researched and are often difficult to achieve competitive performance. For example, the 6D pose estimation methods of GigaPose and FoundPose, which establish the correspondence between the observed scene and the rendered template through simple feature matching, are often affected by correspondence noise and outliers, resulting in inaccurate pose prediction. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose an object 6D pose estimation method, system, electronic device and storage medium, which aims to improve the accuracy of 6D pose prediction of objects in images.

[0006] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application proposes an object 6D pose estimation method, comprising the following steps:

[0007] Obtaining a template set of a target object model and a target observation image;

[0008] determining a best matching template by feature matching the target observation image with each template in the template set, and determining a coarse correspondence between the best matching template and the target observation image;

[0009] determining an affine transformation matrix between the target observation image and the best matching template according to the coarse correspondence, and applying the affine transformation matrix to the best matching template for pixel alignment, to determine a smooth correspondence between the target observation image and the best matching template;

[0010] determining a coordinate offset according to a regression offset block between feature maps of the target observation image and the best matching template, and updating the smooth correspondence according to the coordinate offset to obtain an accurate correspondence between the target observation image and the best matching template;

[0011] determining an object 6D pose in the target observation image according to the accurate correspondence.

[0012] In some embodiments, the template set is obtained by the following steps:

[0013] obtaining a target object model;

[0014] rendering the target object model from multiple desired viewpoints to obtain a template set;

[0015] The target observation image is obtained by the following steps:

[0016] obtaining a scene image;

[0017] identifying a target object in the scene image using a zero-shot segmentation technique, and cropping a detection box of the target object from the scene image to obtain a target observation image.

[0018] In some embodiments, the step of determining a best matching template by feature matching the target observation image with each template in the template set comprises the following steps:

[0019] extracting features from the target observation image and each template in the template set respectively to obtain first features of the target observation image and second features of each template;

[0020] determining a matching score of each template with the target observation image by similarity between the first features and the second features, and determining a best matching template according to the matching score of each template.

[0021] In some embodiments, the step of determining an affine transformation matrix between the target observation image and the best matching template according to the coarse correspondence comprises the following steps:

[0022] encoding the coarse correspondence to obtain a correspondence map;

[0023] performing affine transformation relationship regression between the target observation image and the best matching template according to the correspondence map to obtain an affine transformation matrix;

[0024] wherein the affine transformation matrix is represented as follows

[0025]

[0026] wherein a represents an in-plane rotation angle, s represents and a relative ratio between (t u ,t v ) represents a two-dimensional translation amount in the image relative to the object centroid.

[0027] In some embodiments, the application of the affine transformation matrix to the best matching template for pixel alignment to determine the smooth correspondence between the target observation image and the best matching template comprises the following steps:

[0028] determining the corresponding position of the pixel position on the best matching template according to the affine transformation matrix and the pixel position of the target observation image;

[0029] obtaining the smooth correspondence between the target observation image and the best matching template according to the corresponding position of each pixel position of the target observation image on the best matching template.

[0030] In some embodiments, the determination of the coordinate offset amount according to the regression offset block between the feature maps of the target observation image and the best matching template comprises the following steps:

[0031] inputting the target observation image and the best matching template into a dense prediction transformer respectively to obtain a plurality of first layered feature maps of the target observation image and a plurality of second layered feature maps of the best matching template;

[0032] determining the regression offset block of the corresponding layer according to the first layered feature map and the second layered feature map of the corresponding layer;

[0033] scaling the two-dimensional position of the first layered feature map in the regression offset block, and aligning and adjusting the second layered feature map with the adjusted first layered feature map to obtain a third layered feature map;

[0034] performing correlation lookup on the first layered feature map and the second layered feature map in the regression offset block to obtain a correlation feature map;

[0035] determining the coordinate offset of each pixel according to the first layered feature map, the third layered feature map and the correlation feature map.

[0036] In some embodiments, the determining the object 6D pose in the target observation image according to the accurate correspondence comprises the following steps:

[0037] obtaining the confidence of the coordinate offset of the pixel;

[0038] when the confidence of the coordinate offset of the pixel is greater than an expected threshold, determining the corresponding position of the pixel in the target observation image in the best matching template according to the accurate correspondence, and determining the set of two-dimensional and three-dimensional associated point pairs according to the corresponding positions of the pixels;

[0039] determining the object 6D pose in the target observation image according to the set of associated point pairs.

[0040] To achieve the above object, another aspect of the embodiment of the present application proposes an object 6D pose estimation system, comprising:

[0041] a first module for obtaining a template set of a target object model and a target observation image;

[0042] a second module for performing feature matching between the target observation image and each template in the template set to determine a best matching template, and determining a rough correspondence between the best matching template and the target observation image;

[0043] a third module for determining an affine transformation matrix between the target observation image and the best matching template according to the rough correspondence, and applying the affine transformation matrix to the best matching template for pixel alignment to determine a smooth correspondence between the target observation image and the best matching template;

[0044] a fourth module for determining a coordinate offset according to a regression offset block between feature maps of the target observation image and the best matching template, and updating the smooth correspondence according to the coordinate offset to obtain an accurate correspondence between the target observation image and the best matching template;

[0045] a fifth module for determining the object 6D pose in the target observation image according to the accurate correspondence.

[0046] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the method of the above embodiment.

[0047] To achieve the above object, another aspect of the embodiment of the present application provides a storage medium, which is a computer readable storage medium for computer readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the method of the above embodiment.

[0048] The object 6D pose estimation method, system, electronic device and storage medium provided by the present application analyze the best matching template in the 3D template set with the target observation image in the first stage, and determine the rough correspondence relationship between the best matching template and the target observation image; in the second stage, the affine transformation matrix between the target observation image and the best matching template is determined according to the rough correspondence relationship, and the affine transformation matrix is applied to the best matching template for pixel alignment to determine the smooth correspondence relationship between the target observation image and the best matching template, so as to filter out the correspondence relationship noise and outliers; in the third stage, the coordinate offset is determined according to the regression offset block between the feature maps of the target observation image and the best matching template, and the smooth correspondence relationship is updated according to the coordinate offset to obtain the accurate and precise correspondence relationship; finally, the object 6D pose in the target observation image is determined according to the precise correspondence relationship. The present application improves the accuracy of pose estimation through step-by-step pixel-to-pixel correspondence learning. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of the object 6D pose estimation method provided by the embodiment of the present application;

[0050] Figure 2 is a schematic diagram of the PicoPose learning framework provided by the embodiment of the present application;

[0051] Figure 3 is a process schematic diagram of the three stages of pixel correspondence learning provided by the embodiment of the present application;

[0052] Figure 4 is a feature correspondence graph of the target observation image and the template under affine transformation provided by the embodiment of the present application;

[0053] Figure 5 is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed descriptions will be given below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0055] It should be noted that although the functional modules are divided in the system and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the system or the sequence in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0057] First, the terms related to the embodiments of the present application are explained as follows:

[0058] DINOv2, an advanced computer vision self-supervised model released by Meta. DINOv2 is a new method for training high-performance computer vision models, which uses self-supervised learning to learn from any image set and produces powerful visual representations. These representations perform well on downstream tasks such as classification, segmentation, image retrieval, and depth estimation, without the need for fine-tuning.

[0059] ViT-L, a version of the Vision Transformer (ViT) model, represents the Large model, i.e. the large version. It achieves global feature extraction of images by dividing them into a series of small patches and inputting these patches into a Transformer model for processing. This processing approach enables ViT-L to perform well on computer vision tasks such as image classification, object detection, and semantic segmentation.

[0060] Dense Prediction Transformer (DPT), an architecture that uses a vision Transformer instead of a convolutional network as the backbone for dense prediction tasks. Dense prediction tasks refer to predicting the label for each pixel or image block in an image. Such tasks usually require the model to capture detailed information in the image and accurately predict the class, depth, edge, etc. of each pixel or image block.

[0061] The framework of the embodiment method of the present application is described in whole, and the embodiment of the present application proposes a new step-by-step pixel-to-pixel correspondence learning framework, which is called PicoPose, to realize accurate new object pose estimation based on RGB images. PicoPose gradually optimizes the correspondence between the RGB observation image and the template in three stages, greatly improving the accuracy of the object pose calculated by these correspondences. Specifically, as shown in Figure 2 the RGB image (Scene image) containing a complex scene and an object CAD model (CAD model) are given. First, the object is rendered from multiple perspectives of the CAD model to obtain multiple templates (Templates). These templates are then combined with zero-shot segmentation (Zero-Shot Segmentation) technology to detect the target object in the RGB scene to obtain the target observation image. Then, PicoPose adopts a three-stage learning process of progressive pixel-to-pixel correspondence learning (Progressive Pixel-to-Pixel Correspondence Learning), that is, by template matching (Template Matching) to determine the best matching template of the detected object, and to learn the fine-grained pixel-to-pixel correspondence relationship. Since each pixel in the template corresponds to a three-dimensional surface point on the CAD model, a pair is established between the two-dimensional position in the observation image and the corresponding three-dimensional point in the template, and then the 6D pose of the image object (Object pose) is calculated by PnP / RANSAC algorithm. The specific description of the three stages of progressive pixel-to-pixel correspondence learning is as follows.

[0062] The first stage is feature matching to obtain coarse correspondence (Feature Matching for Coarse Correspondence). In this stage, PicoPose can use the visual transformer backbone network to extract the features of the RGB target observation image and the rendered template, identify the best matching template, and obtain the coarse correspondence between the template and the observation image.

[0063] The second stage is transformation estimation to optimize smooth correspondence (Transformation Estimation for Smooth Correspondence). In this stage, PicoPose represents the coarse correspondence as a correspondence graph and regresses a two-dimensional affine transformation (including in-plane rotation, scaling, and two-dimensional translation) to smooth the coarse correspondence to obtain smooth correspondence, so as to filter outliers in the first stage.

[0064] The third stage is local refinement for fine correspondence. In this stage, PicoPose applies an affine transformation to the feature map of the best matching template and learns the correspondence offset within the local region through multiple offset regression modules to gradually achieve fine pixel-to-pixel correspondence.

[0065] Based on the above learning framework, the embodiment of the present application provides an object 6D pose estimation method, system, electronic device and storage medium, aiming to improve the accuracy of object 6D pose prediction in images.

[0066] The object 6D pose estimation method, system, electronic device and storage medium provided by the embodiment of the present application are specifically explained by the following embodiments. First, the object 6D pose estimation method in the embodiment of the present application is described.

[0067] The object 6D pose estimation method provided by the embodiment of the present application relates to the technical field of computer vision. The object 6D pose estimation method provided by the embodiment of the present application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the object 6D pose estimation method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0069] Figure 1 is an optional flowchart of the object 6D pose estimation method provided by the embodiments of the present application, Figure 1 The method in the above embodiment can include, but is not limited to, steps S101-S105.

[0070] Step S101, obtaining a template set of a target object model and a target observation image;

[0071] Step S102, performing feature matching between the target observation image and each template in the template set to determine a best matching template, and determining a coarse correspondence relationship between the best matching template and the target observation image;

[0072] Step S103, determining an affine transformation matrix between the target observation image and the best matching template according to the coarse correspondence relationship, and applying the affine transformation matrix to the best matching template for pixel alignment to determine a smooth correspondence relationship between the target observation image and the best matching template;

[0073] Step S104, determining a coordinate offset according to a regression offset block between the target observation image and a feature map of the best matching template, and updating the smooth correspondence relationship according to the coordinate offset to obtain an accurate correspondence relationship between the target observation image and the best matching template;

[0074] Step S105, determining an object 6D pose in the target observation image according to the accurate correspondence relationship.

[0075] In step S101 of some embodiments, the target object model is a three-dimensional model of a target object, which can be a three-dimensional CAD model, wherein the object pose of the target object model is represented by six degrees of freedom parameters, including three-dimensional rotation and three-dimensional translation. The target observation image refers to an RGB image containing the target object.

[0076] In some embodiments, the template set in step S101 is obtained by the following steps:

[0077] Step S201, obtaining a target object model;

[0078] Step S202, rendering the target object model from multiple desired viewing angles to obtain a template set;

[0079] In some embodiments, the target observation image in step S101 is obtained by the following steps:

[0080] Step S301, obtaining a scene image;

[0081] Step S301, identifying the target object in the scene image using a zero-shot segmentation technique, and cropping a detection box of the target object from the scene image to obtain a target observation image.

[0082] Exemplarily, for a given clustered scene RGB image and a CAD model of a target object, first, render object templates from each view of the CAD model to obtain templates of the target object from each view, and each template forms a template set. Then, use a zero-shot segmentation technique to identify and crop the region containing the target object from the scene image (i.e., the RGB image) to obtain a target observation image. Further, resize the target observation image and the target object templates to a fixed size H x W, and represent the target observation image as The template set is represented as where N is the number of templates. In the subsequent process, PicoPose can use the target observation image and the template set as inputs to search for the best matching template (represented as ), and gradually learn the pixel-to-pixel correspondence between and in three stages. Each foreground pixel on corresponds to a 3D surface point on the CAD model, so that 2D point pairs can be established on , and their corresponding 3D point pairs can be established on , and then the above point pairs can be used to calculate the 6D object pose.

[0083] In step S102 of some embodiments, in the first stage of PicoPose, the visual transformer backbone network can be used to extract the features of the target observation image and each rendered template, and the best matching template can be identified through feature matching, and then the rough correspondence between the template and the target observation image about the pixel points can be obtained.

[0084] In some embodiments, the step of determining the best matching template by feature matching the target observation image with each template in the template set in step S102 can include, but is not limited to, steps S401 to S402:

[0085] Step S401, respectively, extract features from the target observation image and each template in the template set to obtain first features of the target observation image and second features of each template;

[0086] Step S402, determine the matching scores of each template and the target observation image by the similarity of the first features and the second features, and determine the best matching template according to the matching scores of each template.

[0087] Exemplarily, please refer to the first stage (Stage 1) part of Figure 3 , for the resized target object image (RGB observation) and the object templates (Object templates) from each view of the CAD model The ViT-L backbone network pre-trained by DINOv2 is used to extract the features of the target observation image and each object template respectively, and the corresponding feature representation is obtained as follows: and Where D is the feature dimension and M is the number of patches. It is understandable that other network models for image feature extraction can be used for feature extraction, and the embodiments of the present application do not impose specific restrictions. Then, feature matching is performed based on the extracted features to identify the best-matched template, and at the same time, a rough correspondence between the target observation image and the best-matched template is determined. Feature matching can be achieved by calculating the similarity between all features of the target observation image and the object template, and establishing a rough correspondence through feature similarity. Specifically, by performing feature matching on each template Observation image of target The similarity between the features is scored to retrieve the best matching template For each template By comparing the foreground pixels of the observed image (determined by the above zero-sample segmentation step) with The template matching score c is obtained by taking the average of the maximum feature cosine similarity of the pixels in i , from the template collection Among them, select the template with the highest template matching score As the best matching template, a rough pixel-to-pixel correspondence of obvious features between the best matching template and the target observation image is established. In addition, and The feature similarity between Each pixel in The most similar pixels in the image are used to provide a rough correspondence.

[0088] In step S103 of some embodiments, in the first stage of PicoPose, feature matching is used to obtain and There is a rough and sparse correspondence between , however, these correspondences are as follows Figure 3 The results show that the distribution of the pixels observed by the target image is usually messy, containing noise and many outliers. Based on this, in the second stage of PicoPose, the two-dimensional affine transformation (including in-plane rotation, scaling and two-dimensional translation) between the target observation image and the best matching template is regressed according to the rough correspondence to obtain the affine transformation matrix. The affine transformation matrix is ​​then applied to the best matching template for pixel alignment to determine the smooth correspondence between the target observation image and the best matching template, thereby smoothing the rough correspondence and filtering outliers.

[0089] In some embodiments, in step S103, the step of determining the affine transformation matrix between the target observation image and the best matching template according to the coarse correspondence can include, but is not limited to, steps S501-S502:

[0090] In step S501, the coarse correspondence is encoded to obtain a correspondence map;

[0091] In step S502, the affine transformation relationship between the target observation image and the best matching template is regressed according to the correspondence map to obtain the affine transformation matrix.

[0092] Exemplarily, please refer to Figure 3 the second stage (Stage 2) part, considering that and The coarse correspondence between the two images can effectively capture the changes in the rotation angle, scale and translation, so as to learn the basic mode of the affine transformation relationship (the affine transformation relationship adopts an affine transformation matrix ), and reduce the related learning difficulty. Therefore, instead of directly connecting the related features and to regress , the coarse correspondence obtained in the first stage is used to regress, specifically, the coarse correspondence in the first stage is encoded into a correspondence map to conform to the input format of the subsequent network, and then the correspondence map is input into the network constructed based on the multi-layer perception mechanism (MLP) to regress the affine transformation (Affine Transformation) of the in-plane rotation (In-plane rotation) angle α, the scale (Scale) s and the translation (Translation) (t u ,t v ) to obtain the affine transformation matrix 2D affine transformation matrix can be parameterized by four degrees of freedom (DoF), as follows:

[0093]

[0094] Wherein, α represents the in-plane rotation angle, s represents and the relative proportion between the two images, (t u ,t v ) represents the 2D translation of the object center in the two images. By applying to transform the template , it is helpful to Pixel alignment is performed to achieve smooth correspondences. Various affine transformations (including in-plane rotation, scaling, and 2D translation) visualize the correspondence map between the features of the RGB observation (marked by star) and the template features as shown in Figure 4 . Furthermore, by combining the with the viewpoint rotation of , one can give the 6D object pose.

[0095] In some embodiments, the step of determining the smooth correspondences between the target observation image and the best matching template in step S103, by applying the affine transformation matrix to the best matching template for pixel alignment, can include but is not limited to steps S601-S602:

[0096] Step S601, determining the corresponding position of a pixel position on the best matching template according to the affine transformation matrix and the pixel position of the target observation image;

[0097] Step S602, obtaining the smooth correspondences between the target observation image and the best matching template according to the corresponding positions of each pixel position of the target observation image on the best matching template.

[0098] Exemplarily, continuing to refer to the second stage of Figure 3 , for any pixel position (u, v) on , one can use to obtain the corresponding position (u', v') on , as shown below:

[0099]

[0100] where the deviation between (u, v) and (u', v') can be interpreted as the common "optical flow". For all pixels in , their corresponding positions in are calculated by the above formula and denoted as to represent the smooth correspondences. Subsequently, by using as the index to collect the features from , the affine transformation applied to the feature map of , including rotation, scaling, and translation, is implemented to generate the transformed feature map aligned with .

[0101] In step S104 of some embodiments, in the third stage of PicoPose, the affine transformation matrix is applied to the multiple layered feature maps of the best matching template to obtain the offset regression block, and the corresponding offset in the local region is learned by the multiple offset regression modules to update the smooth correspondences to Fine-grained correspondence adjustment in local areas is implemented to gradually achieve a fine pixel-to-pixel correspondence.

[0102] In some embodiments, in step S104, the step of determining the coordinate offset according to the regression offset block between the target observation image and the feature map of the best matching template can include, but is not limited to, steps S701-S705:

[0103] In step S701, the target observation image and the best matching template are respectively input into a dense prediction transformer to obtain a plurality of first layered feature maps of the target observation image and a plurality of second layered feature maps of the best matching template.

[0104] In step S702, a regression offset block of a corresponding layer is determined according to the first layered feature map and the second layered feature map of the corresponding layer.

[0105] In step S703, the two-dimensional position of the first layered feature map in the regression offset block is scaled and adjusted, and the second layered feature map is aligned and adjusted with the adjusted first layered feature map to obtain a third layered feature map.

[0106] In step S704, the first layered feature map and the second layered feature map in the regression offset block are searched for correlation to obtain a correlation feature map.

[0107] In step S705, the coordinate offset of each pixel is determined according to the first layered feature map, the third layered feature map, and the correlation feature map of different layers.

[0108] Exemplarily, please refer to the third stage (Stage 3) part of Figure 3 First, a dense prediction transformer (DPT) is applied to the best matching template and the target observation image to respectively generate L first layered feature maps of the best matching template and L second layered feature maps of the target observation image where H l ×W l represents the spatial size, and D l is the number of channels of the l th th feature map. Each first layered feature map and the corresponding second layered feature map form an offset regression block. L offset regression blocks are used to iteratively update

[0109] For the l th th offset regression block, the size of the current is adjusted to H l ×W l× 2, and scaled by dividing each 2D position in the first hierarchical feature map by the image size to ensure spatial consistency. Denote the resized and scaled version as which is used as an index to collect features from the second hierarchical feature map , resulting in a transformed third hierarchical feature map with the same size as Feature alignment is achieved. Then, the correlation between the first and second hierarchical features is analyzed by a Correlation Lookup module, resulting in a correlation feature map to explicitly provide the correlation and to make it easier to learn the offset. Then and are concatenated to form the input of two stacked convolutional sequences that are used to regress the coordinate offsets for each feature map. A certainty map about each predicted coordinate offset can also be output. is interpolated to a size of HxWx2 by scaling the two-dimensional coordinates in it by the image size and adding to to update. The certainty map represents the confidence of the regressed offset and is upsampled to produce This embodiment uses L offset regression blocks to update the refined with its certainty map

[0110] In step S105 of some embodiments, since each pixel in the template corresponds to a three-dimensional surface point on the CAD model, after establishing the exact correspondence between the two-dimensional position in the target observation image and the corresponding three-dimensional point in the template, the 6D pose of the object can be calculated by PnP / RANSAC algorithm or the like.

[0111] In some embodiments, step S105 can include but is not limited to steps S801-S803:

[0112] Step S801, obtain the confidence of the coordinate offset of the pixel;

[0113] Step S802, when the confidence of the coordinate offset of the pixel is greater than the expected threshold, determine the corresponding position of the pixel in the best matching template in the target observation image according to the exact correspondence, and determine the set of two-dimensional and three-dimensional corresponding point pairs according to the corresponding positions of the pixels.

[0114] Step S803, determine the object 6D pose in the target observation image according to the set of associated point pairs.

[0115] In this embodiment, for each foreground pixel in , if its corresponding certainty (i.e., confidence) exceeds 0.5, the precise position can be used to look up the corresponding pixel in , which is associated with a surface point of the 3D object. Therefore, all pixel-to-pixel correspondences generate relevant 2D-3D associated point pairs, and the final object pose is calculated based on the set of associated point pairs by PnP / RANSAC algorithm.

[0116] According to some embodiments of the present application, the beneficial effects of the object 6D pose estimation method of the embodiments of the present application are described below in combination with experimental results.

[0117] (1) Experimental setup

[0118] The model is trained on the ShapeNet-Objects and Google-Scanned-Objects synthetic datasets provided in the MegaPose paper, using a total of 2,000,000 training images. The index evaluation is carried out on the seven core datasets of the BOP benchmark, including LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB and YCB-V. The standard BOP evaluation protocol is used to report the average recall (AR) with respect to three error functions, namely the visible surface difference (VSD), the maximum symmetric perception surface distance (MSSD) and the maximum symmetric perception projection distance (MSPD).

[0119] (2) Comparison with existing methods

[0120] The PicoPose learning framework method proposed in this embodiment is evaluated on the seven core datasets of the BOP benchmark, together with existing methods. The robustness of PicoPose is enhanced by using the top five templates in the second and third stages to learn fine correspondences and select the most matching pose with these correspondences. The quantitative results are shown in Table 1, where PicoPose is significantly better than other methods, highlighting its excellent zero-shot capability for novel object pose estimation through progressive correspondence learning. For example, the PicoPose single model using the top five templates is 19.6% and 9.2% higher than the GigaPose and FoundPose single models in AR, respectively. In Table 1, the results under the iterative refinement proposed in MegaPose are also reported, where PicoPose is always superior to other methods.

[0121] Table 1 Pose estimation results of different methods on the seven core datasets of the BOP benchmark

[0122]

[0123] (3) Ablation Experiments

[0124] Effectiveness of Progressive Correspondence Learning. PicoPose is key in its design of progressive pixel-to-pixel correspondence learning. To evaluate this, first check the impact of improved correspondence quality by testing the results obtained from the respective correspondences of the three stages using PnP / RANSAC. As shown in Table 2, the pose accuracy improves with finer correspondences, supporting the core claim of this paper. The results of the second stage improve the accuracy by 5.7% on average by smoothing the correspondences of the first stage, while the third stage further improves the accuracy by 14.7% on the three datasets by locally refining the correspondences of the second stage.

[0125] Table 2 Results of the three stages on three different datasets

[0126]

[0127] Effect of Template Number. Following the setup in the GigaPose paper, N = 162 templates are used per object in the evaluation experiments. In Table 3, additional quantitative results are shown for GigaPose and the proposed PicoPose with different numbers of templates. The results show that the effect improves with increasing numbers of templates, as both methods rely on template matching to select the best matching template for the target object. However, the improvement slows down with increasing numbers of templates. Notably, PicoPose is more effective than GigaPose when using fewer templates, which further highlights the advantage of PicoPose.

[0128] Table 3 Quantitative comparison of different numbers of templates on three different datasets

[0129]

[0130] Effectiveness of Third Stage Stacked Offset Regression Blocks. In the third stage, L offset regression blocks are used, specifically, L = 3 offset regression blocks are used in the experiments, and the feature space size in DPT is set to 16x16, 32x32, and 64x64 to refine the correspondences within local regions. Table 4 reports the results of different offset regression blocks. As shown in Table 4, the performance gradually improves with more blocks, indicating that finer correspondences can be achieved by stepwise local refinement.

[0131] Table 4 Quantitative results of different offset regression blocks (denoted as “OR Block”) in the third stage

[0132]

[0133] The embodiment of the present application also provides an object 6D pose estimation system, comprising:

[0134] A first module is configured to acquire a template set of a target object model and a target observation image;

[0135] A second module is configured to perform feature matching on the target observation image and each template in the template set to determine a best matching template, and determine a coarse correspondence relationship between the best matching template and the target observation image;

[0136] A third module is configured to determine an affine transformation matrix between the target observation image and the best matching template according to the coarse correspondence relationship, and apply the affine transformation matrix to the best matching template to perform pixel alignment, and determine a smooth correspondence relationship between the target observation image and the best matching template;

[0137] A fourth module is configured to determine a coordinate offset according to a regression offset block between feature maps of the target observation image and the best matching template, and update the smooth correspondence relationship according to the coordinate offset to obtain an accurate correspondence relationship between the target observation image and the best matching template;

[0138] A fifth module is configured to determine an object 6D pose in the target observation image according to the accurate correspondence relationship.

[0139] It can be understood that the content in the above object 6D pose estimation method embodiment is applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above object 6D pose estimation method embodiment, and achieves the same beneficial effects as the above object 6D pose estimation method embodiment.

[0140] The embodiment of the present application also provides an electronic device, which comprises a memory, a processor, a program stored in the memory and capable of running on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above object 6D pose estimation method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0141] Please refer to Figure 5 , Figure 5 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0142] The processor 501 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0143] The memory 502 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 502 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 502 and are called and executed by the processor 501 to implement the object 6D pose estimation method of the embodiments of the present application.

[0144] The input / output interface 503 is configured to implement information input and output.

[0145] The communication interface 504 is configured to implement the communication interaction between the device and other devices. The communication can be implemented by a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0146] The bus 505 is configured to transmit information between the components (for example, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504) of the device.

[0147] The processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are connected to each other through the bus 505 to realize the communication connection between the components in the device.

[0148] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and is configured to store computer readable information. The storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the object 6D pose estimation method.

[0149] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely from the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0151] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0152] The system embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0153] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0154] The terms "first", "second", "third", "fourth" and the like used in the description of the present application and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0155] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0156] In several embodiments provided in the application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection between systems or units through some interfaces, and can be electrical, mechanical or other forms.

[0157] The units described above as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0158] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0159] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0160] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for estimating 6D pose of an object, characterized in that: The following steps are involved: Acquire a template set of a target object model and a target observation image; the template set is a 3D template set; Performing feature matching between the target observation image and each template in the template set to determine the best matching template, and determining a rough correspondence between the best matching template and the target observation image; Determining an affine transformation matrix between the target observation image and the best matching template based on the rough correspondence, and applying the affine transformation matrix to the best matching template for pixel alignment to determine a smooth correspondence between the target observation image and the best matching template; Determining a coordinate offset according to a regression offset block between the target observation image and the feature map of the best matching template, and updating the smooth correspondence according to the coordinate offset to obtain an accurate correspondence between the target observation image and the best matching template; determining a 6D pose of the object in the target observation image according to the precise correspondence; Applying the affine transformation matrix to the best matching template to perform pixel alignment and determining a smooth correspondence between the target observation image and the best matching template comprises the following steps: Determining a corresponding position of the pixel position on the best matching template according to the affine transformation matrix and the pixel position of the target observation image; Obtaining a smooth correspondence between the target observation image and the best matching template according to corresponding positions of each pixel position of the target observation image on the best matching template; Determining the coordinate offset based on the regression offset block between the target observation image and the feature map of the best matching template includes the following steps: Inputting the target observation image and the best matching template into a dense prediction transformer respectively to obtain a plurality of first hierarchical feature maps of the target observation image and a plurality of second hierarchical feature maps of the best matching template; Determine a regression offset block of the corresponding layer based on the first hierarchical feature map and the second hierarchical feature map of the corresponding layer; Scaling and adjusting the two-dimensional position of the first hierarchical feature map in the regression offset block, and aligning the second hierarchical feature map with the adjusted first hierarchical feature map to obtain a third hierarchical feature map; Performing a correlation search on the first hierarchical feature map and the second hierarchical feature map in the regression offset block to obtain a correlation feature map; Determine the coordinate offset of each pixel according to the first hierarchical feature map, the third hierarchical feature map and the correlation feature map of different layers; Determining the 6D posture of the object in the target observation image according to the precise correspondence includes the following steps: Get the confidence of the pixel coordinate offset; When the confidence level of the pixel coordinate offset is greater than a desired threshold, the corresponding position of the pixel in the target observation image in the best matching template is determined according to the precise correspondence relationship, and two-dimensional and three-dimensional associated point pair sets are determined according to the corresponding positions of the pixels; The 6D pose of the object in the target observation image is determined according to the set of associated point pairs.

2. The object 6D pose estimation method according to claim 1, characterized in that The template set is obtained by the following steps: Get the target object model; Rendering the target object model using multiple desired perspectives to obtain a template set; The target observation image is obtained by the following steps: Get scene image; A zero-sample segmentation technique is used to identify a target object in the scene image, and a detection frame of the target object is cropped from the scene image to obtain a target observation image.

3. The object 6D pose estimation method according to claim 1, characterized in that The step of performing feature matching between the target observation image and each template in the template set to determine the best matching template comprises the following steps: Performing feature extraction on the target observation image and each template in the template set respectively to obtain a first feature of the target observation image and a second feature of each template; The matching scores between each template and the target observation image are determined based on the similarity between the first feature and the second feature, and the best matching template is determined based on the matching scores of each template.

4. The object 6D pose estimation method according to claim 1, characterized in that Determining the affine transformation matrix between the target observation image and the best matching template according to the rough correspondence relationship includes the following steps: Encoding the rough correspondence to obtain a correspondence graph; Regressing the affine transformation relationship between the target observation image and the best matching template according to the correspondence relationship graph to obtain an affine transformation matrix; Among them, the affine transformation matrix is ​​expressed as follows : ； in, represents the in-plane rotation angle, express and The relative proportion between Represents the horizontal and vertical two-dimensional translation relative to the center of mass of the object in the image, represents the target observation image, Indicates the best matching template.

5. A 6D pose estimation system for an object, characterized in that: include: The first module is used to obtain a template set of a target object model and a target observation image; The template set is a 3D template set; A second module is configured to perform feature matching between the target observation image and each template in the template set to determine a best matching template, and to determine a rough correspondence between the best matching template and the target observation image; A third module is configured to determine an affine transformation matrix between the target observation image and the best matching template based on the rough correspondence relationship, and apply the affine transformation matrix to the best matching template for pixel alignment to determine a smooth correspondence relationship between the target observation image and the best matching template; a fourth module, configured to determine a coordinate offset based on a regression offset block between the target observation image and the feature map of the best matching template, and update the smooth correspondence based on the coordinate offset to obtain an accurate correspondence between the target observation image and the best matching template; A fifth module is configured to determine a 6D pose of the object in the target observation image according to the precise correspondence; The third module is specifically used for: Determining a corresponding position of the pixel position on the best matching template according to the affine transformation matrix and the pixel position of the target observation image; Obtaining a smooth correspondence between the target observation image and the best matching template according to corresponding positions of each pixel position of the target observation image on the best matching template; The fourth module is specifically used for: Inputting the target observation image and the best matching template into a dense prediction transformer respectively to obtain a plurality of first hierarchical feature maps of the target observation image and a plurality of second hierarchical feature maps of the best matching template; Determine a regression offset block of the corresponding layer based on the first hierarchical feature map and the second hierarchical feature map of the corresponding layer; Scaling and adjusting the two-dimensional position of the first hierarchical feature map in the regression offset block, and aligning the second hierarchical feature map with the adjusted first hierarchical feature map to obtain a third hierarchical feature map; Performing a correlation search on the first hierarchical feature map and the second hierarchical feature map in the regression offset block to obtain a correlation feature map; Determine the coordinate offset of each pixel according to the first hierarchical feature map, the third hierarchical feature map and the correlation feature map of different layers; The fifth module is specifically used for: Get the confidence of the pixel coordinate offset; When the confidence level of the pixel coordinate offset is greater than a desired threshold, the corresponding position of the pixel in the target observation image in the best matching template is determined according to the precise correspondence relationship, and two-dimensional and three-dimensional associated point pair sets are determined according to the corresponding positions of the pixels; The 6D pose of the object in the target observation image is determined according to the set of associated point pairs.

6. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 4 are implemented.

7. A storage medium, wherein the storage medium is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 4.

Images