Class level object pose estimation method based on Laplace mixture model

Abstract

The invention relates to a category-level object pose estimation method based on a Laplace mixture model. The method comprises the following steps: predicting an NOCS coordinate graph of a to-be-estimated object according to an image of the to-be-estimated object; predicting first Laplacian distribution of the NOCS coordinate graph based on a preset universal 3D information flow, predicting second Laplacian distribution of the NOCS coordinate graph based on a preset feature flow, and obtaining target Laplacian distribution according to the first Laplacian distribution and the second Laplacian distribution; and based on a preset convolutional network, performing pose estimation on the to-be-estimated object according to the target Laplacian distribution to obtain a pose estimation result, and obtaining a size-independent pose of the to-be-estimated object according to the pose estimation result. Therefore, the influence of shape uncertainty and scale uncertainty caused by the shape difference in the category on pose estimation is overcome, high-precision and high-accuracy prediction of the pose of the object at the category level can be realized, and the influence of the scale uncertainty is avoided.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method for estimating the pose of a category-level object based on a Laplace mixture model. Background Art

[0002] Category-level object pose estimation refers to predicting the 9-degree-of-freedom (9DoF) pose of unseen objects in a given category set, including 3-degree-of-freedom 3D rotation, 3-degree-of-freedom 3D translation, and 3-degree-of-freedom 3D size. Category-level object pose estimation has wide applications in robotics, augmented reality (AR), virtual reality (VR), and 3D understanding.

[0003] Although RGB-D (RGB plus depth) based methods have achieved good results in category-level object pose estimation, most methods usually rely heavily on depth sensors, which limits their applicability in general scenarios. Therefore, a category-level object pose estimation method based on RGB (Red, Green, Blue) is proposed as an alternative suitable for deployment on embedded devices such as AR glasses and mobile phones.

[0004] However, the lack of depth information brings two significant challenges to RGB-based category-level object pose estimation: first, the lack of depth information makes the prediction of object shape more complicated and exacerbates the difficulty of dealing with intra-class shape variations; second, relying solely on RGB input introduces inherent scale ambiguity, making the estimation of translation and size an ill-posed problem.

[0005] In order to overcome the above challenges, two solutions have been proposed in related technologies. One is to establish a 3D-3D correspondence by estimating the metric depth and the Normalized Object Coordinate Space (NOCS) coordinates, and solve the pose using the Umeyama algorithm. The other is to establish a 2D-3D correspondence by estimating the NOCS coordinate map and the metric size of the object, and then use the PnP (Perspective-n-Point) algorithm to solve the pose.

[0006] However, although the related technologies have solved the RGB-based pose estimation problem to a certain extent, they still have the following limitations: First, the lack of depth information makes it difficult to accurately measure the shape of an object. The uncertainty of the shape is particularly obvious in certain areas of the image, which makes it more complicated to establish accurate correspondences. The methods of the related technologies usually treat the predicted correspondences of each pixel as equivalent and rely on RANSAC to filter outliers, which reduces the prediction speed and affects the robustness. Second, some methods of the related technologies do not consider scale ambiguity, while others use the same features to predict NOCS maps and metric dimensions. However, inferring metric dimensions from a single RGB image is essentially an ill-posed problem, which leads to unstable training of other components and poor results, which needs to be solved urgently. Summary of the invention

[0007] The present application provides a category-level object pose estimation method based on the Laplace mixture model to overcome the influence of shape uncertainty and scale uncertainty on pose estimation caused by shape differences within categories, and can achieve high-precision and high-accuracy prediction of category-level object poses without being affected by scale uncertainty.

[0008] The first aspect of the present application provides a method for class-level object pose estimation based on a Laplace mixture model, comprising the following steps:

[0009] Acquire an image of the object to be estimated, and predict a NOCS coordinate map of the object to be estimated based on the image of the object to be estimated;

[0010] Predicting a first Laplace distribution of the NOCS coordinate map based on a preset universal 3D information flow, predicting a second Laplace distribution of the NOCS coordinate map based on a preset feature flow, and obtaining a target Laplace distribution according to the first Laplace distribution and the second Laplace distribution;

[0011] Based on a preset convolutional network, the pose of the object to be estimated is estimated according to the target Laplace distribution to obtain a pose estimation result, and the size-independent pose of the object to be estimated is obtained according to the pose estimation result.

[0012] According to an embodiment of the present application, predicting a first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow, and predicting a second Laplace distribution of the NOCS coordinate map based on a preset feature flow, includes:

[0013] Based on the preset universal 3D information flow, predicting a first Laplace distribution of the NOCS coordinate map using a first preset loss function;

[0014] Based on the preset feature flow, predicting a second Laplace distribution of the NOCS coordinate graph using a second preset loss function;

[0015] Wherein, the first preset loss function is:

[0016]

[0017] The second preset loss function is:

[0018]

[0019] Among them, L 3D-dino is the first preset loss function, is the variance of the first Laplace distribution, M vis is the mask of the visible part of the object to be estimated, is the real NOCS coordinate diagram, μ dino is the mean of the first Laplace distribution, L 3D-conv is the second preset loss function, is the variance of the second Laplace distribution, μ conv is the mean of the second Laplace distribution, and λ1 and λ2 are hyperparameters.

[0020] According to an embodiment of the present application, the preset convolutional network is used to estimate the pose of the object to be estimated according to the target Laplace distribution to obtain a pose estimation result, including:

[0021] Obtaining a 2D pixel coordinate map of the image of the object to be estimated;

[0022] Inputting the target Laplace distribution and the 2D pixel coordinate map into the preset convolutional network to obtain the pose estimation result;

[0023] Among them, the pose estimation result is:

[0024]

[0025] Among them, R out is the predicted scale-independent rotation parameter, t out is the predicted scale-independent translation parameter, Φ() is the preset convolutional network, μ dino is the mean of the first Laplace distribution, is the variance of the first Laplace distribution, μ conv is the mean of the second Laplace distribution, is the variance of the second Laplace distribution, C 2D is the 2D pixel coordinate map of the image of the object to be estimated.

[0026] According to an embodiment of the present application, obtaining the size-independent pose of the object to be estimated according to the pose estimation result includes:

[0027] Acquire the 3D size of the object to be estimated and the 3D translation of the object to be estimated, and calculate the original diagonal length of the object tight bounding box of the object to be estimated based on the 3D size of the object to be estimated;

[0028] Normalizing the size of the object to be estimated according to the 3D size of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated to obtain the normalized 3D size of the object to be estimated;

[0029] The translation of the object to be estimated is normalized according to the 3D translation vector of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated, so as to obtain the normalized translation of the object to be estimated, and a predicted translation amount is obtained according to the normalized 3D translation of the object to be estimated, wherein the predicted translation amount is:

[0030] t out =[δ x ,δ y ,δ z ];

[0031] δ x =(o x -c x ) / w box ;

[0032] δ y =(o y -c y ) / h box ;

[0033]

[0034] Among them, t out is the predicted translation parameter, δ x , δ y are the relative offsets of the center of the 2D detection frame in the x and y axes of the image plane, δ z It is the relative value of the object's translation on the z-axis relative to the size of the detection frame. x ,o y ) is the 2D position of the center of the projected object, (c x ,c y ) is the center coordinate of the 2D bounding box, w box is the width of the 2D bounding box, h box is the height of the 2D bounding box, is the normalized distance from the object to be estimated to the camera, s boxis the size of the 2D bounding box, s in is the size of the image of the object to be estimated after scaling.

[0035] According to an embodiment of the present application, obtaining the size-independent posture of the object according to the posture estimation result further includes:

[0036] Predicting a first rotation vector of the object to be estimated along a first direction axis, and predicting a second rotation vector of the object along a second direction axis, wherein the first direction axis is perpendicular to the second direction axis;

[0037] Determining whether the object to be estimated has rotational symmetry;

[0038] If the object to be estimated has rotational symmetry, the second rotation vector is supervised when training the model of the object to be estimated.

[0039] According to the method for estimating the pose of a class-level object based on a Laplace mixture model in an embodiment of the present application, a first Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on a preset general 3D information flow, and a second Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on a preset feature flow, and a target Laplace distribution is obtained according to the first Laplace distribution and the second Laplace distribution; pose estimation is performed on the object to be estimated based on a preset convolutional network to obtain a pose estimation result, and a size-independent pose of the object to be estimated is obtained according to the pose estimation result. Thus, the influence of shape uncertainty and scale uncertainty on pose estimation caused by shape differences within a class is overcome, and high-precision and high-accuracy prediction of the pose of class-level objects can be achieved, and it is not affected by scale uncertainty.

[0040] The second aspect of the present application provides a device for estimating a class-level object pose based on a Laplace mixture model, including:

[0041] An acquisition module, used for acquiring an image of the object to be estimated, and predicting a NOCS coordinate map of the object to be estimated according to the image of the object to be estimated;

[0042] A prediction module, configured to predict a first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow, predict a second Laplace distribution of the NOCS coordinate map based on a preset feature flow, and obtain a target Laplace distribution according to the first Laplace distribution and the second Laplace distribution;

[0043] The pose estimation module is used to perform pose estimation on the object to be estimated based on a preset convolutional network and according to the target Laplace distribution to obtain a pose estimation result, and obtain a size-independent pose of the object to be estimated based on the pose estimation result.

[0044] According to one embodiment of the present application, the prediction module is used to:

[0045] Based on the preset universal 3D information flow, predicting a first Laplace distribution of the NOCS coordinate map using a first preset loss function;

[0046] Based on the preset feature flow, predicting a second Laplace distribution of the NOCS coordinate graph using a second preset loss function;

[0047] Wherein, the first preset loss function is:

[0048]

[0049] The second preset loss function is:

[0050]

[0051] Among them, L 3D-dino is the first preset loss function, is the variance of the first Laplace distribution, M vis is the mask of the visible part of the object to be estimated, is the real NOCS coordinate diagram, μ dino is the mean of the first Laplace distribution, L 3D-conv is the second preset loss function, is the variance of the second Laplace distribution, μ conv is the mean of the second Laplace distribution, and λ1 and λ2 are hyperparameters.

[0052] According to one embodiment of the present application, the posture estimation module is used to:

[0053] Obtaining a 2D pixel coordinate map of the image of the object to be estimated;

[0054] Inputting the target Laplace distribution and the 2D pixel coordinate map into the preset convolutional network to obtain the pose estimation result;

[0055] Among them, the pose estimation result is:

[0056]

[0057] Among them, R out is the predicted scale-independent rotation parameter, t out is the predicted scale-independent translation parameter, Φ() is the preset convolutional network, μ dino is the mean of the first Laplace distribution, is the variance of the first Laplace distribution, μ convis the mean of the second Laplace distribution, is the variance of the second Laplace distribution, C 2D is the 2D pixel coordinate map of the image of the object to be estimated.

[0058] According to one embodiment of the present application, the posture estimation module is used to:

[0059] Acquire the 3D size of the object to be estimated and the 3D translation of the object to be estimated, and calculate the original diagonal length of the object tight bounding box of the object to be estimated based on the 3D size of the object to be estimated;

[0060] Normalizing the size of the object to be estimated according to the 3D size of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated to obtain the normalized 3D size of the object to be estimated;

[0061] The translation of the object to be estimated is normalized according to the 3D translation vector of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated, so as to obtain the normalized translation of the object to be estimated, and a predicted translation amount is obtained according to the normalized 3D translation of the object to be estimated, wherein the predicted translation amount is:

[0062] t out =[δ x ,δ y ,δ z ];

[0063] δ x =(o x -c x ) / w box ;

[0064] δ y =(o y -c y ) / h box ;

[0065]

[0066] Among them, t out is the predicted translation parameter, δ x , δ y are the relative offsets of the center of the 2D detection frame in the x and y axes of the image plane, δ z is the relative value of the object's translation on the z-axis relative to the size of the detection frame, (o x ,o y ) is the 2D position of the center of the projected object, (c x ,c y ) is the center coordinate of the 2D bounding box, w boxis the width of the 2D bounding box, h box is the height of the 2D bounding box, is the normalized distance from the object to be estimated to the camera, s box is the size of the 2D bounding box, s in is the size of the image of the object to be estimated after scaling.

[0067] According to one embodiment of the present application, the posture estimation module is further used to:

[0068] Predicting a first rotation vector of the object to be estimated along a first direction axis, and predicting a second rotation vector of the object along a second direction axis, wherein the first direction axis is perpendicular to the second direction axis;

[0069] Determining whether the object to be estimated has rotational symmetry;

[0070] If the object to be estimated has rotational symmetry, the second rotation vector is supervised when training the model of the object to be estimated.

[0071] According to the Laplace mixture model-based category-level object pose estimation device of the embodiment of the present application, the first Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on the preset general 3D information flow, and the second Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on the preset feature flow, and the target Laplace distribution is obtained according to the first Laplace distribution and the second Laplace distribution; the pose of the object to be estimated is estimated based on the preset convolutional network to obtain the pose estimation result, and the size-independent pose of the object to be estimated is obtained according to the pose estimation result. In this way, the influence of shape uncertainty and scale uncertainty on pose estimation caused by shape differences within the category is overcome, and high-precision and high-accuracy prediction of the pose of the object at the category level can be achieved, and it is not affected by scale uncertainty.

[0072] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the category-level object pose estimation method based on the Laplace mixture model as described in the above embodiment.

[0073] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the category-level object pose estimation method based on the Laplace mixture model as described in the above embodiments.

[0074] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0076] Figure 1 An example diagram of shape uncertainty and scale uncertainty faced by category-level pose estimation based on RGB images;

[0077] Figure 2 A flowchart of a method for class-level object pose estimation based on a Laplace mixture model provided according to an embodiment of the present application;

[0078] Figure 3 A flowchart of class-level object pose estimation based on a Laplace mixture model according to an embodiment of the present application;

[0079] Figure 4 4 is a block diagram of a device for estimating the pose of a class-level object based on a Laplace mixture model according to an embodiment of the present application;

[0080] Figure 5 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0081] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0082] The following describes a method for estimating a class-level object pose based on a Laplace mixture model according to an embodiment of the present application with reference to the accompanying drawings.

[0083] Before introducing the category-level object pose estimation method based on the Laplace mixture model in the embodiment of the present application, the challenges faced by category-level pose estimation based on RGB images in the related art are first introduced.

[0084] Specifically, if Figure 1 As shown in , due to the lack of depth information, the shape uncertainty of each pixel will change. The shape uncertainty is particularly obvious in some areas of the image, such as Figure 1 In the camera shot in (a), the lack of depth information makes it particularly challenging to predict the translation and size of objects relying solely on visual appearance.

[0085] In addition, if Figure 1As shown in (b), it is impossible to determine the metric scale of an object from a single RGB image. For example, consider two scenarios: a larger object is farther away from the camera, and a smaller object is closer to the camera. Although the two objects contrast in size, they may appear exactly the same in the image, resulting in scale ambiguity.

[0086] In response to the shape uncertainty and scale uncertainty problems mentioned above, the present application provides a category-level object pose estimation method based on the Laplace mixture model, which overcomes the influence of shape uncertainty and scale uncertainty on pose estimation caused by shape differences within the category, and can achieve high-precision and high-accuracy prediction of category-level object poses, and is not affected by scale uncertainty.

[0087] The following introduces the category-level object pose estimation method based on the Laplace mixture model proposed in this application.

[0088] Specifically, Figure 2 A flowchart of a method for class-level object pose estimation based on a Laplace mixture model is provided in an embodiment of the present application.

[0089] like Figure 2 As shown, the category-level object pose estimation method based on the Laplace mixture model includes the following steps:

[0090] In step S201 , an image of an object to be estimated is acquired, and a NOCS coordinate map of the object to be estimated is predicted according to the image of the object to be estimated.

[0091] The image of the object to be estimated refers to a two-dimensional RGB image of the object for which pose estimation is required at a specific viewing angle.

[0092] Optionally, the embodiment of the present application can use the object detector Mask R-CNN (Mask Region-based Convolutional Neural Network) in the prior art to crop the object of interest as the image of the object to be estimated, which is not specifically limited here.

[0093] Furthermore, the image of the object to be estimated can be input into a pre-trained neural network model, and the pre-trained neural network model outputs the NOCS coordinates corresponding to each pixel point, thereby obtaining a predicted NOCS coordinate map.

[0094] In step S202, a first Laplace distribution of the NOCS coordinate map is predicted based on a preset general 3D information flow, and a second Laplace distribution of the NOCS coordinate map is predicted based on a preset feature flow, and a target Laplace distribution is obtained according to the first Laplace distribution and the second Laplace distribution.

[0095] It should be noted that, unlike the related art which deterministically predicts the Normalized Object Coordinate Space (NOCS) mapping, the embodiment of the present application models the NOCS coordinates as a probability distribution and introduces additional information of its variance, thereby explicitly measuring the shape uncertainty of each point.

[0096] In order to more fully understand the shape of the object to be estimated, this application uses a Laplacian mixture model (LMM), which combines two independent Laplace distributions from different information sources. This application chooses to use the Laplace distribution because it has a stronger ability to handle outliers than other distributions (such as Gaussian distribution), by utilizing two parallel information streams to capture different aspects of the geometry of the object to be estimated and independently predict the Laplace distribution of the NOCS map.

[0097] In order to extract generalizable, category-independent three-dimensional features, the embodiment of the present application adopts DINOv2 (a computer vision self-supervised model) as a generalized three-dimensional information flow. As an excellent basic model, DINOv2 has demonstrated the ability to establish zero-sample semantic correspondences across images and extract rich 3D information from images. Specifically, DINOv2 can extract SE(3) consistent local features to establish semantic correspondences between objects of different shapes and poses, which is consistent with the SE(3) invariance of NOCS coordinates, thereby facilitating the learning process of NOCS graphs. Therefore, the pre-trained DINOv2 is integrated into the framework as a preset general 3D information flow to predict the first Laplace distribution of the NOCS graph, thereby extracting category-independent 3D features.

[0098] However, relying solely on DINOv2 is not enough, as it is deficient in extracting category-specific features. Since DINOv2 is trained in a category-independent manner, the category-independent training method makes DINOv2 lack category-specific knowledge that is critical for pose estimation. This result emphasizes the need for complementary methods to enrich information about the geometry of objects. Therefore, in order to make up for this deficiency, in addition to utilizing the preset general 3D information flow, this application further trains a backbone network (i.e., a convolutional network) specifically designed to capture category-specific information as the preset feature flow to extract category-specific features.

[0099] Further, in some embodiments, a first Laplace distribution of the NOCS coordinate graph is predicted based on a preset universal 3D information flow, and a second Laplace distribution of the NOCS coordinate graph is predicted based on a preset feature flow, including: based on the preset universal 3D information flow, predicting the first Laplace distribution of the NOCS coordinate graph using a first preset loss function; based on the preset feature flow, predicting the second Laplace distribution of the NOCS coordinate graph using a second preset loss function.

[0100] Specifically, in order to estimate a single Laplace distribution Laplace(μ,σ2), the features of the preset general 3D information flow and the preset feature flow are used to independently estimate the pixel-level mean μ and variance σ 2 The Laplace uncertainty loss is used to simultaneously learn the mean μ of the Laplace distribution and the variance σ2 of the Laplace distribution, where the first preset loss function is:

[0101]

[0102] The second preset loss function is:

[0103]

[0104] Among them, L 3D-dino is the first preset loss function, is the variance of the first Laplace distribution, M vis is the mask of the visible part of the object to be estimated, is the real NOCS coordinate diagram, μ dino is the mean of the first Laplace distribution, L 3D-conv is the second preset loss function, is the variance of the second Laplace distribution, μ conv is the mean of the second Laplace distribution, and λ1 and λ2 are hyperparameters.

[0105] When the coordinate error of the mean μ of the Laplace distribution in the NOCS coordinate system is large, the L1-distance term (i.e. dominates, and in order to reduce this term, the variance σ of the Laplace distribution 2 is forced to increase; when the coordinate error is small, in order to reduce the logarithmic term, the variance σ of the Laplace distribution 2 is encouraged to decrease. Therefore, the variance σ of the Laplace distribution 2 It is learned in a self-supervised manner.

[0106] Finally, the first Laplace distribution and the second Laplace distribution independently predicted by the two branches are merged to form the final predicted target Laplace distribution.

[0107] In step S203, based on a preset convolutional network, the pose of the object to be estimated is estimated according to the target Laplace distribution to obtain a pose estimation result, and the pose of the object to be estimated that is independent of its size is obtained according to the pose estimation result.

[0108] In some embodiments, based on a preset convolutional network, the pose estimation of the object to be estimated is performed according to the target Laplace distribution to obtain a pose estimation result, including: obtaining a 2D pixel coordinate map of the image of the object to be estimated; inputting the target Laplace distribution and the 2D pixel coordinate map into the preset convolutional network to obtain a pose estimation result.

[0109] It is understandable that since it is very complicated to solve the pose explicitly through the target Laplace distribution, the embodiment of the present application establishes a two-dimensional to three-dimensional correspondence after estimating the Laplace mixture model, and adopts a preset convolutional network Φ as a PnP module, thereby allowing end-to-end training and efficient reasoning to solve the pose problem using the PnP module constructed by the preset convolutional network. The PnP module benefits from the Laplace mixture model mainly in two aspects: first, the PnP module can dynamically aggregate the diverse object geometry information captured by the dual information streams; second, the PnP module can evaluate the reliability of different information sources through the variance information in the LMM, thereby giving more reliable information a higher weight in the pose solution process. This method not only improves the accuracy of pose estimation, but also enhances the robustness of the model to different object shapes and pose changes.

[0110] Specifically, the embodiment of the present application can process the image of the object to be estimated by using an image processing library (such as OpenCV) in the prior art, thereby obtaining a 2D pixel coordinate map of the image of the object to be estimated, and combining the target Laplace distribution and the 2D pixel coordinate map C 2D Input to the preset convolutional network to predict the scale-independent rotation parameter R out and the translation parameter t out ,Right now:

[0111]

[0112] Among them, R out is the predicted scale-independent rotation parameter, t out is the predicted scale-independent translation parameter, Φ() is the preset convolutional network, μ dino is the mean of the first Laplace distribution, is the variance of the first Laplace distribution, μ conv is the mean of the second Laplace distribution, is the variance of the second Laplace distribution, C 2D is the 2D pixel coordinate map of the image of the object to be estimated.

[0113] Furthermore, in some embodiments, a size-independent pose of an object to be estimated is obtained based on a pose estimation result, including: obtaining the 3D size of the object to be estimated and the 3D translation of the object to be estimated, and calculating the original diagonal length of a tight bounding box of the object to be estimated based on the 3D size of the object to be estimated; normalizing the size of the object to be estimated according to the 3D size of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated to obtain the normalized 3D size of the object to be estimated; normalizing the translation of the object according to the 3D translation vector of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated to obtain the normalized translation of the object to be estimated, and obtaining a predicted translation amount based on the normalized 3D translation of the object to be estimated.

[0114] Specifically, since it is impossible to determine the metric scale of the object to be estimated based on only one RGB image, and due to the lack of depth information, it becomes particularly challenging to predict the translation and size of the object based solely on the visual appearance. Therefore, the embodiment of the present application is for a size of s={s x ,s y ,s z}, first calculate the original diagonal length of the tight bounding box of the object to be estimated based on the 3D size s of the object to be estimated, and then normalize the size of the object to be estimated according to the 3D size of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated as follows:

[0115]

[0116] Among them, s norm is the normalized 3D size of the object to be estimated, s x 、s y 、s z are the normalized sizes in the x, y, and z directions respectively, and d is the original diagonal length of the tight bounding box of the object.

[0117] Furthermore, the average normalized size s of the category is pre-calculated avg , and predict the normalized size difference s out , where s out =s norm -s avg The range of the difference is normalized to be around zero, which is convenient for network prediction.

[0118] Furthermore, corresponding to the size of the object to be estimated, the translation of the object to be estimated is t={t x ,t y ,t z It is also normalized using the original diagonal length d of the object's tight bounding box:

[0119]

[0120] Among them, t norm is the normalized translation of the object to be estimated, t x ,t y ,t z are the components of the estimated translation in the x, y, and z directions at the absolute scale, d is the original diagonal length of the tight bounding box of the object, The components of the normalized distance from the object to be estimated to the camera in the x, y, and z directions.

[0121] Furthermore, similar to the RGB-based work, the 3D translation is separated into two components: the 2D position of the center of the projected object ( x ,o y ) and the normalized distance from the object to the camera

[0122] Since the input image is cropped and scaled according to the detection results, the regression is a translation parameter that is independent of the detection and scale. This means that the model needs to be able to handle objects of different sizes and positions without specific preprocessing of the input image. This strategy makes the model more general and robust, and can accurately estimate the pose of objects in various practical scenarios. Among them, the predicted translation is:

[0123] t out =[δ x ,δ y ,δ z ];

[0124] δ x =(o x -c x ) / w box ;

[0125] δ y =(o y -c y ) / h box ;

[0126]

[0127] Among them, t out is the predicted translation, δ x , δ y are the relative offsets of the center of the 2D detection frame in the x and y axes of the image plane, δ z It is the relative value of the object's translation on the z-axis relative to the size of the detection frame. x ,o y ) is the 2D position of the center of the projected object, (cx ,c y ) is the center coordinate of the detected 2D bounding box, w box is the width of the detected 2D bounding box, h box is the height of the detected 2D bounding box, is the normalized distance from the object to be estimated to the camera, s box is the size of the detected 2D bounding box, s in is the size of the image of the object to be estimated after scaling.

[0128] Among them, the size of the 2D bounding box s box The larger value of the width of the 2D bounding box and the height of the 2D bounding box, that is, s box =max(w box ,h box ). The size of the image of the object to be estimated after scaling s in is the larger value of the width of the image of the object to be estimated after scaling and the height of the image of the object to be estimated after scaling, that is, s in =max(w in ,h in ), where w in is the width of the image of the object to be estimated after scaling, h in is the height of the scaled image of the object to be estimated.

[0129] Furthermore, in some embodiments, obtaining a size-independent posture of an object based on a posture estimation result also includes: predicting a first rotation vector of the object to be estimated along a first direction axis, and predicting a second rotation vector of the object along a second direction axis, wherein the first direction axis and the second direction axis are perpendicular; determining whether the object to be estimated has rotational symmetry; if the object to be estimated has rotational symmetry, supervising the second rotation vector when training a model of the object to be estimated.

[0130] The first direction axis may be the x-axis, and the second direction axis may be the y-axis.

[0131] Specifically, in the representation of rotation R, a first rotation vector along the x-axis and a second rotation vector along the y-axis are predicted, R out =[r x ,r y ](in ), where the first rotation vector and the second rotation vector correspond to the first two columns of the rotation matrix respectively.

[0132] However, when dealing with objects with rotational symmetry (such as bowls and bottles), these objects have symmetry around a particular axis, which results in the first rotation vector r xTherefore, when training models for these objects with rotational symmetry, only the second rotation vector r y Supervision is performed to avoid the first rotation vector r x Caused by blurry issues.

[0133] In order to help those skilled in the art to more clearly and intuitively understand the category-level object pose estimation method based on the Laplace mixture model in the embodiment of the present application, a detailed description is given below in conjunction with specific embodiments.

[0134] It can be understood that the purpose of this application is to solve the problem of RGB-based category-level object pose estimation. Specifically, given an RGB image containing objects from a predefined category set, the goal of this application is to detect all object instances present in the scene and accurately estimate their 9-degree-of-freedom (9DoF) poses, where the 9DoF poses include 3-degree-of-freedom rotation R∈SO(3), 3-degree-of-freedom translation t∈R3, and 3-degree-of-freedom size s∈R3.

[0135] To this end, the present application proposes a method for pose estimation using a Laplace mixture model, which performs RGB-based category-level object pose estimation by modeling the object shape as a Laplace mixture model.

[0136] Specifically, if Figure 3 As shown in the figure, we first use the existing object detector Mask R-CNN to crop out the object of interest as input. Then, we use two information streams to independently predict the two Laplace distributions of the NOCS coordinate map. Next, we use the LMM-based PnP solution method to use the estimated LMM for pose estimation. Finally, the PnP module calculates the size-independent pose.

[0137] According to the method for estimating the pose of a class-level object based on a Laplace mixture model in an embodiment of the present application, a first Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on a preset general 3D information flow, and a second Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on a preset feature flow, and a target Laplace distribution is obtained according to the first Laplace distribution and the second Laplace distribution; pose estimation is performed on the object to be estimated based on a preset convolutional network to obtain a pose estimation result, and a size-independent pose of the object to be estimated is obtained according to the pose estimation result. Thus, the influence of shape uncertainty and scale uncertainty on pose estimation caused by shape differences within a class is overcome, and high-precision and high-accuracy prediction of the pose of class-level objects can be achieved, and it is not affected by scale uncertainty.

[0138] Next, a category-level object pose estimation device based on a Laplace mixture model proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0139] Figure 4 It is a block diagram of a device for estimating the pose of a category-level object based on a Laplace mixture model according to an embodiment of the present application.

[0140] like Figure 4 As shown, the category-level object pose estimation device 10 based on the Laplace mixture model includes: an acquisition module 100, a prediction module 200 and a pose estimation module 300.

[0141] Among them, the acquisition module 100 is used to acquire an image of the object to be estimated, and predict the NOCS coordinate map of the object to be estimated based on the image of the object to be estimated; the prediction module 200 is used to predict the first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow, and predict the second Laplace distribution of the NOCS coordinate map based on a preset feature flow, and obtain the target Laplace distribution according to the first Laplace distribution and the second Laplace distribution; the pose estimation module 300 is used to perform pose estimation on the object to be estimated according to the target Laplace distribution based on a preset convolutional network, obtain a pose estimation result, and obtain a size-independent pose of the object to be estimated according to the pose estimation result.

[0142] Further, in some embodiments, the prediction module 200 is used to: predict a first Laplace distribution of the NOCS coordinate map using a first preset loss function based on a preset general 3D information flow; predict a second Laplace distribution of the NOCS coordinate map using a second preset loss function based on a preset feature flow; wherein the first preset loss function is:

[0143]

[0144] The second preset loss function is:

[0145]

[0146] Among them, L 3D-dino is the first preset loss function, is the variance of the first Laplace distribution, M vis is the mask of the visible part of the object to be estimated, is the real NOCS coordinate diagram, μ dino is the mean of the first Laplace distribution, L 3D-conv is the second preset loss function, is the variance of the second Laplace distribution, μ conv is the mean of the second Laplace distribution, and λ1 and λ2 are hyperparameters.

[0147] Further, in some embodiments, the pose estimation module 300 is used to: obtain a 2D pixel coordinate map of the image of the object to be estimated; input the target Laplace distribution and the 2D pixel coordinate map into a preset convolutional network to obtain a pose estimation result; wherein the pose estimation result is:

[0148]

[0149] Among them, R out is the predicted scale-independent rotation parameter, t out is the predicted scale-independent translation parameter, Φ() is the preset convolutional network, μ dino is the mean of the first Laplace distribution, is the variance of the first Laplace distribution, μ conv is the mean of the second Laplace distribution, is the variance of the second Laplace distribution, C 2D is the 2D pixel coordinate map of the image of the object to be estimated.

[0150] Further, in some embodiments, the pose estimation module 300 is used to: obtain the 3D size of the object to be estimated and the 3D translation of the object to be estimated, and calculate the original diagonal length of the object's tight bounding box based on the 3D size of the object to be estimated; normalize the size of the object to be estimated according to the 3D size of the object to be estimated and the original diagonal length of the object's tight bounding box to obtain the normalized 3D size of the object to be estimated; normalize the translation of the object according to the 3D translation vector of the object to be estimated and the original diagonal length of the object's tight bounding box to obtain the normalized translation of the object to be estimated, and obtain a predicted translation amount according to the normalized 3D translation of the object to be estimated, wherein the predicted translation amount is:

[0151] t out =[δ x ,δ y ,δ z ];

[0152] δ x =(o x -c x ) / w box ;

[0153] δ y =(o y -c y ) / h box ;

[0154]

[0155] Among them, t out is the predicted translation, δx , δ y are the relative offsets of the center of the 2D detection frame in the x and y axes of the image plane, δ z It is the relative value of the object's translation on the z-axis relative to the size of the detection frame. x ,o y ) is the 2D position of the center of the projected object, (c x ,c y ) is the center coordinate of the 2D bounding box, w box is the width of the 2D bounding box, h box is the height of the 2D bounding box, is the normalized distance from the object to be estimated to the camera, s box is the size of the 2D bounding box, s in is the size of the image of the object to be estimated after scaling.

[0156] Furthermore, in some embodiments, the pose estimation module 300 is also used to: predict a first rotation vector of the object to be estimated along a first direction axis, and predict a second rotation vector of the object along a second direction axis, wherein the first direction axis and the second direction axis are perpendicular; determine whether the object to be estimated has rotational symmetry; if the object to be estimated has rotational symmetry, supervise the second rotation vector when training the model of the object to be estimated.

[0157] It should be noted that the aforementioned explanation of the embodiment of the category-level object pose estimation method based on the Laplace mixture model is also applicable to the category-level object pose estimation device based on the Laplace mixture model of this embodiment, and will not be repeated here.

[0158] According to the Laplace mixture model-based category-level object pose estimation device of the embodiment of the present application, the first Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on the preset general 3D information flow, and the second Laplace distribution of the NOCS coordinate map of the object to be estimated is predicted based on the preset feature flow, and the target Laplace distribution is obtained according to the first Laplace distribution and the second Laplace distribution; the pose of the object to be estimated is estimated based on the preset convolutional network to obtain the pose estimation result, and the size-independent pose of the object to be estimated is obtained according to the pose estimation result. In this way, the influence of shape uncertainty and scale uncertainty on pose estimation caused by shape differences within the category is overcome, and high-precision and high-accuracy prediction of the pose of the object at the category level can be achieved, and it is not affected by scale uncertainty.

[0159] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0160] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0161] When the processor 502 executes the program, the category-level object pose estimation method based on the Laplace mixture model provided in the above embodiment is implemented.

[0162] Furthermore, the electronic device further comprises:

[0163] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0164] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0165] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0166] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0167] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0168] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0169] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned category-level object pose estimation method based on the Laplace mixture model.

[0170] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0171] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0172] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for class-level object pose estimation based on a Laplace mixture model, characterized in that: The following steps are involved: Acquire an image of the object to be estimated, and predict a NOCS coordinate map of the object to be estimated according to the image of the object to be estimated; Predicting a first Laplace distribution of the NOCS coordinate map based on a preset universal 3D information flow, predicting a second Laplace distribution of the NOCS coordinate map based on a preset feature flow, and obtaining a target Laplace distribution according to the first Laplace distribution and the second Laplace distribution; Based on a preset convolutional network, the pose of the object to be estimated is estimated according to the target Laplace distribution to obtain a pose estimation result, and the size-independent pose of the object to be estimated is obtained according to the pose estimation result.

2. The method according to claim 1, characterized in that The predicting of the first Laplace distribution of the NOCS coordinate map based on a preset universal 3D information flow, and predicting the second Laplace distribution of the NOCS coordinate map based on a preset feature flow, includes: Based on the preset universal 3D information flow, predicting a first Laplace distribution of the NOCS coordinate map using a first preset loss function; Based on the preset feature flow, predicting a second Laplace distribution of the NOCS coordinate graph using a second preset loss function; Wherein, the first preset loss function is: The second preset loss function is: Among them, L 3D-dino is the first preset loss function, is the variance of the first Laplace distribution, M vis is the mask of the visible part of the object to be estimated, is the real NOCS coordinate diagram, μ dino is the mean of the first Laplace distribution, L 3D-conv is the second preset loss function, is the variance of the second Laplace distribution, μ conv is the mean of the second Laplace distribution, and λ1 and λ2 are hyperparameters.

3. The method according to claim 1, characterized in that The method of performing pose estimation on the object to be estimated based on the preset convolutional network according to the target Laplace distribution to obtain a pose estimation result includes: Obtaining a 2D pixel coordinate map of the image of the object to be estimated; Inputting the target Laplace distribution and the 2D pixel coordinate map into the preset convolutional network to obtain the pose estimation result; Among them, the pose estimation result is: Among them, R out is the predicted scale-independent rotation parameter, t out is the predicted scale-independent translation parameter, Φ() is the preset convolutional network, μ dino is the mean of the first Laplace distribution, is the variance of the first Laplace distribution, μ conv is the mean of the second Laplace distribution, is the variance of the second Laplace distribution, C 2D is the 2D pixel coordinate map of the image of the object to be estimated.

4. The method according to claim 1, characterized in that: The step of obtaining a size-independent pose of the object to be estimated according to the pose estimation result includes: Acquire the 3D size of the object to be estimated and the 3D translation of the object to be estimated, and calculate the original diagonal length of the object tight bounding box of the object to be estimated based on the 3D size of the object to be estimated; Normalizing the size of the object to be estimated according to the 3D size of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated to obtain the normalized 3D size of the object to be estimated; The translation of the object to be estimated is normalized according to the 3D translation vector of the object to be estimated and the original diagonal length of the tight bounding box of the object to be estimated, so as to obtain the normalized translation of the object to be estimated, and a predicted translation amount is obtained according to the normalized 3D translation of the object to be estimated, wherein the predicted translation amount is: t out =[δ x ,d y ,d z ]; d x =(o x -c x ) / w box ; d y =(o y -c y ) / h box ; Among them, t out is the predicted translation parameter, δ x , δ y are the relative offsets of the center of the 2D detection frame in the x and y axes of the image plane, δ z is the relative value of the object's translation on the z-axis relative to the size of the detection frame, (o x ,o y ) is the 2D position of the center of the projected object, (c x ,) is the center coordinate of the 2D bounding box, w box is the width of the 2D bounding box, h box is the height of the 2D bounding box, is the normalized distance from the object to be estimated to the camera, s box is the size of the 2D bounding box, s in is the size of the image of the object to be estimated after scaling.

5. The method according to claim 1, characterized in that The step of obtaining the size-independent posture of the object according to the posture estimation result further includes: Predicting a first rotation vector of the object to be estimated along a first direction axis, and predicting a second rotation vector of the object along a second direction axis, wherein the first direction axis is perpendicular to the second direction axis; Determining whether the object to be estimated has rotational symmetry; If the object to be estimated has rotational symmetry, the second rotation vector is supervised when training the model of the object to be estimated.

6. A class-level object pose estimation device based on a Laplace mixture model, characterized in that: include: An acquisition module, used for acquiring an image of the object to be estimated, and predicting a NOCS coordinate map of the object to be estimated according to the image of the object to be estimated; A prediction module, configured to predict a first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow, predict a second Laplace distribution of the NOCS coordinate map based on a preset feature flow, and obtain a target Laplace distribution according to the first Laplace distribution and the second Laplace distribution; The pose estimation module is used to perform pose estimation on the object to be estimated based on a preset convolutional network and according to the target Laplace distribution to obtain a pose estimation result, and obtain a size-independent pose of the object to be estimated based on the pose estimation result.

7. The device according to claim 6, characterized in that The prediction module is used to: Based on the preset universal 3D information flow, predicting a first Laplace distribution of the NOCS coordinate map using a first preset loss function; Based on the preset feature flow, predicting a second Laplace distribution of the NOCS coordinate graph using a second preset loss function; Wherein, the first preset loss function is: The second preset loss function is: Among them, L 3D-dino is the first preset loss function, is the variance of the first Laplace distribution, M vis is the mask of the visible part of the object to be estimated, is the real NOCS coordinate diagram, μ dino is the mean of the first Laplace distribution, L 3D-conv is the second preset loss function, is the variance of the second Laplace distribution, μ conv is the mean of the second Laplace distribution, and λ1 and λ2 are hyperparameters.

8. The device according to claim 6, characterized in that The posture estimation module is used to: Obtaining a 2D pixel coordinate map of the image of the object to be estimated; Inputting the target Laplace distribution and the 2D pixel coordinate map into the preset convolutional network to obtain the pose estimation result; Among them, the pose estimation result is: Among them, R out is the predicted scale-independent rotation parameter, t out is the predicted scale-independent translation parameter, Φ() is the preset convolutional network, μ dino is the mean of the first Laplace distribution, is the variance of the first Laplace distribution, μ conv is the mean of the second Laplace distribution, is the variance of the second Laplace distribution, C 2D is the 2D pixel coordinate map of the image of the object to be estimated.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the category-level object pose estimation method based on the Laplace mixture model as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the category-level object pose estimation method based on the Laplace mixture model as described in any one of claims 1 to 5.

Images