Class-level object pose estimation method based on laplace mixture model

Abstract

The application relates to a class-level object pose estimation method based on a Laplace mixture model, which comprises the following steps: predicting a NOCS coordinate graph of an object to be estimated according to an image of the object to be estimated; predicting a first Laplace distribution of the NOCS coordinate graph based on a preset general 3D information flow, predicting a second Laplace distribution of the NOCS coordinate graph based on a preset feature flow, and obtaining a target Laplace distribution according to the first Laplace distribution and the second Laplace distribution; performing pose estimation on the object to be estimated according to the target Laplace distribution based on a preset convolutional network, obtaining a pose estimation result, and obtaining a size-independent pose of the object to be estimated according to the pose estimation result. Therefore, the influence of shape uncertainty and scale uncertainty on pose estimation caused by intra-class shape differences is overcome, high-precision and high-accuracy prediction of the class-level object pose can be realized, and the influence of scale uncertainty is avoided.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a category-level object pose estimation method based on a Laplacian mixture model. Background Technology

[0002] Category-level object pose estimation refers to predicting the 9-DOF pose of an unseen object in a given set of categories. This includes 3D rotation (3D), translation (3D), and size (3D). Category-level object pose estimation has wide applications in robotics, augmented reality (AR), virtual reality (VR), and 3D understanding.

[0003] While RGB-D (RGB plus depth) based methods have achieved good results in category-level object pose estimation, most methods typically rely heavily on depth sensors, which limits their applicability in general scenarios. Therefore, an RGB (Red, Green, Blue) based category-level object pose estimation method is proposed as an alternative suitable for deployment in embedded devices such as AR glasses and mobile phones.

[0004] However, the lack of depth information presents two significant challenges to RGB-based class-level object pose estimation: First, the prediction of object shape becomes more complex due to the absence of depth information, exacerbating the difficulty of handling intra-class shape changes; second, relying solely on RGB input introduces inherent scale ambiguity, making translation and size estimation ill-posed problems.

[0005] In related technologies, two solutions have been proposed to overcome the above challenges. One is to establish a 3D-3D correspondence by estimating the metric depth and the coordinates in the Normalized Object Coordinate Space (NOCS), and then solve the pose using the Umeyama algorithm. The other is to establish a 2D-3D correspondence by estimating the NOCS coordinate map and the metric dimensions of the object separately, and then solve the pose using the PnP (Perspective-n-Point) algorithm.

[0006] However, while related technologies have addressed the RGB-based pose estimation problem to some extent, they still have the following limitations: First, the lack of depth information makes accurate object shape measurement difficult, and shape uncertainty is particularly pronounced in certain regions of the image. This makes establishing precise correspondences more complex. Related technologies typically treat the predicted correspondences for each pixel as equivalent and rely on RANSAC to filter outout values, which reduces prediction speed and affects robustness. Second, some related technologies do not consider scale ambiguity, while others use the same features to predict NOCS maps and measure dimensions. However, inferring dimensions from a single RGB image is inherently an ill-posed problem, leading to unstable training of other components and poor results, which urgently needs to be addressed. Summary of the Invention

[0007] This application provides a category-level object pose estimation method based on a Laplace mixture model to overcome the influence of shape uncertainty and scale uncertainty caused by shape differences within a category on pose estimation. It can achieve high-precision and high-accuracy prediction of object pose at the category level and is not affected by scale uncertainty.

[0008] The first aspect of this application provides a category-level object pose estimation method based on a Laplace mixture model, comprising the following steps: Acquire an image of the object to be estimated, and predict the NOCS coordinate map of the object to be estimated based on the image of the object to be estimated; The first Laplace distribution of the NOCS coordinate map is predicted based on a preset general 3D information flow, and the second Laplace distribution of the NOCS coordinate map is predicted based on a preset feature flow. The target Laplace distribution is obtained based on the first Laplace distribution and the second Laplace distribution. Based on a pre-defined convolutional network, the pose of the object to be estimated is estimated according to the target Laplacian distribution to obtain the pose estimation result, and the size-independent pose of the object to be estimated is obtained based on the pose estimation result.

[0009] According to one embodiment of this application, predicting the first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow, and predicting the second Laplace distribution of the NOCS coordinate map based on a preset feature flow, includes: Based on the preset general 3D information flow, the first Laplace distribution of the NOCS coordinate map is predicted using the first preset loss function; Based on the preset feature flow, the second Laplace distribution of the NOCS coordinate map is predicted using the second preset loss function; The first preset loss function is: ; The second preset loss function is: ; in, The first preset loss function is... Let V be the variance of the first Laplace distribution. This is the mask for the visible portion of the object to be estimated. This is a true NOCS coordinate map. The mean of the first Laplace distribution. The second preset loss function, Let Variance be the variance of the second Laplace distribution. The mean of the second Laplace distribution. and This is a hyperparameter.

[0010] According to one embodiment of this application, the pose estimation of the object to be estimated based on the target Laplacian distribution using a preset convolutional network to obtain the pose estimation result includes: Obtain a 2D pixel coordinate map of the image of the object to be estimated; The target Laplacian distribution and the 2D pixel coordinate map are input into the preset convolutional network to obtain the pose estimation result; The pose estimation result is as follows: ; in, For the predicted scale-independent rotation parameters, For the predicted scale-independent translation parameters, For the pre-defined convolutional network, The mean of the first Laplace distribution. Let V be the variance of the first Laplace distribution. The mean of the second Laplace distribution. Let Variance be the variance of the second Laplace distribution. This is a 2D pixel coordinate map of the image of the object to be estimated.

[0011] According to one embodiment of this application, obtaining the size-independent pose of the object to be estimated based on the pose estimation result includes: Obtain the 3D dimensions and 3D translation of the object to be estimated, and calculate the original diagonal length of the object tight bounding box of the object to be estimated based on the 3D dimensions of the object to be estimated. The size of the object to be estimated is normalized based on the 3D size of the object and the original diagonal length of the object's tight bounding box to obtain the normalized 3D size of the object to be estimated. The translation of the object is normalized based on its 3D translation vector and the original diagonal length of its tight bounding box, resulting in a normalized translation. The predicted translation amount is then obtained based on this normalized 3D translation, where the predicted translation amount is: ; ; ; ; in, For the predicted translation parameters, , These are the relative offsets from the center of the 2D detection box along the x and y axes of the image plane, respectively. This is the relative value of the object's translation along the z-axis to the size of the detection box. The 2D position of the center of the projected object. The center coordinates of the 2D bounding box. The width of the 2D bounding box. The height of the 2D bounding box. To estimate the normalized distance from the object to the camera, The size of the 2D bounding box. The size of the scaled image of the object to be estimated.

[0012] According to one embodiment of this application, obtaining the size-independent pose of the object based on the pose estimation result further includes: A first rotation vector of the object to be estimated is predicted along a first direction axis, and a second rotation vector of the object is predicted along a second direction axis, wherein the first direction axis and the second direction axis are perpendicular; Determine whether the object to be estimated has rotational symmetry; If the object to be estimated has rotational symmetry, the second rotation vector is supervised when training the model of the object to be estimated.

[0013] According to the embodiment of this application, the category-level object pose estimation method based on the Laplacian mixture model predicts the first Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset general 3D information flow, and predicts the second Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset feature flow. The target Laplacian distribution is obtained based on the first and second Laplacian distributions. The object's pose is estimated using a preset convolutional network to obtain the pose estimation result, and the size-independent pose of the object is obtained based on the pose estimation result. Therefore, this method overcomes the influence of shape uncertainty and scale uncertainty caused by intra-category shape differences on pose estimation, achieving high-precision and high-accuracy prediction of category-level object poses, and is unaffected by scale uncertainty.

[0014] A second aspect of this application provides a category-level object pose estimation device based on a Laplace mixture model, comprising: An acquisition module is used to acquire an image of the object to be estimated and predict the NOCS coordinate map of the object to be estimated based on the image of the object to be estimated. The prediction module is used to predict the first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow, and to predict the second Laplace distribution of the NOCS coordinate map based on a preset feature flow, and to obtain the target Laplace distribution based on the first Laplace distribution and the second Laplace distribution. The pose estimation module is used to estimate the pose of the object to be estimated based on a preset convolutional network and the target Laplacian distribution, to obtain the pose estimation result, and to obtain the size-independent pose of the object to be estimated based on the pose estimation result.

[0015] According to one embodiment of this application, the prediction module is configured to: Based on the preset general 3D information flow, the first Laplace distribution of the NOCS coordinate map is predicted using the first preset loss function; Based on the preset feature flow, the second Laplace distribution of the NOCS coordinate map is predicted using the second preset loss function; The first preset loss function is: ; The second preset loss function is: ; in, The first preset loss function is... Let V be the variance of the first Laplace distribution. This is the mask for the visible portion of the object to be estimated. This is a true NOCS coordinate map. The mean of the first Laplace distribution. The second preset loss function, Let Variance be the variance of the second Laplace distribution. The mean of the second Laplace distribution. and This is a hyperparameter.

[0016] According to one embodiment of this application, the pose estimation module is used for: Obtain a 2D pixel coordinate map of the image of the object to be estimated; The target Laplacian distribution and the 2D pixel coordinate map are input into the preset convolutional network to obtain the pose estimation result; The pose estimation result is as follows: ; in, For the predicted scale-independent rotation parameters, For the predicted scale-independent translation parameters, For the pre-defined convolutional network, The mean of the first Laplace distribution. Let V be the variance of the first Laplace distribution. The mean of the second Laplace distribution. Let Variance be the variance of the second Laplace distribution. This is a 2D pixel coordinate map of the image of the object to be estimated.

[0017] According to one embodiment of this application, the pose estimation module is used for: Obtain the 3D dimensions and 3D translation of the object to be estimated, and calculate the original diagonal length of the object tight bounding box of the object to be estimated based on the 3D dimensions of the object to be estimated. The size of the object to be estimated is normalized based on the 3D size of the object and the original diagonal length of the object's tight bounding box to obtain the normalized 3D size of the object to be estimated. The translation of the object is normalized based on its 3D translation vector and the original diagonal length of its tight bounding box, resulting in a normalized translation. The predicted translation amount is then obtained based on this normalized 3D translation, where the predicted translation amount is: ; ; ; ; in, For the predicted translation parameters, , These are the relative offsets from the center of the 2D detection box along the x and y axes of the image plane, respectively. This is the relative value of the object's translation along the z-axis to the size of the detection box. The 2D position of the center of the projected object. The center coordinates of the 2D bounding box. The width of the 2D bounding box. The height of the 2D bounding box. To estimate the normalized distance from the object to the camera, The size of the 2D bounding box. The size of the scaled image of the object to be estimated.

[0018] According to one embodiment of this application, the pose estimation module is further configured to: A first rotation vector of the object to be estimated is predicted along a first direction axis, and a second rotation vector of the object is predicted along a second direction axis, wherein the first direction axis and the second direction axis are perpendicular; Determine whether the object to be estimated has rotational symmetry; If the object to be estimated has rotational symmetry, the second rotation vector is supervised when training the model of the object to be estimated.

[0019] According to the embodiments of this application, a category-level object pose estimation device based on a Laplacian mixture model predicts a first Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset general 3D information flow, and predicts a second Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset feature flow. A target Laplacian distribution is obtained based on the first and second Laplacian distributions. A pose estimation of the object to be estimated is performed based on a preset convolutional network to obtain a pose estimation result, and a size-independent pose of the object to be estimated is obtained based on the pose estimation result. Therefore, the influence of shape uncertainty and scale uncertainty caused by intra-category shape differences on pose estimation is overcome, enabling high-precision and high-accuracy prediction of category-level object poses, unaffected by scale uncertainty.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the category-level object pose estimation method based on the Laplacian mixture model as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the category-level object pose estimation method based on the Laplace mixture model as described in the above embodiments.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 Example diagrams illustrating the shape and scale uncertainties faced by category-level pose estimation based on RGB images; Figure 2 This is a flowchart of a category-level object pose estimation method based on a Laplace mixture model provided in an embodiment of this application; Figure 3 This is a flowchart of a category-level object pose estimation based on a Laplace mixture model according to an embodiment of this application; Figure 4 This is a block diagram of a category-level object pose estimation device based on a Laplace mixture model according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0025] The following describes a category-level object pose estimation method based on a Laplace mixture model according to embodiments of this application, with reference to the accompanying drawings.

[0026] Before introducing the category-level object pose estimation method based on the Laplacian mixture model in the embodiments of this application, we will first introduce the challenges faced by category-level pose estimation based on RGB images in related technologies.

[0027] Specifically, such as Figure 1 As shown, due to the lack of depth information, the shape uncertainty of each pixel changes, and this shape uncertainty is particularly noticeable in certain areas of the image, such as... Figure 1The camera lens in (a) lacks depth information, making it particularly challenging to predict the translation and size of objects based solely on visual appearance.

[0028] In addition, such as Figure 1 As shown in (b), it is impossible to determine the scale of an object based on a single RGB image. For example, consider two scenarios: a larger object is farther from the camera, while a smaller object is closer to the camera. Although the two objects are contrasting in size, they may appear exactly the same in the image, resulting in scale ambiguity.

[0029] To address the aforementioned shape and scale uncertainties, this application provides a category-level object pose estimation method based on a Laplace mixture model. This method overcomes the impact of shape and scale uncertainties caused by intra-category shape differences on pose estimation, achieving high-precision and high-accuracy prediction of category-level object poses, and is unaffected by scale uncertainties.

[0030] The following section introduces the category-level object pose estimation method based on the Laplace mixture model proposed in this application.

[0031] Specifically, Figure 2 This is a flowchart illustrating a category-level object pose estimation method based on a Laplace mixture model, provided in an embodiment of this application.

[0032] like Figure 2 As shown, this category-level object pose estimation method based on the Laplace mixture model includes the following steps: In step S201, an image of the object to be estimated is acquired, and the NOCS coordinate map of the object to be estimated is predicted based on the image of the object to be estimated.

[0033] Here, the image of the object to be estimated refers to the two-dimensional RGB image of the object whose pose estimation is required from a specific viewpoint.

[0034] Alternatively, embodiments of this application may use the existing object detector Mask R-CNN (MaskRegion-based Convolutional Neural Network) to crop out the image of the object of interest as the object to be estimated, without making specific limitations here.

[0035] Furthermore, the image of the object to be estimated can be input into a pre-trained neural network model, which outputs the NOCS coordinates corresponding to each pixel, thereby obtaining the predicted NOCS coordinate map.

[0036] In step S202, the first Laplace distribution of the NOCS coordinate map is predicted based on the preset general 3D information flow, and the second Laplace distribution of the NOCS coordinate map is predicted based on the preset feature flow. The target Laplace distribution is obtained according to the first Laplace distribution and the second Laplace distribution.

[0037] It should be noted that, unlike the deterministic prediction of the Normalized Object Coordinate Space (NOCS) mapping in related technologies, the embodiments of this application model the NOCS coordinates as a probability distribution and introduce additional information about its variance, thereby explicitly measuring the shape uncertainty of each point.

[0038] To gain a more comprehensive understanding of the shape of the object to be estimated, this application employs a Laplacian Mixture Model (LMM), which combines two independent Laplacian distributions from different information sources. The Laplacian distribution was chosen because it is more capable of handling outliers than other distributions (such as Gaussian distributions), capturing different aspects of the geometry of the object to be estimated by utilizing two parallel information streams, and independently predicting the Laplacian distribution of the NOCS mapping.

[0039] To extract generalizable, category-independent 3D features, this application employs DINOv2 (a self-supervised computer vision model) as a generalized 3D information flow. DINOv2, as an excellent foundational model, has demonstrated the ability to establish zero-sample semantic correspondences across images and extract rich 3D information from images. Specifically, DINOv2 can extract SE(3) consistent local features to establish semantic correspondences between objects of different shapes and poses, which aligns with the SE(3) invariance of NOCS coordinates, thus aiding the learning process of NOCS graphs. Therefore, pre-trained DINOv2 is integrated into the framework as a preset general 3D information flow to predict the first Laplacian distribution of NOCS graphs, thereby extracting category-independent 3D features.

[0040] However, relying solely on DINOv2 is insufficient, as it falls short in extracting category-specific features. Because DINOv2 is trained in a category-agnostic manner, this training method results in a lack of category-specific knowledge crucial for pose estimation. This underscores the need for supplementary methods to enrich information about object geometry. Therefore, to address this deficiency, this application, in addition to utilizing a pre-defined general 3D information flow, further trains a backbone network (i.e., a convolutional network) specifically designed to capture category-specific information as a pre-defined feature flow to extract category-specific features.

[0041] Furthermore, in some embodiments, predicting the first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow and predicting the second Laplace distribution of the NOCS coordinate map based on a preset feature flow includes: predicting the first Laplace distribution of the NOCS coordinate map using a first preset loss function based on a preset general 3D information flow; and predicting the second Laplace distribution of the NOCS coordinate map using a second preset loss function based on a preset feature flow.

[0042] Specifically, to estimate a single Laplace distribution Laplace(µ, σ²), the pixel-level mean is independently estimated using features from a pre-defined general 3D information flow and a pre-defined feature flow. and variance The Laplace uncertainty loss is used to simultaneously learn the mean µ and variance σ² of the Laplace distribution, where the first preset loss function is:

[0043] The second preset loss function is:

[0044] in, The first preset loss function is... Let V be the variance of the first Laplace distribution. This is the mask for the visible portion of the object to be estimated. This is a true NOCS coordinate map. The mean of the first Laplace distribution. The second preset loss function, Let Variance be the variance of the second Laplace distribution. The mean of the second Laplace distribution. and This is a hyperparameter.

[0045] When the coordinate error of the mean μ of the Laplace distribution in the NOCS coordinate system is large, the L1-distance term (i.e. The variance of the Laplace distribution is dominant, and to reduce this term, the variance of the Laplace distribution is... Forced to increase; when the coordinate error is small, in order to reduce the logarithmic term, the variance of the Laplace distribution increases. It is encouraged to decrease. Therefore, the variance of the Laplace distribution... It learns through self-monitoring.

[0046] Finally, the first and second Laplace distributions predicted independently by the two branches are merged to form the final predicted target Laplace distribution.

[0047] In step S203, based on a preset convolutional network, the pose of the object to be estimated is estimated according to the target Laplacian distribution to obtain the pose estimation result, and the size-independent pose of the object to be estimated is obtained based on the pose estimation result.

[0048] In some embodiments, the pose estimation of the object to be estimated is performed based on a preset convolutional network according to the target Laplacian distribution to obtain the pose estimation result, including: obtaining a 2D pixel coordinate map of the image of the object to be estimated; inputting the target Laplacian distribution and the 2D pixel coordinate map into the preset convolutional network to obtain the pose estimation result.

[0049] Understandably, since explicitly solving for pose using the target Laplacian distribution is very complex, this embodiment establishes a 2D-to-3D correspondence after estimating the Laplacian mixture model and uses a pre-defined convolutional network Φ as the PnP module. This allows for end-to-end training and efficient inference, enabling the PnP module built using the pre-defined convolutional network to solve the pose problem. The PnP module benefits from the Laplacian mixture model in two main ways: First, the PnP module can dynamically aggregate diverse object geometric information captured by dual information streams; second, the PnP module can evaluate the reliability of different information sources using variance information in the LMM, thus giving higher weights to more reliable information during pose solving. This method not only improves the accuracy of pose estimation but also enhances the model's robustness to changes in object shape and pose.

[0050] Specifically, embodiments of this application can process the image of the object to be estimated using existing image processing libraries (such as OpenCV) to obtain a 2D pixel coordinate map of the image of the object to be estimated, and then combine the target Laplacian distribution with the 2D pixel coordinate map. Input is fed into a pre-defined convolutional network to predict scale-independent rotation parameters. Translation parameters ,Right now:

[0051] in, For the predicted scale-independent rotation parameters, For the predicted scale-independent translation parameters, For the pre-defined convolutional network, The mean of the first Laplace distribution. Let V be the variance of the first Laplace distribution. The mean of the second Laplace distribution. Let Variance be the variance of the second Laplace distribution. This is a 2D pixel coordinate map of the image of the object to be estimated.

[0052] Further, in some embodiments, obtaining the size-independent pose of the object to be estimated based on the pose estimation result includes: acquiring the 3D size and 3D translation of the object to be estimated, and calculating the original diagonal length of the object's tight bounding box based on the 3D size; normalizing the size of the object to be estimated based on the 3D size and the original diagonal length of the object's tight bounding box to obtain the normalized 3D size of the object to be estimated; normalizing the translation of the object based on the 3D translation vector and the original diagonal length of the object's tight bounding box to obtain the normalized translation of the object to be estimated, and obtaining the predicted translation amount based on the normalized 3D translation of the object to be estimated.

[0053] Specifically, since the metric scale of the object to be estimated cannot be determined from a single RGB image, and due to the lack of depth information, predicting the translation and size of an object based solely on its visual appearance becomes particularly challenging. Therefore, embodiments of this application address this issue when considering an object with a size of... The object is first estimated based on its 3D dimensions. Calculate the original diagonal length of the object's tight bounding box, and then normalize the object's size based on its 3D dimensions and the original diagonal length of the tight bounding box. ; ; in, The normalized 3D dimensions of the object to be estimated. These are the normalized dimensions in the x, y, and z directions, respectively. The original diagonal length of the object's tight bounding box.

[0054] Furthermore, the average normalized size of this category is pre-calculated. And predict the difference in normalized size. ,in, The range of the difference is normalized to be near zero, which facilitates network prediction.

[0055] Furthermore, corresponding to the size of the object to be estimated, the translation of the object to be estimated... Also use the original diagonal length of the object's tight bounding box. Normalization is performed:

[0056] in, To estimate the translation of the object after normalization, These represent the components of the translation to be estimated in the x, y, and z directions at the absolute scale. The original diagonal length of the object's tight bounding box. The components of the normalized distance from the object to the camera in the x, y, and z directions are given.

[0057] Furthermore, similar to RGB-based work, 3D translation is divided into two components: the 2D position of the center of the projected object. and the normalized distance from the object to the camera .

[0058] Since the input image is cropped and scaled based on the detection results, the regression is for translation parameters that are independent of detection and scale. This means the model needs to be able to handle objects of different sizes and positions without requiring specific preprocessing of the input image. This strategy makes the model more general and robust, enabling it to accurately estimate the pose of objects in various real-world scenarios. The predicted translation is: ; ; ; ; in, For the predicted translation, , These are the relative offsets from the center of the 2D detection box along the x and y axes of the image plane, respectively. This is the relative value of the object's translation along the z-axis to the size of the detection box. The 2D position of the center of the projected object. The center coordinates of the detected 2D bounding box. The width of the detected 2D bounding box. The height of the detected 2D bounding box. To estimate the normalized distance from the object to the camera, The size of the detected 2D bounding box. The size of the scaled image of the object to be estimated.

[0059] Among them, the size of the 2D bounding box The larger of the width and height of the 2D bounding box, i.e. The size of the scaled image of the object to be estimated. The larger of the width and height of the scaled image of the object to be estimated is denoted as . ,in, The width of the scaled image of the object to be estimated. The height of the scaled image of the object to be estimated.

[0060] Furthermore, in some embodiments, obtaining the size-independent pose of the object based on the pose estimation result further includes: predicting a first rotation vector of the object to be estimated along a first direction axis and predicting a second rotation vector of the object along a second direction axis, wherein the first direction axis and the second direction axis are perpendicular; determining whether the object to be estimated has rotational symmetry; if the object to be estimated has rotational symmetry, supervising the second rotation vector when training the model of the object to be estimated.

[0061] Wherein, the first direction axis can be x The axis, the second direction axis can be y axis.

[0062] Specifically, regarding rotation R The representation predicts along the x First rotation vector along axis and along y The second rotation vector of the axis, (in ∈ ), where the first rotation vector and the second rotation vector correspond to the first two columns of the rotation matrix, respectively.

[0063] However, when dealing with objects that have rotational symmetry (such as bowls and bottles), these objects are symmetric about a specific axis, resulting in a first rotation vector. Ambiguity is introduced. Therefore, when training models of these rotationally symmetric objects, only the second rotation vector is considered. Supervise to avoid the first rotation vector The resulting ambiguity.

[0064] To enable those skilled in the art to more clearly and intuitively understand the category-level object pose estimation method based on the Laplace mixture model in the embodiments of this application, a detailed description is provided below in conjunction with specific embodiments.

[0065] Understandably, the purpose of this application is to solve the problem of RGB-based category-level object pose estimation. Specifically, given an RGB image containing objects from a predefined category set, the goal of this application is to detect all existing object instances in the scene and accurately estimate their 9-DOF poses, where the 9DoF pose includes 3-DOF rotations R∈SO(3), 3-DOF translations t∈R3, and 3-DOF dimensions s∈R3.

[0066] To address this, this application proposes a method for pose estimation using a Laplacian mixture model, which performs RGB-based category-level object pose estimation by modeling the object shape as a Laplacian mixture model.

[0067] Specifically, such as Figure 3 As shown, firstly, the existing object detector Mask R-CNN is used to crop out the object of interest as input. Then, two information streams are used to independently predict the two Laplacian distributions of the NOCS coordinate map. Next, a PnP solution method based on LMM is used to perform pose estimation using the estimated LMM. Finally, the PnP module calculates the size-independent pose.

[0068] According to the embodiment of this application, the category-level object pose estimation method based on the Laplacian mixture model predicts the first Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset general 3D information flow, and predicts the second Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset feature flow. The target Laplacian distribution is obtained based on the first and second Laplacian distributions. The object's pose is estimated using a preset convolutional network to obtain the pose estimation result, and the size-independent pose of the object is obtained based on the pose estimation result. Therefore, this method overcomes the influence of shape uncertainty and scale uncertainty caused by intra-category shape differences on pose estimation, achieving high-precision and high-accuracy prediction of category-level object poses, and is unaffected by scale uncertainty.

[0069] Next, referring to the accompanying drawings, a category-level object pose estimation device based on a Laplace mixture model, according to an embodiment of this application, is described.

[0070] Figure 4 This is a block diagram of a category-level object pose estimation device based on a Laplace mixture model according to an embodiment of this application.

[0071] like Figure 4 As shown, the category-level object pose estimation device 10 based on the Laplace mixture model includes: an acquisition module 100, a prediction module 200, and a pose estimation module 300.

[0072] The module includes an acquisition module 100, which acquires an image of the object to be estimated and predicts the NOCS coordinate map of the object based on the image; a prediction module 200, which predicts the first Laplacian distribution of the NOCS coordinate map based on a preset general 3D information flow and predicts the second Laplacian distribution of the NOCS coordinate map based on a preset feature flow, and obtains the target Laplacian distribution based on the first and second Laplacian distributions; and a pose estimation module 300, which estimates the pose of the object to be estimated based on a preset convolutional network and the target Laplacian distribution, obtains the pose estimation result, and obtains the size-independent pose of the object to be estimated based on the pose estimation result.

[0073] Further, in some embodiments, the prediction module 200 is used to: predict a first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow using a first preset loss function; and predict a second Laplace distribution of the NOCS coordinate map based on a preset feature flow using a second preset loss function; wherein the first preset loss function is: ; The second preset loss function is:

[0074] in, The first preset loss function is... Let V be the variance of the first Laplace distribution. This is the mask for the visible portion of the object to be estimated. This is a true NOCS coordinate map. The mean of the first Laplace distribution. The second preset loss function, Let Variance be the variance of the second Laplace distribution. The mean of the second Laplace distribution. and This is a hyperparameter.

[0075] Further, in some embodiments, the pose estimation module 300 is used to: acquire a 2D pixel coordinate map of an image of the object to be estimated; input the target Laplacian distribution and the 2D pixel coordinate map into a preset convolutional network to obtain a pose estimation result; wherein, the pose estimation result is: ; in, For the predicted scale-independent rotation parameters, For the predicted scale-independent translation parameters, For the pre-defined convolutional network, The mean of the first Laplace distribution. Let V be the variance of the first Laplace distribution. The mean of the second Laplace distribution. Let Variance be the variance of the second Laplace distribution. This is a 2D pixel coordinate map of the image of the object to be estimated.

[0076] Further, in some embodiments, the pose estimation module 300 is configured to: acquire the 3D dimensions and 3D translation of the object to be estimated, and calculate the original diagonal length of the object's tight bounding box based on the 3D dimensions; normalize the dimensions of the object to be estimated according to the 3D dimensions and the original diagonal length of the object's tight bounding box to obtain the normalized 3D dimensions of the object to be estimated; normalize the translation of the object according to the 3D translation vector and the original diagonal length of the object's tight bounding box to obtain the normalized translation of the object to be estimated, and obtain the predicted translation amount based on the normalized 3D translation of the object to be estimated, wherein the predicted translation amount is: ; ; ; ; in, For the predicted translation, , These are the relative offsets from the center of the 2D detection box along the x and y axes of the image plane, respectively. This is the relative value of the object's translation along the z-axis to the size of the detection box. The 2D position of the center of the projected object. The center coordinates of the 2D bounding box. The width of the 2D bounding box. The height of the 2D bounding box. To estimate the normalized distance from the object to the camera, The size of the 2D bounding box. The size of the scaled image of the object to be estimated.

[0077] Furthermore, in some embodiments, the pose estimation module 300 is also used to: predict a first rotation vector of the object to be estimated along a first direction axis and predict a second rotation vector of the object along a second direction axis, wherein the first direction axis and the second direction axis are perpendicular; determine whether the object to be estimated has rotational symmetry; if the object to be estimated has rotational symmetry, supervise the second rotation vector when training the model of the object to be estimated.

[0078] It should be noted that the foregoing explanation of the embodiment of the category-level object pose estimation method based on the Laplace mixture model also applies to the category-level object pose estimation device based on the Laplace mixture model in this embodiment, and will not be repeated here.

[0079] According to the embodiments of this application, a category-level object pose estimation device based on a Laplacian mixture model predicts a first Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset general 3D information flow, and predicts a second Laplacian distribution of the NOCS coordinate map of the object to be estimated based on a preset feature flow. A target Laplacian distribution is obtained based on the first and second Laplacian distributions. A pose estimation of the object to be estimated is performed based on a preset convolutional network to obtain a pose estimation result, and a size-independent pose of the object to be estimated is obtained based on the pose estimation result. Therefore, the influence of shape uncertainty and scale uncertainty caused by intra-category shape differences on pose estimation is overcome, enabling high-precision and high-accuracy prediction of category-level object poses, unaffected by scale uncertainty.

[0080] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0081] When the processor 502 executes the program, it implements the category-level object pose estimation method based on the Laplace mixture model provided in the above embodiments.

[0082] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.

[0083] The memory 501 is used to store computer programs that can run on the processor 502.

[0084] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0085] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0086] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0087] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0088] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described category-level object pose estimation method based on a Laplace mixture model.

[0089] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0090] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0091] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A category-level object pose estimation method based on a Laplace mixture model, characterized in that, Includes the following steps: Acquire an image of the object to be estimated, and predict the NOCS coordinate map of the object to be estimated based on the image of the object to be estimated; The first Laplace distribution of the NOCS coordinate map is predicted based on a preset general 3D information flow, and the second Laplace distribution of the NOCS coordinate map is predicted based on a preset feature flow. The target Laplace distribution is obtained based on the first Laplace distribution and the second Laplace distribution. Based on a preset convolutional network, the pose of the object to be estimated is estimated according to the target Laplacian distribution to obtain the pose estimation result, and the size-independent pose of the object to be estimated is obtained according to the pose estimation result. The method of predicting the first Laplace distribution of the NOCS coordinate graph based on a preset general 3D information flow and predicting the second Laplace distribution of the NOCS coordinate graph based on a preset feature flow includes: predicting the first Laplace distribution of the NOCS coordinate graph using a first preset loss function based on the preset general 3D information flow; and predicting the second Laplace distribution of the NOCS coordinate graph using a second preset loss function based on the preset feature flow. The first preset loss function is: ； The second preset loss function is: ； in, The first preset loss function is... Let V be the variance of the first Laplace distribution. This is the mask for the visible portion of the object to be estimated. This is a true NOCS coordinate map. The mean of the first Laplace distribution. The second preset loss function, Let Variance be the variance of the second Laplace distribution. The mean of the second Laplace distribution. and This is a hyperparameter.

2. The method according to claim 1, characterized in that, The pre-defined convolutional network estimates the pose of the object to be estimated based on the target Laplacian distribution, obtaining the pose estimation result, including: Obtain a 2D pixel coordinate map of the image of the object to be estimated; The target Laplacian distribution and the 2D pixel coordinate map are input into the preset convolutional network to obtain the pose estimation result; The pose estimation result is as follows: ； in, For the predicted scale-independent rotation parameters, For the predicted scale-independent translation parameters, For the pre-defined convolutional network, The mean of the first Laplace distribution. Let V be the variance of the first Laplace distribution. The mean of the second Laplace distribution. Let Variance be the variance of the second Laplace distribution. This is a 2D pixel coordinate map of the image of the object to be estimated.

3. The method according to claim 1, characterized in that, The step of obtaining the size-independent pose of the object to be estimated based on the pose estimation result includes: Obtain the 3D dimensions and 3D translation of the object to be estimated, and calculate the original diagonal length of the object tight bounding box of the object to be estimated based on the 3D dimensions of the object to be estimated. The size of the object to be estimated is normalized based on the 3D size of the object and the original diagonal length of the object's tight bounding box to obtain the normalized 3D size of the object to be estimated. The translation of the object is normalized based on its 3D translation vector and the original diagonal length of its tight bounding box, resulting in a normalized translation. The predicted translation amount is then obtained based on this normalized 3D translation, where the predicted translation amount is: ； ； ； ； in, For the predicted translation parameters, , These are the relative offsets from the center of the 2D detection box along the x and y axes of the image plane, respectively. This is the relative value of the object's translation along the z-axis to the size of the detection box. The 2D position of the center of the projected object. The center coordinates of the 2D bounding box. The width of the 2D bounding box. The height of the 2D bounding box. To estimate the normalized distance from the object to the camera, The size of the 2D bounding box. The size of the scaled image of the object to be estimated.

4. The method according to claim 1, characterized in that, The step of obtaining the size-independent pose of the object based on the pose estimation result further includes: A first rotation vector of the object to be estimated is predicted along a first direction axis, and a second rotation vector of the object is predicted along a second direction axis, wherein the first direction axis and the second direction axis are perpendicular; Determine whether the object to be estimated has rotational symmetry; If the object to be estimated has rotational symmetry, the second rotation vector is supervised when training the model of the object to be estimated.

5. A category-level object pose estimation device based on a Laplace mixture model, characterized in that, include: An acquisition module is used to acquire an image of the object to be estimated and predict the NOCS coordinate map of the object to be estimated based on the image of the object to be estimated. The prediction module is used to predict the first Laplace distribution of the NOCS coordinate map based on a preset general 3D information flow, and to predict the second Laplace distribution of the NOCS coordinate map based on a preset feature flow, and to obtain the target Laplace distribution based on the first Laplace distribution and the second Laplace distribution. The pose estimation module is used to estimate the pose of the object to be estimated based on a preset convolutional network and the target Laplacian distribution, to obtain the pose estimation result, and to obtain the size-independent pose of the object to be estimated based on the pose estimation result. The prediction module is used to: predict the first Laplace distribution of the NOCS coordinate map based on the preset general 3D information flow using a first preset loss function; and predict the second Laplace distribution of the NOCS coordinate map based on the preset feature flow using a second preset loss function. The first preset loss function is: ； The second preset loss function is: ； in, The first preset loss function is... Let V be the variance of the first Laplace distribution. This is the mask for the visible portion of the object to be estimated. This is a true NOCS coordinate map. The mean of the first Laplace distribution. The second preset loss function, Let Variance be the variance of the second Laplace distribution. The mean of the second Laplace distribution. and This is a hyperparameter.

6. The apparatus according to claim 5, characterized in that, The pose estimation module is used for: Obtain a 2D pixel coordinate map of the image of the object to be estimated; The target Laplacian distribution and the 2D pixel coordinate map are input into the preset convolutional network to obtain the pose estimation result; The pose estimation result is as follows: ； in, For the predicted scale-independent rotation parameters, For the predicted scale-independent translation parameters, For the pre-defined convolutional network, The mean of the first Laplace distribution. Let V be the variance of the first Laplace distribution. The mean of the second Laplace distribution. Let Variance be the variance of the second Laplace distribution. This is a 2D pixel coordinate map of the image of the object to be estimated.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the category-level object pose estimation method based on the Laplace mixture model as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the category-level object pose estimation method based on the Laplace mixture model as described in any one of claims 1-4.

Images