Object pose estimation method and apparatus, electronic device, and storage medium

Abstract

This disclosure provides an object pose estimation method, apparatus, electronic device, and storage medium. Based on a first binocular image containing the target object acquired by a binocular camera, the method determines the depth and mask information of the target object; based on the depth and mask information, it determines the initial pose of the target object; based on the 3D model information and the initial pose of the target object, it performs differentiable rendering to obtain first rendering information corresponding to the target object; based on first reference information extracted from the first binocular image and the first rendering information, iteratively optimizes the initial pose to obtain the first target pose of the target object, where the first reference information and the first rendering information have the same information type. Therefore, object pose estimation can be performed efficiently and accurately, and it can be applied to pose estimation of different types of objects in different scenarios, exhibiting strong generalization ability.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to an object pose estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Object pose estimation refers to determining the position and orientation of an object in three-dimensional space based on visual input (including images, point clouds, etc.). This technology is widely used in fields such as robotics, autonomous driving, and augmented reality.

[0003] In related technologies, pose estimation methods based on supervised deep learning can be used. However, this method heavily relies on the sample data used to train the deep learning model. The trained deep learning model cannot be applied to pose estimation of new objects that do not appear in the sample data, resulting in poor generalization. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide an object pose estimation method, apparatus, electronic device, and storage medium.

[0005] A first aspect of this disclosure provides an object pose estimation method, comprising:

[0006] Based on the first binocular image containing the target object acquired by the binocular camera, the depth information and mask information of the target object are determined;

[0007] Based on the depth information and mask information of the target object, the initial pose of the target object is determined;

[0008] Based on the three-dimensional model information of the target object and the initial pose, differentiable rendering is performed to obtain the first rendering information corresponding to the target object;

[0009] Based on the first reference information and the first rendering information extracted from the first binocular image, the initial pose is iteratively optimized to obtain the first target pose of the target object, wherein the first reference information and the first rendering information have the same information type.

[0010] A second aspect of this disclosure provides an object pose estimation apparatus, including a processor configured to:

[0011] Based on the first binocular image containing the target object acquired by the binocular camera, the depth information and mask information of the target object are determined;

[0012] Based on the depth information and mask information of the target object, the initial pose of the target object is determined;

[0013] Based on the three-dimensional model information of the target object and the initial pose, differentiable rendering is performed to obtain the first rendering information corresponding to the target object;

[0014] Based on the first reference information and the first rendering information extracted from the first binocular image, the initial pose is iteratively optimized to obtain the first target pose of the target object, wherein the first reference information and the first rendering information have the same information type.

[0015] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the object pose estimation method of any of the above embodiments.

[0016] A fourth aspect of this disclosure provides an electronic device, the electronic device comprising:

[0017] Memory is used to store processor-executable instructions;

[0018] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the object pose estimation method of any of the above embodiments.

[0019] In this embodiment, the depth and mask information of the target object can be determined based on a first binocular image containing the target object. Then, based on the depth and mask information, the initial pose of the target object can be determined. This yields an initial pose with minimal deviation from the true pose of the target object, improving the accuracy of the initial pose estimation. Subsequently, differentiable rendering can be performed based on the 3D model information and the initial pose of the target object to obtain first rendering information corresponding to the target object. Using first reference information extracted from the first binocular image of the same type as the first rendering information as a reference benchmark, the initial pose is iteratively optimized. This gradually reduces the difference between the first rendering information obtained from the optimized pose and the first reference information, resulting in the first target pose of the target object. This object pose estimation method does not require sample data and is unaffected by object type or sample data sampling. It can be applied to pose estimation of different types of objects in different scenarios, exhibiting strong generalization ability.

[0020] Furthermore, this method starts with an initial pose that deviates little from the true pose of the target object and performs iterative pose optimization. This avoids the inability to enter the convergence region due to a large initial pose deviation during the pose iteration optimization process, which helps to improve the accuracy of obtaining the first target pose. In addition, starting with an initial pose that deviates little from the true pose of the target object allows for faster iterative optimization to obtain the first target pose of the target object used to represent the true pose of the target object, which helps to improve the efficiency of pose iteration optimization and thus helps to improve the efficiency of object pose estimation. Therefore, object pose estimation can be completed efficiently and accurately. Attached Figure Description

[0021] Figure 1 This is an application diagram illustrating an exemplary application to which this disclosure applies.

[0022] Figure 2 This is a flowchart illustrating an exemplary embodiment of the object pose estimation method provided in this disclosure.

[0023] Figure 3 This is a schematic diagram of the process for determining the initial attitude provided in an exemplary embodiment of this disclosure.

[0024] Figure 4 This is a schematic diagram of the process for determining a set of candidate poses provided by an exemplary embodiment of this disclosure.

[0025] Figure 5 This is a schematic diagram of a process for determining a plurality of first candidate poses provided by an exemplary embodiment of the present disclosure.

[0026] Figure 6 This is a schematic diagram of a process for selecting an initial pose from a set of candidate poses, provided by an exemplary embodiment of this disclosure.

[0027] Figure 7 This is a schematic diagram of the process for determining the first target posture provided in an exemplary embodiment of this disclosure.

[0028] Figure 8 This is a schematic diagram of an architecture for object pose estimation provided in an exemplary embodiment of this disclosure.

[0029] Figure 9 This is a schematic flowchart illustrating the process of determining the second target pose of a target object corresponding to a second binocular image, provided by an exemplary embodiment of this disclosure.

[0030] Figure 10 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0031] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0032] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0033] Application Overview

[0034] In developing this disclosure, the inventors discovered that current object pose estimation methods include registration methods based on geometric features. This method first extracts the geometric features of the object whose pose needs to be estimated, uses these extracted geometric features as the object's initial pose, then matches the extracted geometric features with preset object model geometric features, and finally determines the object's actual pose based on the matching result.

[0035] However, this method requires high accuracy of the initial pose. If the initial pose is significantly off, the accuracy of the estimated object pose will be poor. Moreover, the effective extraction of geometric features is highly dependent on the texture and geometric details of the object surface. However, the surfaces of objects without texture or with weak texture lack obvious texture differences and distinguishable geometric details. Therefore, this method has limited ability to estimate the pose of objects without texture or with weak texture.

[0036] Therefore, there is currently another pose estimation method based on supervised deep learning, which trains a deep learning model with collected sample data and then uses the trained deep learning model to perform pose estimation.

[0037] However, the deep learning model trained in this way can only be used to estimate the pose of the object types included in the sample data, and cannot be applied to the pose estimation of new objects that have not appeared in the sample data, resulting in poor generalization.

[0038] Exemplary Applications

[0039] Object pose estimation can estimate the position and orientation of an object in three-dimensional space. The object pose estimation method provided in this disclosure can be applied to any scenario that requires object pose estimation.

[0040] For example, in autonomous driving and intelligent driving scenarios, the pose estimation method provided in this disclosure can be used to estimate the pose of objects around the vehicle to support driving decisions. In robot movement scenarios, the pose estimation method provided in this disclosure can be used to estimate the pose of objects around the robot to enable the robot to perform environmental modeling, obstacle avoidance, and navigation. In augmented reality scenarios, the pose estimation method provided in this disclosure can be used to estimate the pose of objects in the real scene to more accurately overlay virtual objects onto the real scene.

[0041] Figure 1 This is an exemplary application diagram of an embodiment of the present disclosure, wherein the object pose estimation device 110 includes at least one processor. The at least one processor can be any processor that supports deep learning algorithms, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), etc.

[0042] At least one processor in the object pose estimation device 110 can acquire binocular images of an object captured by the binocular camera 120, and perform object pose estimation based on the binocular images. The binocular camera 120 may include a left-eye camera and a right-eye camera, which can acquire images of the target object from different viewpoints to obtain binocular images. These binocular images include a left-eye image acquired by the left-eye camera and a right-eye image acquired by the right-eye camera.

[0043] In one possible implementation, during the object pose estimation process, the depth information and mask information of the target object can be determined based on the first binocular image of the target object acquired by the binocular camera 120. Based on the depth information and mask information, a preliminary pose estimation is performed to obtain an initial pose with a small deviation from the true pose of the target object. Then, a pose estimation method based on differentiable rendering is adopted. Starting from the initial pose, the first reference information of the object extracted from the binocular image is used as a reference to iteratively optimize the initial pose to obtain the first target pose of the target object. This can efficiently and accurately complete the pose estimation of the target object.

[0044] Exemplary methods

[0045] Figure 2This is a schematic flowchart of an object pose estimation method provided in an exemplary embodiment of this disclosure. Figure 2 As shown, the object pose estimation method includes the following steps:

[0046] S210: Based on the first binocular image containing the target object acquired by the binocular camera, determine the depth information and mask information of the target object.

[0047] In some implementations, a stereo camera can be used to acquire images of the target object to be estimated, resulting in a first stereo image including the target object. This first stereo image can be an image of the target object pre-acquired by the stereo camera, used for pose estimation based on the pre-acquired stereo image. Alternatively, the first stereo image can be an image acquired in real-time by the stereo camera, used for real-time pose estimation of the target object. Optionally, the first stereo image acquired by the stereo camera can be an RGB (Red, Green, Blue) image.

[0048] In this embodiment of the disclosure, the first binocular image acquired by the binocular camera includes a left-eye image and a right-eye image. The left-eye image and right-eye image can be a pair of images captured by the left-eye camera and right-eye camera, which are on the same horizontal line, targeting a target object. In one possible implementation, a binocular depth estimation method can be used to estimate the depth of the first binocular image to obtain the depth information of the target object in the first binocular image. The depth information of the target object can be used to represent the distance from the target object to the binocular camera.

[0049] Optionally, the binocular depth estimation method can be a deep learning-based method, such as inputting the first binocular image into a pre-trained depth estimation model to obtain the depth information of the binocular image. Alternatively, a stereo matching algorithm can be used to calculate the disparity corresponding to the target object in the left and right images included in the first binocular image, and then the depth information of the target object can be calculated based on the disparity. Of course, other binocular depth estimation methods can also be used to determine the depth information of the target object in the binocular image, and this embodiment does not limit this.

[0050] Optionally, the depth information of the target object may include at least one of the left-eye depth information of the target object corresponding to the left-eye image and the right-eye depth information of the target object corresponding to the right-eye image. The left-eye depth information is the vertical distance from the target object to the optical center of the left-eye camera, and the right-eye depth information is the vertical distance from the target object to the optical center of the right-eye camera.

[0051] In this embodiment of the disclosure, the mask information of the target object is used to characterize whether each pixel in the image belongs to the target object. Optionally, the mask information of the target object may include at least one of the left-eye mask information of the target object corresponding to the left-eye image and the right-eye mask information corresponding to the right-eye image. In one possible implementation, an image segmentation model can be used to segment the first binocular image to obtain the mask information corresponding to the target object in the first binocular image. For example, for the left-eye image, the left-eye image and the prompt of the target object can be input into the image segmentation model, so that the image segmentation model can segment the left-eye image based on the prompt information to obtain the left-eye mask information of the target object.

[0052] Optionally, the image segmentation model can be the SAM (Segment Anything Model), SAM2 model, Mask R-CNN model, etc., and this embodiment does not limit it.

[0053] S220 determines the initial pose of the target object based on its depth information and mask information.

[0054] The object pose estimation method provided in this embodiment includes a coarse estimation stage and a fine estimation stage. The coarse estimation stage (i.e., the stage corresponding to S210 to S220) determines the initial pose of the object, and the fine estimation stage (i.e., the stage corresponding to S230 to S240) optimizes the initial pose to obtain the final pose of the object.

[0055] In this embodiment, the object's pose is a 3D pose. Optionally, the 3D pose can be characterized by translation and rotation parameters. The translation parameters characterize the object's position and can optionally be represented by a three-dimensional vector (x, y, z). The rotation parameters characterize the object's orientation and can be described using a 6D continuous rotation representation method. This method uses six parameters, corresponding to the first two columns of a rotation matrix. Processing these six parameters yields the rotation matrix, which can then be used as the aforementioned rotation parameters. This representation method is continuous, facilitating parameter regression in subsequent pose iteration optimization processes. Optionally, the initial pose may include initial translation and initial rotation parameters.

[0056] In one possible implementation, multiple candidate poses of the target object in the camera coordinate system corresponding to the stereo camera can be generated based on depth information and mask information. Then, the initial pose of the target object is determined from among these multiple candidate poses. In this implementation, at least one of the position and orientation of the target object in the camera coordinate system differs for each candidate pose. In various embodiments of this disclosure, "multiple" means "at least two".

[0057] In one possible implementation, multiple candidate poses can be determined based on left eye depth information and left eye mask information, and / or multiple candidate poses can be determined based on right eye depth information and right eye mask information, and the initial pose of the target object can be determined from the multiple candidate poses.

[0058] S230: Based on the 3D model information and initial pose of the target object, perform differentiable rendering to obtain the first rendering information corresponding to the target object.

[0059] The 3D model information of the target object is used to represent the object's 3D geometry in space. Optionally, the 3D model information can be represented using a 3D mesh, which consists of multiple elements such as vertices, edges, and faces. A vertex is the basic building block of the mesh, and each vertex can include information such as 3D coordinates, color, and texture. An edge is a line segment connecting two vertices, and a face is a planar region composed of three or more vertices. Multiple vertices, edges, and faces form a closed 3D surface through specific connections, used to characterize the object's spatial structure.

[0060] In one possible implementation, the 3D model information of the target object can be obtained through a preset configuration file, which includes the 3D model information of the target object. For example, the preset configuration file may include a Mesh (i.e., a 3D mesh model) file. Of course, the preset configuration file may also include other files that can characterize the 3D model information of the target object, and this disclosure does not limit this.

[0061] In another possible implementation, the 3D model information of the target object can be generated based on binocular images. Optionally, at least one of the left and right images included in the first binocular image can be input into a pre-trained graph-based 3D model to obtain the 3D model information of the target object. For example, the graph-based 3D model can be a mixed-mode 3D model, a Trellis model, a SAM 3D model, etc.

[0062] In one possible implementation, after obtaining the 3D model information of the target object, the model structure included in the 3D model information can be simplified, and differentiable rendering can be performed based on the simplified 3D model information to improve subsequent rendering efficiency. In one example, the mesh included in the 3D model information can be simplified to reduce the number of vertices and faces of the mesh, so that the number of vertices and faces of the simplified mesh is less than a preset threshold. For example, a quadratic error metric (QEM) combined with a greedy edge-folding algorithm can be used to simplify the mesh. This algorithm mainly reduces the number of vertices of the mesh through edge-folding operations, thereby reducing the number of vertices and faces of the mesh. Specifically, during each fold, the geometric distortion caused by folding each edge can be quantified based on the QEM error, and the edge with the minimum folding cost can be greedily selected for folding. By iterating the above edge-folding process, the edge-folding process can be stopped when the number of vertices of the folded mesh reaches the target number of vertices or the QEM error reaches the maximum error threshold, resulting in a simplified mesh. Of course, other methods that can simplify 3D model information can also be used in this disclosure, and this disclosure does not limit them.

[0063] In the differentiable rendering process, based on the 3D model information and initial pose of the target object, the 3D geometric model of the target object can be projected onto a 2D observation space to obtain the observation information of the object in the 2D observation space, i.e., the first rendering information. That is, the first rendering information of the target object can include the observation information corresponding to the target object in the 2D space. Optionally, the first rendering information of the target object can include at least one of the rendered mask information, depth information, edge information, RGB information, and the rendered image of the target object. It is understood that the first rendering information of the target object is obtained through the initial pose and 3D model projection of the target object in 3D space; that is, the mask information and depth information in the first rendering information are combined with the initial pose and 3D model information of the target object in 3D space. The mask information and depth information in this first rendering information may be the same as or different from the depth information and mask information of the target object determined based on the first binocular image in step 210.

[0064] S240, based on the first reference information and the first rendering information extracted from the first binocular image, the initial pose is iteratively optimized to obtain the first target pose of the target object, wherein the information types of the first reference information and the first rendering information are the same.

[0065] The first target pose is the target pose for the first binocular image, that is, the pose of the target object at the moment the first binocular image is captured.

[0066] In one possible implementation, the iterative optimization process of the initial pose can use the first stereo image as a reference image. Specifically, the first reference information corresponding to the target object with the same information type as the first rendering information can be extracted from the first stereo image. For example, when the first rendering information includes mask information, the first reference information can include the mask information of the target object extracted from the first stereo image; when the first rendering information includes RGB information, the first reference information can include the RGB information of the target object extracted from the first stereo image.

[0067] Optimizing the initial pose based on the first reference information allows the first rendered information obtained by differentiable rendering based on the optimized pose to gradually approach the first reference information. Furthermore, after meeting the optimization stopping condition, the iterative optimization process of the initial pose can be stopped, and the pose obtained from the last optimization can be determined as the first target pose of the target object. Optionally, the optimization stopping condition may include at least one of the following: the number of iterations reaches a preset threshold, the function value of the loss function constructed based on the difference between the first reference information and the first rendered information converges, or the iteration optimization time reaches a preset time.

[0068] In this embodiment, the depth and mask information of the target object can be determined based on a first binocular image containing the target object. Then, based on the depth and mask information, the initial pose of the target object can be determined. This yields an initial pose with minimal deviation from the true pose of the target object, improving the accuracy of the initial pose estimation. Subsequently, differentiable rendering can be performed based on the 3D model information and the initial pose of the target object to obtain first rendering information corresponding to the target object. Using first reference information extracted from the first binocular image (of the same type as the first rendering information) as a reference benchmark, the initial pose is iteratively optimized, gradually reducing the difference between the rendering information obtained based on the optimized pose and the first reference information, thus obtaining the first target pose of the target object. This object pose estimation method does not require sample data and is unaffected by object type or sample data sampling. It can be applied to pose estimation of different types of objects in different scenarios, exhibiting strong generalization ability.

[0069] In this approach, starting with an initial pose that deviates slightly from the true pose of the target object, iterative pose optimization can avoid the problem of failing to enter the convergence region due to a large initial pose deviation during the pose iteration optimization process. This helps improve the accuracy of the first target pose of the target object obtained through iterative optimization. In addition, starting with an initial pose that deviates slightly from the true pose of the target object allows for faster iterative optimization to obtain the first target pose of the target object used to represent the true pose of the target object, which helps improve the efficiency of pose iteration optimization and thus the efficiency of object pose estimation. As a result, object pose estimation can be completed efficiently and accurately.

[0070] In one possible implementation, such as Figure 3 As shown, it illustrates a flowchart of determining the initial pose of a target object according to an exemplary embodiment of this disclosure. Figure 2 Based on the illustrated embodiment, operation S220 may include the following steps:

[0071] S2201 determines the set of candidate poses for the target object based on depth information and mask information.

[0072] The candidate pose set includes multiple candidate poses of the target object in the camera coordinate system corresponding to the stereo camera. Different candidate poses correspond to different rotation angles of the target object in the camera coordinate system.

[0073] In one possible implementation, the initial position of the target object in the camera coordinate system can be determined based on depth information and mask information. Based on the initial position, multiple rotation angles corresponding to the target object can be generated. Then, based on the initial position and multiple rotation angles, multiple candidate poses of the target object in the camera coordinate system corresponding to the stereo camera can be determined, thus obtaining a set of candidate poses.

[0074] In one possible implementation, multiple candidate poses corresponding to the target object can be generated based on the left-eye depth information and left-eye mask information determined by the left-eye image, to obtain a set of candidate poses, and / or, multiple candidate poses corresponding to the target object can be generated based on the right-eye depth information and right-eye mask information determined by the right-eye image, to obtain a set of candidate poses.

[0075] S2203 performs differentiable rendering based on each candidate pose and 3D model information to obtain the rendered image corresponding to each candidate pose.

[0076] For any candidate pose included in the candidate pose set, differentiable rendering can be performed by combining the 3D model information of the target object to obtain the corresponding rendered image.

[0077] Optionally, the process of obtaining the corresponding rendered image by differentiable rendering based on each candidate pose and 3D model information can be executed synchronously or asynchronously; this embodiment does not limit this.

[0078] S2205, based on each rendered image and the first stereo image, select the initial pose from the candidate pose set.

[0079] In one possible implementation, the initial pose can be obtained by selecting the pose that is closest to the real pose of the target object from multiple candidate poses based on the similarity between each rendered image and the first binocular image.

[0080] In this embodiment, during the process of determining the initial pose of the target object, multiple candidate poses can be generated based on the mask information and depth information of the target object. Through differentiable rendering, the rendered images corresponding to each candidate pose are obtained. Then, the initial pose is selected based on the rendered images and the first binocular image. Thus, the initial pose that is closer to the real pose of the target object can be selected from multiple candidate poses, which helps to improve the efficiency and accuracy of obtaining the first target pose of the target object through subsequent iterative optimization based on the initial pose.

[0081] Optionally, the depth information of the target object determined based on the first binocular image may include the left-eye depth information of the target object corresponding to the left eye image in the first binocular image, and / or the right-eye depth information of the target object corresponding to the right eye image in the first binocular image. Optionally, the mask information of the target object determined based on the first binocular image may include the left-eye mask information of the target object corresponding to the left eye image in the first binocular image, and / or the right-eye mask information of the target object corresponding to the right eye image in the first binocular image.

[0082] In one possible implementation, Figure 3 Based on the illustrated embodiment, the above operation S2201 may include at least one of the following steps S2210 and S2212, namely, the candidate pose set may include multiple first candidate poses and / or multiple second candidate poses. Figure 4 The diagram shows the case where operation S2201 includes S2210 and S2212. S2210 and S2212 can be executed in parallel or sequentially. (This embodiment does not limit the execution order of S2210 and S2212.)

[0083] S2210, based on the left eye depth information and left eye mask information of the target object corresponding to the left eye image in the first binocular image, determine multiple first candidate poses of the target object.

[0084] The left-eye image was captured by the left-eye camera of the binocular camera system.

[0085] In one possible implementation, the first initial position of the target object can be determined based on the left eye depth information and the left eye mask information, and multiple first candidate poses of the target object can be generated with the first initial position as the center.

[0086] In one exemplary embodiment, such as Figure 5 As shown, it illustrates a flowchart of the process for determining multiple first candidate poses of a target object, which may include the following operations:

[0087] S2210a determines the first initial position of the target object based on the left eye depth information and the left eye mask information.

[0088] In one possible implementation, based on the left-eye depth information and the intrinsic parameters of the left-eye camera, the two-dimensional pixels belonging to the target object indicated in the left-eye mask information can be back-projected into three-dimensional space to obtain a three-dimensional point cloud corresponding to the target object in the camera coordinate system. Specifically, for any two-dimensional pixel belonging to the target object, based on its pixel coordinates and depth, the two-dimensional pixel can be back-projected into three-dimensional space to obtain the three-dimensional coordinates of the corresponding three-dimensional point. The three-dimensional points corresponding to all the two-dimensional pixels constitute the three-dimensional point cloud corresponding to the target object. After determining the three-dimensional point cloud corresponding to the target object, the first initial position of the target object can be determined based on the three-dimensional coordinates of each three-dimensional point in the three-dimensional point cloud.

[0089] In one example, the mean operation can be performed on each of the three-dimensional coordinates included in the three-dimensional point cloud to obtain the geometric mean position of the target object in the camera coordinate system, and this geometric mean position can be determined as the first initial position of the target object.

[0090] In another example, the minimum bounding box algorithm can be used to calculate the minimum bounding box of the 3D point cloud of the target object, and the center position of the minimum bounding box can be determined as the first initial position of the target object.

[0091] S2210b, based on the first initial position, performs camera position sampling to obtain multiple virtual camera positions.

[0092] After determining the initial position of the target object, multiple rotation angles corresponding to the target object can be determined based on this initial position. Optionally, multiple virtual camera orientations can be sampled, with different virtual camera orientations corresponding to different rotation angles of the target object.

[0093] In one possible implementation, Fibonacci sampling can be used to uniformly sample multiple positions on a sphere centered at a first initial position, resulting in multiple virtual camera positions. For example, the `fibonacci_sphere_sampling` function can be called to generate multiple virtual camera positions centered at the first initial position.

[0094] S2210c performs rotation angle sampling for any virtual camera position to obtain the orientation of multiple virtual cameras corresponding to the virtual camera position.

[0095] In one possible implementation, for any virtual camera position, multiple rotation angles can be uniformly sampled within a preset angle range to obtain multiple virtual camera orientations.

[0096] For example, starting with the initial position, generate j virtual camera positions. Then, for any virtual camera position, sample k rotation angles within a preset angle range to obtain j*k virtual camera orientations, where j and k are positive integers, and the preset angle range can be [0, 2]. .

[0097] S2210d determines multiple rotation angles corresponding to the target object based on the orientation of multiple virtual cameras corresponding to the positions of multiple virtual cameras.

[0098] Different virtual camera orientations correspond to different rotation angles of the target object. One virtual camera orientation can correspond to one rotation angle of the target object. Combining the above example, j*k virtual camera orientations can be used to determine the rotation angles of j*k target objects.

[0099] Optionally, the virtual camera orientation can be converted into the rotation angle of the target object. Alternatively, the sampled virtual camera orientation can be treated as multiple rotation angles of the target object.

[0100] S2210e, based on multiple rotation angles and a first initial position, obtains multiple first candidate poses.

[0101] For any given rotation angle and a first initial position, a first candidate pose can be obtained by combining them. By combining multiple rotation angles with the first initial position, multiple first candidate poses can be obtained.

[0102] S2212, based on the right eye depth information and right eye mask information of the target object corresponding to the right eye image in the first binocular image, determine multiple second candidate poses of the target object.

[0103] The right-eye image was captured by the right-eye camera of the binocular camera system.

[0104] In one possible implementation, a second initial position corresponding to the target object can be determined based on the right-eye depth information and right-eye mask information corresponding to the right-eye image. Multiple second candidate poses of the target object are then generated centered on this second initial position. The specific method for determining the multiple second candidate poses of the target object based on the right-eye depth information and right-eye mask information corresponding to the right-eye image can refer to the method described above for determining the multiple first candidate poses of the target object based on the left-eye depth information and left-eye mask information corresponding to the left-eye image, and will not be repeated here.

[0105] In this embodiment, multiple first candidate poses based on the left eye image and multiple second candidate poses based on the right eye image can be determined based on the depth information and mask information corresponding to the left eye image and the right eye image, respectively, to obtain a candidate pose set composed of multiple first candidate poses and multiple second candidate poses. This can improve the pose coverage of the candidate pose set and help improve the accuracy of selecting the initial pose from the candidate pose set.

[0106] Furthermore, in this embodiment, during the process of determining multiple candidate poses based on depth information and mask information, an initial position can be determined based on depth information and mask information. With the initial position as the center, multiple virtual camera positions are uniformly sampled. Then, for each virtual camera position, multiple virtual camera orientations are uniformly sampled. This allows for obtaining multiple rotation angles of the target object corresponding to the multiple virtual camera orientations. This can cover a variety of candidate poses that may be close to the real pose of the target object, which helps to improve the accuracy of selecting the initial pose from the candidate pose set.

[0107] Optionally, the candidate pose set includes multiple first candidate poses and / or multiple second candidate poses.

[0108] When the candidate pose set includes multiple first candidate poses, a first target rendering image is determined from the first rendering images based on the similarity between the first rendered image and the left eye image corresponding to each first candidate pose, and the first candidate pose corresponding to the first target rendering image is determined as the initial pose. In one example, the similarity of the first target rendering image is higher than the similarity of other first rendered images, that is, the first rendered image with the highest similarity to the left eye image can be determined as the first target rendering image. In another example, any first rendered image with a similarity higher than a first similarity threshold can be determined as the first target rendering image. Then, the first candidate pose corresponding to the first target rendering image can be determined as the initial pose. The specific method for calculating the similarity between the first rendered image and the left eye image can be referred to the implementation of step S2250a in the following embodiments.

[0109] When the candidate pose set includes multiple second candidate poses, a second target rendering image is determined from the second rendered images based on the similarity between the second rendered image and the right eye image corresponding to each second candidate pose, and the candidate pose corresponding to the second target rendering image is determined as the initial pose. In one example, the second rendered image with the highest similarity to the right eye image can be determined as the second target rendering image. In another example, any second rendered image with a similarity higher than a second similarity threshold can be determined as the second target rendering image.

[0110] In one possible implementation, when the candidate pose set includes multiple first candidate poses and multiple second candidate poses, such as Figure 6 As shown, a flowchart illustrating the process of selecting an initial pose from a set of candidate poses provided by an exemplary embodiment of this disclosure is illustrated. The above operation S2205 may include the following steps:

[0111] S2250, based on the first rendered image and left eye image corresponding to each first candidate pose, selects a subset of first poses from multiple first candidate poses.

[0112] In one possible implementation, for any first candidate pose, the similarity between the first rendered image and the left eye image obtained based on the first candidate pose can be calculated, and a subset of first poses can be selected from multiple first candidate poses based on the similarity, wherein the similarity of the first candidate poses included in the subset of first poses is higher than the similarity of the other first candidate poses in the multiple first candidate poses.

[0113] In one possible implementation, the process of selecting a subset of first poses from a plurality of first candidate poses includes the following steps:

[0114] S2250a, calculate the first similarity between each first rendered image and the left eye image respectively.

[0115] In one possible implementation, preset features can be extracted from the first rendered image and the left eye image respectively, the similarity between the preset features can be calculated, and the similarity between the preset features can be determined as the first similarity between the first rendered image and the left eye image.

[0116] Optionally, the preset feature can be the edge information of the target object. For any first rendered image, the cosine similarity of the edge information of the target object included in the first rendered image and the left-eye image can be calculated to obtain the first similarity.

[0117] In one example, the moment information of the target object's edges can be extracted from the first rendered image and the left eye image, and the cosine similarity between the edge moments can be calculated to obtain the first similarity. Optionally, the edge moments can be normalized central moments, which can be the sum of the central moments and the Hu moments.

[0118] S2250b: Based on each first similarity, determine the first candidate pose that meets the first similarity condition, and obtain the first pose subset.

[0119] In one possible implementation, after determining the first similarity corresponding to each first rendered image, the first candidate pose with higher similarity can be selected from multiple first candidate poses based on the first similarity.

[0120] Optionally, a first candidate pose with high similarity can be selected by using a first similarity criterion. The first similarity criterion includes that the first similarity belongs to the top N highest similarities among multiple first similarities, where N is a positive integer, and / or that the first similarity is higher than the first similarity threshold.

[0121] In one example, the first similarity threshold can be a preset threshold, such as 80%, which can determine the first candidate pose corresponding to the first similarity that is higher than the first similarity threshold as the first candidate pose that meets the first similarity condition.

[0122] In another example, N can be a preset positive integer, for example, 10. After calculating the first similarity corresponding to each first rendered image, the multiple first similarities can be sorted to obtain the top N first similarities with the highest similarity. The first candidate pose corresponding to each of the top N first similarities with the highest similarity is determined as the first candidate pose that meets the first similarity condition.

[0123] In another example, multiple first similarities above the first similarity threshold can be filtered out firstly. Among the multiple first similarities filtered out, the N first similarities with the highest similarity are selected, and the first candidate poses corresponding to each of the N first similarities are determined as the first candidate poses that meet the first similarity condition.

[0124] S2252, based on the second rendered image and right eye image corresponding to each second candidate pose, select a subset of second poses from multiple second candidate poses.

[0125] In one possible implementation, for any second candidate pose, the similarity between the second rendered image and the right eye image can be calculated, and a subset of second poses can be selected from multiple second candidate poses based on the similarity, wherein the similarity of the second candidate poses included in the subset of second poses is higher than the similarity of the other second candidate poses in the multiple second candidate poses.

[0126] In one possible implementation, the process of selecting a subset of second poses from a plurality of second candidate poses includes the following steps:

[0127] S2252a, calculate the second similarity between each second rendered image and the right eye image respectively.

[0128] S2252b: Based on each second similarity, determine the second candidate poses that meet the second similarity conditions, and obtain the second pose subset.

[0129] Optionally, the second similarity condition includes the second similarity belonging to the top M highest similarities among multiple second similarities, where M is a positive integer, and / or the second similarity being higher than the second similarity threshold.

[0130] The specific implementation methods of S2252a-S2252b can be referred to the above operations S2250a-S2250b, and will not be repeated here. It should be noted that, if N and M are preset positive integers, they can be the same or different. The first similarity threshold and the second similarity threshold can be the same or different.

[0131] S2254, select the initial pose from the same poses included in the first pose subset and the second pose subset.

[0132] After obtaining the first and second attitude subsets, we can determine the common attitudes included in both subsets. That is, common attitudes refer to attitudes that belong to both the first and second attitude subsets simultaneously.

[0133] Optionally, the first and second pose subsets may include at least one identical pose. If a pose is included, that pose may be designated as the initial pose.

[0134] In the case of multiple identical poses, any one of the identical poses can be used as the initial pose.

[0135] Alternatively, in another possible implementation, the initial pose can be selected from multiple identical poses based on the similarity corresponding to the same pose. This process may include the following operations:

[0136] S2254a, when the first pose subset and the second pose subset include at least two identical poses, for any identical pose, a similarity score corresponding to the identical pose is determined based on the first similarity and the second similarity corresponding to the identical pose.

[0137] The same pose belongs to both the first pose subset and the second pose subset, that is, it belongs to both the first candidate pose and the second candidate pose, and has a first similarity and a second similarity. The sum of the first similarity and the second similarity corresponding to the same pose can be determined as the similarity score of the same pose, or the average of the first similarity and the second similarity corresponding to the same pose can be determined as the similarity score of the same pose. This embodiment does not limit this.

[0138] S2254b identifies the same pose with the highest similarity score as the initial pose.

[0139] Among them, the same pose with the highest similarity score is the pose in the candidate pose set that is closest to the true pose of the target object, and it can be determined as the initial pose.

[0140] In this embodiment, the candidate pose set includes multiple first candidate poses and multiple second candidate poses. Firstly, a subset of first poses and a subset of second poses with high similarity can be selected from the first candidate poses and the second candidate poses, respectively. Then, an initial pose is selected from the same poses included in the first pose subset and the second pose subset, so that the initial pose is closer to the true pose of the target object. This can avoid the problem of not being able to enter the convergence region in the subsequent iterative optimization process and improve the accuracy of pose estimation.

[0141] In one possible implementation, such as Figure 7 As shown, it illustrates a flowchart of iterative optimization of an initial pose provided by an exemplary embodiment of this disclosure. Figure 2 Based on the illustrated embodiment, operation S240 may include the following steps:

[0142] S2401, Based on the difference between modal information of the same type in the first reference information and the first rendering information, determine the rendering loss.

[0143] In this embodiment of the disclosure, the first reference information and the first rendering information include information of the same type. Based on the modal information of the same type, the rendering loss can be determined based on the differences between the modal information of the same type. Optionally, the rendering loss can be the sum of the losses corresponding to modal information of various information types.

[0144] Optionally, the modal information includes at least one of depth information, mask information, RGB information, and edge information. In one example, if the modal information includes only one type of information, the rendering loss can be the loss corresponding to that one type of information. For example, if the modal information is depth information, the rendering loss can be the depth loss determined based on the depth information in the first reference information and the first rendering information. In another example, if the modal information includes multiple types of information, the rendering loss can be the sum of the losses corresponding to the modal information of the same type. For example, if the modal information includes depth information, mask information, RGB information, and edge information, the rendering loss can be the weighted sum of the depth loss determined based on the depth information, the mask loss determined based on the mask information, the RGB loss determined based on the RGB information, and the edge loss determined based on the edge information. For example, the rendering loss is shown in the following formula (1): (1)

[0145] in, Indicates deep loss. The weights are the weights corresponding to the depth loss. Indicates mask loss. The weights corresponding to the mask loss, Indicates RGB loss, The weights corresponding to RGB loss, Indicates marginal loss. is the weight corresponding to the edge loss, and L is the rendering loss.

[0146] Optionally, a preset loss function can be used to calculate the loss between modal information. For example, the preset loss function can be the L1 loss function, L2 loss function, FocalDice loss function, etc. When it is necessary to calculate the loss between multiple modal information, the preset loss function used for different modal information can be the same; for example, the L1 loss function can be used for all modal information. Alternatively, the preset loss function used for different modal information can also be different. For example, the rendering loss corresponding to depth information, RGB information, and edge information can be calculated using the L1 loss function, while the rendering loss corresponding to mask information can be calculated using the FocalDice loss function.

[0147] In one possible implementation, the first rendering information may include at least one of left-eye rendering information and right-eye rendering information. Optionally, differentiable rendering can be performed based on the 3D model information of the target object, its initial pose, and the camera parameters corresponding to the left-eye camera in the stereo camera setup to obtain the left-eye rendering information of the target object. Optionally, differentiable rendering can be performed based on the 3D model information of the target object, its initial pose, and the camera parameters corresponding to the right-eye camera in the stereo camera setup to obtain the right-eye rendering information of the target object.

[0148] When the first rendering information includes left-eye rendering information, the left-eye reference information of the first binocular image can be compared with the modal information of the same type in the left-eye rendering information. A first loss is determined based on the difference, and then a rendering loss is determined based on the first loss. That is, when the first rendering information includes left-eye rendering information, the first reference information includes modal information extracted from the left-eye image that is of the same type as the information in the left-eye rendering information. Optionally, the modal information included in the left-eye rendering information may include at least one of depth information, mask information, RGB information, and edge information, and the corresponding first loss may include at least one of depth loss, mask loss, RGB loss, and edge loss.

[0149] When the first rendering information includes right-eye rendering information, the right-eye reference information of the first binocular image can be compared with the modal information of the same type in the right-eye rendering information. A second loss is determined based on the difference, and then the rendering loss is determined based on the second loss. That is, when the first rendering information includes right-eye rendering information, the right-eye reference information includes modal information extracted from the right-eye image that is of the same type as the information in the right-eye rendering information. Optionally, the modal information included in the right-eye rendering information may include at least one of depth information, mask information, RGB information, and edge information, and the corresponding second loss may include at least one of depth loss, mask loss, RGB loss, and edge loss.

[0150] When the first rendering information includes left-eye rendering information and right-eye rendering information, the rendering loss can be determined based on the first loss and the second loss. Optionally, the rendering loss can be the sum of the first loss and the second loss.

[0151] S2403, based on rendering loss, iteratively optimizes the initial pose to obtain the first target pose of the target object.

[0152] Optionally, when iteratively optimizing the initial pose based on the rendering loss, the pose can be optimized using stochastic gradient descent (SGD). After one iteration, differentiable rendering can be performed again based on the iterated pose to obtain the updated first rendering information. Then, the rendering loss is updated based on the updated first rendering information, and the pose is optimized again based on the updated rendering loss. In the process of optimizing the pose using SGD, the loss gradient can be calculated based on the rendering loss, and the pose parameters can be updated along the gradient descent direction. The magnitude of parameter adjustment along the gradient descent direction can be controlled by the learning rate. The change in pose parameters can be determined based on the learning rate and the loss gradient. Optionally, in the iterative optimization process, the learning rate can be decayed exponentially, as shown in the following formula (2): (2)

[0153] in, This represents the learning rate at the current step. The initial preset learning rate, The iteration number is denoted as iterator, and iters is the decay step size.

[0154] In one possible implementation, the iterative optimization of the initial pose can be stopped after the rendering loss value converges, thus obtaining the first target pose of the target object. Optionally, the convergence of the rendering loss value can be determined when the loss convergence condition is met. The loss convergence condition includes at least one of the following: the fluctuation value of the rendering loss value obtained in consecutive iterations is less than a preset threshold; the rendering loss value obtained in consecutive iterations no longer decreases; and the learning rate decays to within a preset threshold and the loss value no longer decreases.

[0155] In this embodiment, the rendering loss can be determined based on multimodal information. The gradient vectors generated by the losses of various modal information can be in the same gradient space. When updating the pose based on gradient descent, multiple heterogeneous features can be optimized in the same gradient space. That is, multi-scale optimization can be performed in the gradient space, which helps to broaden the convergence region. Moreover, even if the information of a certain modality is missing, the pose can still be optimized based on the loss of other modalities, which can avoid the problem of getting trapped in local optima and help improve the accuracy of the first target pose of the target object iterated to.

[0156] Furthermore, in this embodiment, a first loss can be determined based on the left-eye rendering information and the left-eye image, and a second loss can be determined based on the right-eye rendering information and the right-eye image. The sum of the first loss and the second loss is determined as the rendering loss. The initial pose can be iteratively optimized using both the left-eye image and the right-eye image as references, which helps to improve the accuracy of the first target pose of the target object obtained through iteration.

[0157] In one possible implementation, the object pose estimation process can be as follows: Figure 8 As shown, the rendering process includes two stages. In the first stage, based on the stereo images of the target object captured by the stereo camera, the depth information and mask information of the target object are extracted. Specifically, the RGB information corresponding to the stereo image and the target object's prompt are input into the image segmentation model to obtain the target object's mask information. Based on the target object's depth and mask information, a set of candidate poses can be determined. For any candidate pose in the set, differentiable rendering is performed using a simplified mesh of the target object to obtain the rendered image corresponding to each candidate pose. The initial pose is selected based on the similarity between the rendered image corresponding to each candidate pose and the stereo image.

[0158] Based on the first stage of rendering, the initial pose can be determined. In the second stage of rendering, differentiable rendering can be performed based on the initial pose and the simplified mesh of the target object. The initial pose is iteratively optimized using a render-and-compare approach, that is, the initial pose is iteratively optimized based on the rendering loss determined by the first rendering information obtained from the rendering and the first reference information extracted from the stereo image. The rendering loss can be at least one of depth loss, mask loss, RGB loss, and edge loss.

[0159] In one possible implementation, the first binocular image can be a pair of binocular images from a binocular image sequence captured by a binocular camera. The binocular image sequence includes multiple pairs of binocular images of the target object captured by the binocular camera at different times. If the second binocular image in the binocular image sequence is a binocular image captured at the time following the acquisition time corresponding to the first binocular image in the sequence, then... Figure 9 As shown, the process of determining the pose of the target object in the second binocular image may include the following steps:

[0160] S260: After obtaining the first target pose of the target object, perform differentiable rendering based on the three-dimensional model information of the target object and the first target pose to obtain the second rendering information corresponding to the target object.

[0161] When the second binocular image is a binocular image acquired at the next moment after the acquisition time corresponding to the first binocular image, the first target pose determined based on the first binocular image can be used as the initial pose for estimating the target object pose in the second binocular image. Based on the first target pose and the three-dimensional model information of the target object, the second rendering information of the target object is obtained.

[0162] In one possible implementation, if the interval between the acquisition time of the first binocular image and the acquisition time of the second binocular image is less than a preset duration threshold, the first target pose determined based on the first binocular image can be used as the initial pose for estimating the target object pose in the second binocular image to avoid large errors. If the interval between the acquisition time of the first binocular image and the acquisition time of the second binocular image is greater than the preset duration threshold, the initial pose corresponding to the second binocular image can be determined using the method described in the above embodiments for determining the initial pose corresponding to the first binocular image.

[0163] S270, for the second stereo image included in the stereo image sequence, based on the second reference information and the second rendering information extracted from the second stereo image, the pose of the first target is iteratively optimized to obtain the second target pose of the target object corresponding to the second stereo image.

[0164] Optionally, the second reference information and the second rendering information have the same information type. Specifically, the method of iteratively optimizing the first target pose to obtain the second target pose of the target object based on the second reference information and the second rendering information extracted from the second binocular image can refer to the method in the above embodiment of iteratively optimizing the initial pose to obtain the first target pose based on the first reference information and the first rendering information, and will not be repeated here.

[0165] Optionally, the first binocular image can be the first pair of binocular images included in the binocular image sequence. For the first pair of binocular images, the first target pose of the first binocular image can be determined using the method provided in the above embodiments, which is the pose of the target object in the first pair of binocular images. Then, for the second binocular image following the first binocular image, the first target pose can be used as the initial pose, and the first target pose can be iteratively optimized to obtain the second target pose of the second binocular image. Then, the second target pose can be used as the initial pose of the third binocular image acquired at the next moment corresponding to the acquisition time of the second binocular image, and the pose of the target object in the third binocular image can be estimated. Thus, by repeating this process, the poses of the target objects corresponding to each of the multiple pairs of binocular images included in the binocular image sequence can be obtained.

[0166] In this embodiment, the first target pose of the first binocular image is used as the initial pose of the second binocular image to estimate the pose of the target object in the second binocular image. Thus, time-based object pose estimation can be extended to achieve accurate estimation of object pose sequences.

[0167] Any object pose estimation method provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any object pose estimation method provided in this disclosure can be executed by a processor, such as by a processor executing any object pose estimation method mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0168] Exemplary device

[0169] This disclosure also provides an object pose estimation apparatus for implementing the object pose estimation method of any embodiment of this disclosure. In an optional embodiment of this disclosure, the object pose estimation includes a processor, which is configured to:

[0170] Based on the first binocular image containing the target object acquired by the binocular camera, the depth information and mask information of the target object are determined.

[0171] The initial pose of the target object is determined based on its depth information and mask information.

[0172] Differentiable rendering is performed based on the 3D model information and initial pose of the target object to obtain the first rendering information corresponding to the target object;

[0173] Based on the first reference information and the first rendering information extracted from the first binocular image, the initial pose is iteratively optimized to obtain the first target pose of the target object. The information types of the first reference information and the first rendering information are the same.

[0174] In one possible implementation, when the processor determines the initial pose of the target object based on its depth information and mask information, it is further configured to:

[0175] Based on depth information and mask information, a set of candidate poses for the target object is determined. The set of candidate poses includes multiple candidate poses of the target object in the camera coordinate system corresponding to the stereo camera. Different candidate poses correspond to different rotation angles of the target object in the camera coordinate system.

[0176] Differentiable rendering is performed based on each candidate pose and 3D model information to obtain the rendered image corresponding to each candidate pose.

[0177] Based on each rendered image and the first stereo image, the initial pose is selected from the candidate pose set.

[0178] In one possible implementation, the depth information includes the left-eye depth information of the target object corresponding to the left eye image in the first binocular image and / or the right-eye depth information of the target object corresponding to the right eye image in the first binocular image, and the mask information includes the left-eye mask information of the target object corresponding to the left eye image in the first binocular image and / or the right-eye mask information of the target object corresponding to the right eye image in the first binocular image.

[0179] When the processor determines the candidate pose set of the target object based on depth information and mask information, it is further configured as follows:

[0180] Based on the left-eye depth information and left-eye mask information of the target object corresponding to the left-eye image in the first binocular image, multiple first candidate poses of the target object are determined. The left-eye image is acquired by the left-eye camera of the binocular camera.

[0181] And / or, based on the right eye depth information and right eye mask information of the target object corresponding to the right eye image in the first binocular image, determine multiple second candidate poses of the target object. The candidate pose set includes multiple first candidate poses and / or multiple second candidate poses. The right eye image is acquired by the right eye camera of the binocular camera.

[0182] When the processor selects initial poses from the candidate pose set based on each rendered image and the first stereo image, it is further configured as follows:

[0183] When the candidate pose set includes multiple first candidate poses, based on the similarity between the first rendered image and the left eye image corresponding to each first candidate pose, the first target rendered image is determined from the first rendered image, and the first candidate pose corresponding to the first target rendered image is determined as the initial pose.

[0184] Alternatively, if the candidate pose set includes multiple second candidate poses, based on the similarity between the second rendered image and the right eye image corresponding to each second candidate pose, a second target rendered image is determined from the second rendered image, and the second candidate pose corresponding to the second target rendered image is determined as the initial pose.

[0185] Alternatively, when the candidate pose set includes multiple first candidate poses and multiple second candidate poses, a first pose subset is selected from the multiple first candidate poses based on the first rendered image and left eye image corresponding to each first candidate pose; a second pose subset is selected from the multiple second candidate poses based on the second rendered image and right eye image corresponding to each second candidate pose; and an initial pose is selected from the same poses included in the first pose subset and the second pose subset, where the same pose refers to a pose that belongs to both the first pose subset and the second pose subset.

[0186] In one possible implementation, when the processor filters a subset of first poses from multiple first candidate poses based on the first rendered image and the left eye image corresponding to each first candidate pose, it is further configured to:

[0187] Calculate the first similarity between each first rendered image and the left eye image;

[0188] Based on each first similarity, a first candidate pose that meets the first similarity condition is determined, and a first pose subset is obtained. The first similarity condition includes that the first similarity belongs to the top N highest similarities among multiple first similarities, where N is a positive integer, and / or that the first similarity is higher than the first similarity threshold.

[0189] When the processor selects a subset of second poses from multiple candidate poses based on the second rendered image and the right eye image corresponding to each candidate pose, it is further configured as follows:

[0190] Calculate the second similarity between each second rendered image and the right eye image;

[0191] Based on each second similarity, a second candidate pose that meets the second similarity condition is determined, and a subset of second poses is obtained. The second similarity condition includes that the second similarity belongs to the top M highest similarities among multiple second similarities, where M is a positive integer, and / or that the second similarity is higher than the second similarity threshold.

[0192] When the processor filters the initial poses from the same poses included in the first pose subset and the second pose subset, it is further configured to:

[0193] If the first pose subset and the second pose subset include at least two identical poses, for any identical pose, the similarity score corresponding to the identical pose is determined based on the first similarity and the second similarity corresponding to the identical pose.

[0194] The pose with the highest similarity score is selected as the initial pose.

[0195] In one possible implementation, when the processor determines multiple first candidate poses of the target object based on the left-eye depth information and left-eye mask information of the target object corresponding to the left-eye image in the first binocular image, it is further configured to:

[0196] Based on the left eye depth information and left eye mask information, determine the first initial position of the target object;

[0197] Based on the first initial position, camera position sampling is performed to obtain multiple virtual camera positions;

[0198] For any virtual camera position, perform rotation angle sampling to obtain multiple virtual camera orientations corresponding to the virtual camera position;

[0199] Based on the orientation of multiple virtual cameras corresponding to the positions of multiple virtual cameras, determine multiple rotation angles corresponding to the target object;

[0200] Based on multiple rotation angles and a first initial position, multiple first candidate poses are obtained.

[0201] In one possible implementation, when the processor calculates the first similarity between each first rendered image and the left-eye image, it is further configured to:

[0202] For any first rendered image, calculate the cosine similarity of the edge information of the target object included in the first rendered image and the left eye image respectively, and obtain the first similarity.

[0203] In one possible implementation, when the processor iteratively optimizes the initial pose based on the first reference information and the first rendering information extracted from the first binocular image to obtain the first target pose of the target object, it is further configured as follows:

[0204] The rendering loss is determined based on the difference between modal information of the same type in the first reference information and the first rendering information, wherein the modal information includes at least one of depth information, mask information, RGB information and edge information;

[0205] Based on the rendering loss, the initial pose is iteratively optimized to obtain the first target pose of the target object.

[0206] In one possible implementation, when the processor performs differentiable rendering based on the 3D model information and initial pose of the target object to obtain the first rendering information corresponding to the target object, it is further configured as follows:

[0207] Differentiable rendering is performed based on the 3D model information of the target object, its initial pose, and the camera parameters corresponding to the left eye camera in the binocular camera system to obtain the left eye rendering information of the target object.

[0208] Based on the 3D model information of the target object, the initial pose, and the camera parameters corresponding to the right camera in the binocular camera, differentiable rendering is performed to obtain the right-eye rendering information of the target object. The first rendering information includes the left-eye rendering information and the right-eye rendering information.

[0209] When determining the rendering loss based on the difference between modal information of the same type in the first reference information and the first rendering information, the processor is further configured as follows:

[0210] The first loss is determined based on the difference between the left-eye reference information included in the left-eye image of the first binocular image and the modal information of the same type in the left-eye rendering information;

[0211] The second loss is determined based on the difference between the right-eye reference information included in the right-eye image of the first binocular image and the modal information of the same type in the right-eye rendering information;

[0212] The rendering loss is determined based on the first loss and the second loss.

[0213] In one possible implementation, the processor is further configured to:

[0214] After obtaining the first target pose of the target object, differentiable rendering is performed based on the three-dimensional model information of the target object and the first target pose to obtain the second rendering information corresponding to the target object.

[0215] For the second stereo image included in the stereo image sequence, the pose of the first target is iteratively optimized based on the second reference information and the second rendering information extracted from the second stereo image to obtain the second target pose of the target object corresponding to the second stereo image. The second reference information and the second rendering information have the same information type. The stereo image sequence includes multiple pairs of stereo images of the target object acquired by the stereo camera at different times. The second stereo image is the stereo image acquired at the next moment after the acquisition time corresponding to the first stereo image in the stereo image sequence.

[0216] The exemplary embodiments of this device correspond to the exemplary method section described above in terms of implementation. The corresponding content between the two can be referenced, combined, and cited, and will not be repeated here. The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section described above, and will not be repeated here.

[0217] Exemplary electronic devices

[0218] Figure 10A structural diagram of an electronic device provided in an embodiment of this disclosure includes at least one processor 11 and a memory 12.

[0219] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0220] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the object pose estimation methods and / or other desired functions of the various embodiments of this disclosure described above.

[0221] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0222] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0223] The output device 14 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0224] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0225] Exemplary computer program products and computer-readable storage media

[0226] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the object pose estimation methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0227] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0228] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the object pose estimation methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0229] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0230] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0231] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for estimating the pose of an object, comprising: Based on the first binocular image containing the target object acquired by the binocular camera, the depth information and mask information of the target object are determined; Based on the depth information and mask information of the target object, the initial pose of the target object is determined; Based on the three-dimensional model information of the target object and the initial pose, differentiable rendering is performed to obtain the first rendering information corresponding to the target object; Based on the first reference information and the first rendering information extracted from the first binocular image, the initial pose is iteratively optimized to obtain the first target pose of the target object, wherein the first reference information and the first rendering information have the same information type.

2. The method according to claim 1, wherein, Determining the initial pose of the target object based on its depth information and mask information includes: Based on the depth information and the mask information, a set of candidate poses for the target object is determined. The set of candidate poses includes multiple candidate poses of the target object in the camera coordinate system corresponding to the stereo camera. Different candidate poses correspond to different rotation angles of the target object in the camera coordinate system. Differentiable rendering is performed based on each candidate pose and the 3D model information to obtain the rendered image corresponding to each candidate pose. Based on each of the rendered images and the first stereo image, the initial pose is selected from the candidate pose set.

3. The method according to claim 2, wherein, The depth information includes the left eye depth information of the target object corresponding to the left eye image in the first binocular image and / or the right eye depth information of the target object corresponding to the right eye image in the first binocular image; the mask information includes the left eye mask information of the target object corresponding to the left eye image in the first binocular image and / or the right eye mask information of the target object corresponding to the right eye image in the first binocular image. Determining the candidate pose set of the target object based on the depth information and the mask information includes: Based on the left-eye depth information and left-eye mask information of the target object corresponding to the left-eye image in the first binocular image, multiple first candidate poses of the target object are determined, and the left-eye image is acquired by the left-eye camera of the binocular camera. And / or, based on the right eye depth information and right eye mask information of the target object corresponding to the right eye image in the first binocular image, a plurality of second candidate poses of the target object are determined, the candidate pose set includes the plurality of first candidate poses and / or the plurality of second candidate poses, and the right eye image is acquired by the right eye camera of the binocular camera; The step of selecting the initial pose from the candidate pose set based on each of the rendered images and the first stereo image includes: When the candidate pose set includes the plurality of first candidate poses, based on the similarity between the first rendered image corresponding to each first candidate pose and the left eye image, a first target rendered image is determined from the first rendered image, and the first candidate pose corresponding to the first target rendered image is determined as the initial pose; Alternatively, if the candidate pose set includes the plurality of second candidate poses, a second target rendering image is determined from the second rendering image based on the similarity between the second rendering image corresponding to each second candidate pose and the right eye image, and the second candidate pose corresponding to the second target rendering image is determined as the initial pose; Alternatively, if the candidate pose set includes the plurality of first candidate poses and the plurality of second candidate poses, a first pose subset is selected from the plurality of first candidate poses based on the first rendered image corresponding to each first candidate pose and the left eye image; a second pose subset is selected from the plurality of second candidate poses based on the second rendered image corresponding to each second candidate pose and the right eye image; and the initial pose is selected from the same poses included in the first pose subset and the second pose subset, wherein the same pose refers to a pose that belongs to both the first pose subset and the second pose subset.

4. The method according to claim 3, wherein, The step of selecting a subset of first poses from multiple first candidate poses based on the first rendered image corresponding to each first candidate pose and the left eye image includes: Calculate the first similarity between each of the first rendered images and the left eye image; Based on each of the first similarities, a first candidate pose that meets the first similarity condition is determined, and a first pose subset is obtained, wherein the first similarity condition includes the first similarity belonging to the top N highest similarities among multiple first similarities, where N is a positive integer, and / or the first similarity is higher than the first similarity threshold. The step of selecting a subset of second poses from multiple second candidate poses based on the second rendered image corresponding to each second candidate pose and the right eye image includes: Calculate the second similarity between each of the second rendered images and the right eye image; Based on each of the second similarities, a second candidate pose that meets the second similarity condition is determined, and a subset of the second poses is obtained. The second similarity condition includes that the second similarity belongs to the top M highest similarities among multiple second similarities, where M is a positive integer, and / or that the second similarity is higher than the second similarity threshold. The step of filtering the initial pose from the same poses included in the first pose subset and the second pose subset includes: If the first pose subset and the second pose subset include at least two of the same poses, for any one of the same poses, a similarity score corresponding to the same pose is determined based on the first similarity and the second similarity corresponding to the same pose. The same pose with the highest similarity score is determined as the initial pose.

5. The method according to claim 3, wherein, The step of determining multiple first candidate poses of the target object based on the left-eye depth information and left-eye mask information of the target object corresponding to the left-eye image in the first binocular image includes: Based on the left eye depth information and the left eye mask information, the first initial position of the target object is determined; Based on the first initial position, camera position sampling is performed to obtain multiple virtual camera positions; For any of the aforementioned virtual camera positions, a rotation angle is sampled to obtain multiple virtual camera orientations corresponding to the virtual camera position; Based on the orientation of the virtual camera corresponding to each of the multiple virtual camera positions, determine multiple rotation angles corresponding to the target object; Based on the multiple rotation angles and the first initial position, multiple first candidate postures are obtained.

6. The method according to any one of claims 1 to 5, wherein, The step of iteratively optimizing the initial pose based on the first reference information extracted from the first binocular image and the first rendering information to obtain the first target pose of the target object includes: The rendering loss is determined based on the difference between the modal information of the same type in the first reference information and the first rendering information, wherein the modal information includes at least one of depth information, mask information, RGB information and edge information; Based on the rendering loss, the initial pose is iteratively optimized to obtain the first target pose of the target object.

7. The method according to claim 6, wherein, The step of performing differentiable rendering based on the 3D model information of the target object and the initial pose to obtain the first rendering information corresponding to the target object includes: Based on the 3D model information of the target object, the initial pose, and the camera parameters corresponding to the left eye camera in the binocular camera, differentiable rendering is performed to obtain the left eye rendering information of the target object. Based on the 3D model information of the target object, the initial pose, and the camera parameters corresponding to the right eye camera in the binocular camera, differentiable rendering is performed to obtain the right eye rendering information of the target object. The first rendering information includes the left eye rendering information and the right eye rendering information. The step of determining the rendering loss based on the difference between modal information of the same type in the first reference information and the first rendering information includes: The first loss is determined based on the difference between the left-eye reference information included in the left-eye image of the first binocular image and the modal information of the same information type in the left-eye rendering information; The second loss is determined based on the difference between the right-eye reference information included in the right-eye image of the first binocular image and the modal information of the same information type in the right-eye rendering information; The rendering loss is determined based on the first loss and the second loss.

8. The method according to any one of claims 1 to 5, wherein, The method further includes: After obtaining the first target pose of the target object, differentiable rendering is performed based on the three-dimensional model information of the target object and the first target pose to obtain the second rendering information corresponding to the target object; For the second stereo image included in the stereo image sequence, the pose of the first target is iteratively optimized based on the second reference information and the second rendering information extracted from the second stereo image to obtain the second target pose of the target object corresponding to the second stereo image. The second reference information and the second rendering information have the same information type. The stereo image sequence includes multiple pairs of stereo images acquired by the stereo camera of the target object at different times. The second stereo image is the stereo image acquired at the next moment after the acquisition time corresponding to the first stereo image in the stereo image sequence.

9. An object pose estimation device, comprising a processor, the processor being configured to: Based on the first binocular image containing the target object acquired by the binocular camera, the depth information and mask information of the target object are determined; Based on the depth information and mask information of the target object, the initial pose of the target object is determined; Based on the three-dimensional model information of the target object and the initial pose, differentiable rendering is performed to obtain the first rendering information corresponding to the target object; Based on the first reference information and the first rendering information extracted from the first binocular image, the initial pose is iteratively optimized to obtain the first target pose of the target object, wherein the first reference information and the first rendering information have the same information type.

10. A computer-readable storage medium storing a computer program, which, when executed, implements the object pose estimation method according to claims 1-8.

11. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the object pose estimation method according to claims 1-8.

Images