Pose estimation method and device

Abstract

The pose estimation method comprises the following steps: acquiring a reference image set for performing pose estimation on a target object contained in a target image; wherein the reference images in the reference image set respectively comprise the target object at different visual angles; the view angle of the target object contained in the target image is different from the view angle of the target object contained in the reference image; performing three-dimensional reconstruction on the target object based on the reference image set to obtain a three-dimensional model corresponding to the target object; constructing a three-dimensional bounding box surrounding the target object based on the three-dimensional model, and obtaining position coordinates of angular points of the three-dimensional bounding box; and performing pose estimation calculation on the target object contained in the target image based on the position coordinates of the angular points of the three-dimensional bounding box.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence, and more particularly, relate to a method and apparatus for posture estimation. Background Art

[0002] Pose estimation is the process of estimating the position and orientation of objects in an image in three-dimensional space. Specifically, it involves calculating the position coordinates of the objects in the image in three-dimensional space and their rotation angles relative to a reference coordinate system. This technology is widely used in fields such as computer vision, robotic navigation, and augmented reality. For example, in self-driving cars, pose estimation helps the vehicle understand the position and orientation of surrounding objects, enabling safer driving decisions.

[0003] However, existing pose estimation methods usually rely on a large number of reference images from different perspectives. If sufficient reference images from different perspectives cannot be obtained, the effect of pose estimation will be affected. Summary of the Invention

[0004] This specification proposes a pose estimation method, including:

[0005] Acquire a reference image set for estimating a pose of a target object contained in a target image; wherein the reference images in the reference image set respectively contain the target object at different perspectives; and the perspective of the target object contained in the target image is different from the perspective of the target object contained in the reference image;

[0006] Performing three-dimensional reconstruction of the target object based on the reference image set to obtain a three-dimensional model corresponding to the target object;

[0007] Constructing a three-dimensional bounding box surrounding the target object based on the three-dimensional model, and obtaining position coordinates of corner points of the three-dimensional bounding box;

[0008] Based on the position coordinates of the corner points of the three-dimensional bounding box, a pose estimation calculation is performed on the target object contained in the target image.

[0009] Optionally, the three-dimensional model includes a three-dimensional point cloud;

[0010] Constructing a three-dimensional bounding box surrounding the target object based on the three-dimensional model includes:

[0011] Performing target detection on the three-dimensional point cloud to determine coordinate points related to the target object contained in the three-dimensional point cloud;

[0012] The other coordinate points in the three-dimensional point cloud except the coordinate points related to the target object are deleted, and a three-dimensional bounding box surrounding the target object is constructed based on the remaining coordinate points in the three-dimensional point cloud.

[0013] Optionally, performing pose estimation calculation on the target object contained in the target image based on the position coordinates of the corner points of the three-dimensional bounding box includes:

[0014] Projecting the corner points of the three-dimensional bounding box onto two-dimensional planes corresponding to respective reference images in the reference image set to obtain two-dimensional position coordinates of the corner points on the two-dimensional planes;

[0015] Based on the two-dimensional position coordinates of the corner points on the two-dimensional plane, a pose estimation calculation is performed on the target object contained in the target image.

[0016] Optionally, performing pose estimation calculation on the target object contained in the target image based on the two-dimensional position coordinates of the corner point on the two-dimensional plane includes:

[0017] Constructing a two-dimensional heat map for each reference image based on the two-dimensional position coordinates; wherein the two-dimensional heat map is used to represent the position distribution of the corner points on the two-dimensional plane corresponding to the reference image;

[0018] Image features are extracted from each reference image in the reference image set and the target image, and the extracted image features and the two-dimensional heat maps corresponding to each reference image are input into a pre-trained deep learning model; wherein the deep learning model is used to predict a two-dimensional heat map corresponding to the target image based on the input image features extracted from the reference images, the two-dimensional heat maps corresponding to the reference images, and the image features extracted from the target image; the two-dimensional heat map corresponding to the target image is used to represent the position distribution of the corner points on the two-dimensional plane corresponding to the target image;

[0019] Obtain a two-dimensional heat map corresponding to the target image output by the deep learning model, and establish a correspondence between the two-dimensional position coordinates of the corner points in the two-dimensional heat map corresponding to the target image and the three-dimensional position coordinates of the corner points in the three-dimensional boundary map;

[0020] Based on the corresponding relationship, a pose estimation calculation is performed on the target object contained in the target image to obtain pose information of the target object under the viewing angle of the target image.

[0021] Optionally, constructing a two-dimensional heat map for each reference image based on the two-dimensional position coordinates includes:

[0022] Based on a preset heat map function, mapping the two-dimensional position coordinates into heat map values corresponding to the respective pixel positions contained in the respective reference images; wherein the heat map values represent the degree of proximity between the respective pixel positions and the positions of the corner points;

[0023] A two-dimensional heat map is constructed for each of the reference images based on the heat map values corresponding to the respective pixel positions.

[0024] Optionally, the heat map function includes a heat map function based on Gaussian distribution.

[0025] Optionally, the deep learning model includes an end-to-end decoder based on a Transformer architecture; the decoder is used to decode the image features extracted from the target image with reference to the image features extracted from the reference image and the two-dimensional heat map corresponding to the reference image to obtain a two-dimensional heat map corresponding to the target image.

[0026] Optionally, the model loss used in training the deep learning model includes:

[0027] a first model loss for representing a first error between a heat map value corresponding to each pixel position included in the predicted two-dimensional heat map and a true heat map value corresponding to each pixel position included in the two-dimensional heat map; and

[0028] A second model loss is used to represent a second error between the two-dimensional position coordinates of the corner point in the predicted two-dimensional heat map and the true two-dimensional position coordinates of the corner point in the two-dimensional heat map.

[0029] Optionally, performing pose estimation calculation on the target object contained in the target image based on the corresponding relationship to obtain pose information of the target object under the viewing angle of the target image includes:

[0030] The PnP algorithm is adopted to calculate the position and posture information of the target object contained in the target image under the viewing angle of the target image based on the corresponding relationship.

[0031] This specification also proposes a posture estimation device, comprising:

[0032] an acquisition module, configured to acquire a reference image set for performing pose estimation on a target object contained in a target image; wherein the reference images in the reference image set respectively contain the target object at different perspectives; and the perspective of the target object contained in the target image is different from the perspective of the target object contained in the reference image;

[0033] a reconstruction module, performing three-dimensional reconstruction of the target object based on the reference image set to obtain a three-dimensional model corresponding to the target object;

[0034] A construction module, which constructs a three-dimensional bounding box surrounding the target object based on the three-dimensional model and obtains the position coordinates of the corner points of the three-dimensional bounding box;

[0035] A calculation module performs pose estimation calculation on the target object contained in the target image based on the position coordinates of the corner points of the three-dimensional bounding box.

[0036] In the above embodiment, on the one hand, since the position coordinates of the corner points of the three-dimensional bounding box surrounding the target object constructed based on the reference image can be used to estimate the pose of the target object under the new perspective contained in the target image, and the three-dimensional bounding box surrounding the target object can usually be reliably restored by using only a small number of reference images under sparse perspectives; therefore, by adopting this pose estimation method, there is no need to rely on reference images under dense perspectives, and only a small number of reference images under sparse perspectives can be used to complete the pose estimation for the target object, thereby improving the versatility and adaptability when performing pose estimation, and being able to ensure the accuracy of pose estimation when it is impossible to obtain enough reference images under different perspectives.

[0037] On the other hand, whether the target object contained in the reference image is occluded usually does not directly affect the construction of the three-dimensional bounding box surrounding the target object; therefore, in practical applications, even if the target object contained in the reference image is occluded, the three-dimensional bounding box surrounding the target object can still be reliably reconstructed through limited data, and the position coordinates of the corner points of the three-dimensional bounding box can be used to normally complete the pose estimation of the target object under the new perspective, thereby improving the robustness of the pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 is a flow chart of a pose estimation method shown in one embodiment of this specification;

[0040] Figure 2 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0041] Figure 3 This is a block diagram of a posture estimation device shown in one embodiment of this specification. DETAILED DESCRIPTION

[0042] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0043] Many applications involve using reference images of a target object from different perspectives to estimate the pose of the same target object from a new perspective within a target image. The key to this technology is that even without a full view of the target object, the pose information of the target object from a new perspective can be inferred from a limited number of viewing angles.

[0044] For example, in some applications that support real-time interaction in virtual scenes such as AR / VR (such as AR navigation or VR-based scene construction), there is usually a need for technology to use reference images of the target object at different perspectives to estimate the pose of the target object at a new perspective.

[0045] In related technologies, the technique of using reference images of a target object at different perspectives to estimate the pose of the target object at a new perspective contained in a target image generally adopts the following two technical paradigms:

[0046] In the first technical paradigm, pose estimation can be performed using matching. For example, OnePose and OnePose++ are solutions that use this technical paradigm.

[0047] In this technical paradigm, a set of reference images of the target object from different perspectives is first required. 3D reconstruction techniques are then used to reconstruct the target object from these reference images, generating a 3D point cloud corresponding to the target object. This step aims to construct the 3D spatial structure of the object, which serves as the basis for subsequent pose estimation.

[0048] In addition, a feature matching neural network can also be trained. After reconstructing the three-dimensional point cloud corresponding to the target object, the three-dimensional point cloud and the target image containing the target object under the new perspective can be simultaneously input into the feature matching neural network. The network learns the correspondence between the three-dimensional point cloud and the image pixels contained in the target image, and then calculates the pose information of the target object under the new perspective based on the correspondence. For example, in practical applications, the above correspondence can be used as a calculation parameter to use the PnP (Perspective-n-Point, N-point perspective algorithm) to solve the pose information of the target object under the new perspective.

[0049] However, this technical paradigm usually only performs well in the case of reference images with dense perspectives. When the number of reference images provided is small and the perspectives provided for reference are limited, the reconstruction quality of the target object's three-dimensional reconstruction will be significantly reduced, thereby affecting the final pose estimation effect.

[0050] Moreover, in this technical paradigm, if the target object contained in the reference image is occluded, then when performing three-dimensional reconstruction of the target object, it is usually impossible to reconstruct the occluded part covering the target object, resulting in the inability to match the occluded area in the new perspective image, which may lead to failure of pose estimation and poor overall robustness.

[0051] In the second technical paradigm, pose estimation can be performed using a retrieval method. Gen6D is a solution that adopts this technical paradigm.

[0052] In this technical paradigm, a reference image database annotated with real pose data must first be pre-built. The reference images in this reference image database can contain target objects from different perspectives. When a target image containing the target object from a new perspective is input, the reference database can be searched for a reference image with a perspective that is most similar to the target object contained in the target image. The pose data annotated for the target object contained in this reference image is used as the initial pose estimate for the target image. The target image and the queried reference image can then be input into a neural network to enhance the initial pose to improve accuracy.

[0053] However, this technical paradigm, while the initial pose relies solely on reference images with real pose annotations, can reduce reliance on 3D reconstruction and avoid the problem of poor reconstruction quality under sparse viewpoints, if the reference images in the reference image database have sparse or incomplete viewpoints (e.g., lacking certain key angles), the deviation of the queried initial pose will be large, resulting in failure of the final pose estimation. Therefore, this technical paradigm also performs well only in the case of dense reference viewpoints, and fails seriously when the reference viewpoints are incomplete or generally sparse.

[0054] As can be seen, the current mainstream technical paradigms in the field of pose estimation all suffer from a heavy reliance on dense reference images. When the number of reference images provided is small and the available reference perspectives are limited, the final pose estimation effect is poor.

[0055] In view of this, this specification proposes a new technical paradigm that can estimate the pose of a target object contained in a target image without relying on a reference image with dense perspective.

[0056] Under this new technical paradigm, a three-dimensional bounding box surrounding the target object can be constructed based on a reference image containing the target object from different perspectives. The position coordinates of the corner points of the three-dimensional bounding box are then used as an intermediate representation for pose estimation to estimate the pose of the target object from the new perspective contained in the target image.

[0057] First, a reference image set for estimating a pose of a target object contained in a target image can be obtained; wherein the reference images in the reference image set respectively contain the target object at different perspectives; and the perspective of the target object contained in the target image is different from the perspective of the target object contained in the reference image;

[0058] Secondly, the target object can be reconstructed in three dimensions based on the reference image set to obtain a three-dimensional model corresponding to the target object, and a three-dimensional bounding box surrounding the target object can be constructed based on the three-dimensional model.

[0059] Finally, the position coordinates of the corner points of the three-dimensional bounding box can be obtained, and then the position coordinates of the corner points of the three-dimensional bounding box can be used as an intermediate representation of the pose estimation to perform pose estimation calculation on the target object under the new perspective contained in the target image.

[0060] In this technical solution, on the one hand, the position coordinates of the corner points of the three-dimensional bounding box surrounding the target object constructed based on the reference image can be used to estimate the pose of the target object under the new perspective contained in the target image, and the three-dimensional bounding box surrounding the target object can usually be reliably restored by using only a small number of reference images under sparse perspectives; therefore, this pose estimation method no longer relies on reference images under dense perspectives, and only a small number of reference images under sparse perspectives can be used to complete the pose estimation for the target object, thereby improving the versatility and adaptability of pose estimation, and being able to ensure the accuracy of pose estimation when it is impossible to obtain enough reference images under different perspectives.

[0061] On the other hand, whether the target object contained in the reference image is occluded usually does not directly affect the construction of the three-dimensional bounding box surrounding the target object; therefore, in practical applications, even if the target object contained in the reference image is occluded, the three-dimensional bounding box surrounding the target object can still be reliably reconstructed through limited data, and the position coordinates of the corner points of the three-dimensional bounding box can be used to normally complete the pose estimation of the target object under the new perspective, thereby improving the robustness of the pose estimation.

[0062] See Figure 1 , Figure 1 This is a flowchart of a pose estimation method shown in this specification; the method includes the following execution process:

[0063] Step 102: Acquire a reference image set for estimating a pose of a target object contained in a target image; wherein the reference images in the reference image set respectively contain the target object at different perspectives; and the perspective of the target object contained in the target image is different from the perspective of the target object contained in the reference image.

[0064] The reference images in the reference image set may each include the same target object from different perspectives. The target image may also include the target object, but unlike the reference images in the reference image set, the target object may be captured from a different perspective than the reference images in the reference image set.

[0065] For example, in practical applications, the reference images in the reference image set may be reference images obtained by capturing the same target object from all angles at different viewing angles. The target image may be an image captured from a completely different viewing angle than the reference images in the reference image set.

[0066] The viewpoint of the target object contained in a reference image in the reference image set can be referred to as a reference viewpoint. The viewpoint of the target object contained in the target image is typically a completely new viewpoint that is completely different from that of the reference images. This new viewpoint is the viewpoint at which the pose of the target object needs to be re-estimated.

[0067] When it is necessary to estimate the pose of the target object contained in the target image under this new perspective, a pre-prepared reference image set for estimating the pose of the target object contained in the target image can be obtained, and then subsequent pose estimation calculations can be performed based on the reference images in the reference image set.

[0068] In practical applications, the above-mentioned reference image set can be a reference image set constructed based on reference images of dense perspectives, or a reference image set constructed based on reference images of sparse perspectives, which is not specifically limited in this specification.

[0069] An image set constructed based on dense-view reference images refers to an image set that covers all viewpoints of the target object, constructed from a large number of reference images of the target object with different viewpoints. An image set constructed based on sparse-view reference images refers to an image set that covers only a small number of viewpoints of the target object, constructed from a small number of reference images of the target object with partial viewpoints.

[0070] For example, in practical applications, the same target object can be photographed from all angles at different viewing angles, and then a reference image set covering various viewing angles of the target object can be constructed based on the reference images obtained. Alternatively, the same target object can be photographed from only some viewing angles, and then a reference image set covering only a small number of viewing angles of the target object can be constructed based on the reference images obtained.

[0071] Among them, the number of reference images contained in the reference image set constructed based on the reference image of the dense perspective and the reference image set constructed based on the reference image of the sparse perspective will no longer be specifically limited in this specification. In actual applications, they can be flexibly selected based on specific needs.

[0072] For example, in one example, a reference image set constructed based on dense perspective reference images may specifically include more than 100 reference images from different perspectives, while a reference image set constructed based on sparse perspective reference images may specifically include only 2-5 reference images from different perspectives.

[0073] The specific type and format of the above-mentioned reference image are not specifically limited in this specification. In practical applications, the above-mentioned reference image can specifically cover images of any type and any format; for example, in one example, the above-mentioned reference image can specifically be a color image in RGB format.

[0074] Step 104: reconstruct the target object in three dimensions based on the reference image set to obtain a three-dimensional model corresponding to the target object;

[0075] After obtaining the reference image set, the target object may be three-dimensionally reconstructed based on the reference images in the reference image set to obtain a three-dimensional model corresponding to the target object.

[0076] 3D reconstruction generally refers to the process of recovering a 3D model of an object from one or more 2D images. The goal of 3D reconstruction is to infer information such as the object's shape, size, and position from the 2D image data and generate a 3D representation of the object based on this information.

[0077] The 3D reconstruction method used when performing 3D reconstruction of the target object based on the reference images in the reference image set usually depends on factors such as the amount of available data (single or multiple images), the characteristics of the target object, and the desired accuracy. In practical applications, it can be flexibly selected based on specific needs.

[0078] For example, in one example, a mature 3D reconstruction framework such as DUSt3R (Dense and Unconstrained Stereo3D Reconstruction) can be used to perform 3D reconstruction of the target object.

[0079] In another example, in addition to using a mature 3D reconstruction framework to reconstruct the target object in three dimensions, if the reference images in the above-mentioned reference image set are labeled with real pose information, the target object can also be reconstructed in three dimensions based on the labeled pose information; for example, using the labeled real pose information as constraints, running the relevant 3D reconstruction algorithm, and adopting the optimal solution method to reconstruct the target object in three dimensions.

[0080] It should be noted that, after the target object is three-dimensionally reconstructed, the representation method of the three-dimensional model corresponding to the target object can be flexibly selected based on specific needs in actual applications.

[0081] For example, in one example, the three-dimensional model may be in the form of a three-dimensional point cloud.

[0082] A 3D point cloud is typically composed of a series of positional coordinates generated during the 3D reconstruction process. These positional coordinates are typically the coordinates corresponding to the locations of feature points (usually pixels) on the target object being modeled in the 3D coordinate system. These positional coordinates can be used to represent the changes in the feature points on the target object under different viewing angles.

[0083] It should be emphasized that after using different three-dimensional reconstruction methods to reconstruct the target object, the representation of the three-dimensional model corresponding to the target object may usually be different. Therefore, in practical applications, the specific representation of the three-dimensional model corresponding to the target object usually depends on the three-dimensional reconstruction method actually adopted.

[0084] Step 106: construct a three-dimensional bounding box surrounding the target object based on the three-dimensional model, and obtain the position coordinates of the corner points of the three-dimensional bounding box;

[0085] After the target object is three-dimensionally reconstructed to obtain a three-dimensional model corresponding to the target object, a three-dimensional bounding box surrounding the target object can be further constructed based on the three-dimensional model.

[0086] The 3D bounding box can be a 3D spatial structure that is constructed based on the boundary information of the target object in the 3D model and that can enclose the target object. In practical applications, the boundary of the target object can be first determined in the 3D model, and then a 3D structure that can completely enclose the boundary of the target object can be constructed as the 3D bounding box.

[0087] It should be noted that the shape of the three-dimensional structure is not particularly limited in this specification. In practical applications, the three-dimensional structure can be a regular three-dimensional structure or an irregular three-dimensional structure.

[0088] For example, if the three-dimensional space structure is a regular three-dimensional structure, it can be a rectangular three-dimensional space containing 8 corner points. If the three-dimensional space structure is an irregular three-dimensional structure, it can be an irregular rectangular three-dimensional space containing less than 8 or more than 8 corner points.

[0089] When the three-dimensional model uses different representation formats, the specific method for determining the boundary of the target object in the three-dimensional model will generally vary. In practical applications, the specific method for determining the boundary of the target object in the three-dimensional model can be flexibly selected based on the actual representation format used by the three-dimensional model.

[0090] For example, in one embodiment shown, taking the above-mentioned three-dimensional model in the form of a three-dimensional point cloud as an example, the three-dimensional point cloud usually contains a large number of coordinate points corresponding to the positions of the feature points on the target object in the three-dimensional coordinate system. In this case, target detection can be performed on the three-dimensional point cloud to filter out coordinate points that are not related to the target object, thereby determining the boundary of the target object in the three-dimensional point cloud.

[0091] Specifically, when constructing a three-dimensional bounding box surrounding the target object based on the three-dimensional point cloud, target detection can be first performed on the three-dimensional point cloud to determine the coordinate points related to the target object contained in the three-dimensional point cloud; it should be noted that the target detection algorithm used when performing target detection on the three-dimensional point cloud is no longer specifically limited in this specification. In actual applications, it can be flexibly selected based on specific needs.

[0092] Next, all coordinate points in the 3D point cloud other than those associated with the target object are deleted, leaving only those associated with the target object. The remaining coordinate points in the 3D point cloud can now clearly represent the boundary of the target object. For example, the remaining coordinate points in the 3D point cloud can typically form a sparse geometric structure that can represent the boundary of the target object.

[0093] Finally, a three-dimensional bounding box surrounding the target object can be constructed based on the coordinate points related to the target object retained in the three-dimensional point cloud, that is, the remaining coordinate points in the three-dimensional point cloud.

[0094] For example, the boundary of the target object can be determined based on the coordinate points related to the target object retained in the three-dimensional point cloud, and then a three-dimensional structure that can completely cover all the coordinate points related to the target object is generated based on the boundary initialization as a three-dimensional bounding box surrounding the target object.

[0095] It should be noted that, since the three-dimensional bounding box surrounding the target object can usually be just a sparse geometric structure that can surround the target object and does not rely on a reference image with a dense perspective, in practical applications, usually only a small number of reference images under a sparse perspective are needed to reliably restore the three-dimensional bounding box surrounding the target object; therefore, when the above-mentioned reference image set is an image set constructed based on reference images with a sparse perspective, that is, when the above-mentioned reference image set only contains a small number of reference images under different perspectives, it usually does not affect the construction process of the three-dimensional bounding box.

[0096] After a three-dimensional bounding box surrounding the target object is constructed based on the three-dimensional model corresponding to the target object, the position coordinates of each corner point of the three-dimensional bounding box can be further obtained.

[0097] For example, taking the above-mentioned three-dimensional model as a three-dimensional point cloud, and the above-mentioned three-dimensional bounding box as a regular rectangular three-dimensional space containing 8 corner points, the position coordinates of the coordinate points corresponding to the 8 corner points contained in the three-dimensional space structure in the three-dimensional point cloud can be obtained.

[0098] Step 108 : performing pose estimation calculation on the target object contained in the target image based on the position coordinates of the corner points of the three-dimensional bounding box.

[0099] After obtaining the position coordinates of the corner points of the three-dimensional bounding box surrounding the target object, the position coordinates can be further used as an intermediate representation of pose estimation to perform pose estimation calculation on the target object contained in the target image.

[0100] It should be noted that since the position coordinates of the corner points of the three-dimensional bounding box obtained are usually three-dimensional position coordinates, if the three-dimensional position coordinates of the corner points of the three-dimensional bounding box are directly used as the intermediate representation of the pose estimation, it may affect the computational performance of the pose estimation calculation; therefore, in practical applications, the position coordinates of the corner points of the three-dimensional bounding box can be converted into two-dimensional position coordinates on the basis of ensuring the accuracy of the position coordinates, and then the converted two-dimensional position coordinates are used as the intermediate representation of the pose estimation for pose estimation calculation.

[0101] In one embodiment shown, when performing pose estimation calculation on the target object contained in the target image based on the coordinates of the corner points of the three-dimensional bounding box, the corner points of the three-dimensional bounding box can first be projected onto two-dimensional planes corresponding to each reference image in the reference image set, respectively, to obtain the two-dimensional position coordinates of the corner points on the two-dimensional planes corresponding to each reference image; then, based on the two-dimensional position coordinates of the above-mentioned corner points on the two-dimensional planes corresponding to each reference image, the pose estimation calculation can be performed on the target object contained in the target image.

[0102] It should be noted that when projecting the corner points of the three-dimensional bounding box onto the two-dimensional planes corresponding to the respective reference images in the reference image set, this can be achieved by using a perspective projection function in the field of perspective geometry.

[0103] For example, assuming that each reference image in the reference image set has reference posture information, it can be expressed by ξ = (R, t); wherein ξ represents posture information, R represents the rotation vector of the target object contained in the reference image in the three-dimensional space, and t represents the translation vector of the target object contained in the reference image in the three-dimensional space; it should be noted that R is usually a vector composed of the angles of the target object on the X, Y, and Z axes in the three-dimensional space, and t is usually a vector composed of the positions of the target object on the X, Y, and Z axes in the three-dimensional space. Therefore, for a reference image, its own posture information ξ is usually an object posture with 6 degrees of freedom (i.e., 6 components).

[0104] In this case, the following formula can be used to project the corner points of the 3D bounding box onto the 2D planes corresponding to the reference images:

[0105] p′=πK(Rp+t)

[0106] In the above formula, π represents the perspective projection function; K represents the camera parameters, that is, the camera configuration parameters used to capture the image; P represents the three-dimensional corner points of the three-dimensional bounding box; p′ represents the two-dimensional point after being projected onto the two-dimensional plane of the reference image; R represents the rotation vector of the target object contained in the reference image in the three-dimensional space, and t represents the translation vector of the target object contained in the reference image in the three-dimensional space.

[0107] In practical applications, in order to adapt to the image processing capabilities of some deep learning models that can process image data, when estimating the pose of the target object contained in the target image, the two-dimensional position coordinates of the corner points of the three-dimensional bounding box on the two-dimensional plane corresponding to each reference image can also be converted into a more intuitive image form for processing.

[0108] For example, in one embodiment shown, the two-dimensional position coordinates of the corner points of the three-dimensional bounding box on the two-dimensional plane corresponding to each reference image may be converted into a heat map for processing.

[0109] In this case, when performing pose estimation calculation for the target object contained in the target image based on the two-dimensional position coordinates of the corner points of the three-dimensional bounding box on the two-dimensional plane corresponding to each reference image, first, a two-dimensional heat map can be constructed for each reference image based on the two-dimensional position coordinates of the corner points of the three-dimensional bounding box on the two-dimensional plane corresponding to each reference image; wherein, the two-dimensional heat map can specifically be used to represent the position distribution of the corner points on the two-dimensional plane corresponding to the reference image.

[0110] Secondly, in order to improve the effect of pose estimation, image features can be extracted from each reference image in the reference image set and the target image respectively.

[0111] Furthermore, the extracted image features and the two-dimensional heat maps corresponding to each reference image can be input into a pre-trained deep learning model.

[0112] Among them, the deep learning model can specifically be a model for predicting a two-dimensional heat map corresponding to the target image based on the input image features extracted from the reference image, the two-dimensional heat map corresponding to the reference image, and the image features extracted from the target image.

[0113] In other words, the input data of the deep learning model are the image features extracted from each reference image, the two-dimensional heat map corresponding to each reference image, and the image features extracted from the target image. The output data of the deep learning model is the two-dimensional heat map corresponding to the target image. The two-dimensional heat map corresponding to the target image is specifically used to represent the position distribution of corner points on the two-dimensional plane corresponding to the target image.

[0114] Finally, the two-dimensional heat map corresponding to the target image output by the deep learning model can be obtained. At this time, the two-dimensional heat map corresponding to the target image can contain the two-dimensional position coordinates (that is, two-dimensional projection) of the corner points of the three-dimensional bounding box in the two-dimensional heat map corresponding to the target image.

[0115] In one embodiment shown, when constructing a two-dimensional heat map for each reference image based on the two-dimensional position coordinates of the corner points of the three-dimensional bounding box on the two-dimensional plane corresponding to each reference image, it can be specifically implemented based on a preset heat map function.

[0116] First, based on a preset heatmap function, the two-dimensional position coordinates of the corner points of the three-dimensional bounding box on the two-dimensional plane corresponding to each reference image can be mapped into heatmap values corresponding to each pixel position contained in each reference image. The heatmap value can be used to specifically represent the proximity of each pixel position to the position where the corner point is located. Generally, the larger the heatmap value, the closer the pixel position is to the position where the corner point is located.

[0117] Then, a corresponding two-dimensional heat map may be constructed for each reference image based on the heat map values corresponding to each pixel position contained in each reference image.

[0118] For example, the heat map values corresponding to each pixel position contained in each reference image can be used as the pixel values corresponding to the pixel position for visual rendering to regenerate a two-dimensional heat map that can intuitively reflect the proximity of each pixel position to the position of the corner point.

[0119] It should be noted that the specific type of the heat map function is not particularly limited in this specification, and in actual applications, it can be flexibly selected based on specific needs. For example, a one-hot heat map function can be selected.

[0120] In one embodiment shown, in order to avoid the problem that the transition of corner points in the generated two-dimensional heat map is too abrupt and unnatural, the heat map function may adopt a heat map function based on Gaussian distribution.

[0121] The heat map function based on Gaussian distribution can be expressed as follows:

[0122]

[0123] In the above formula, (x, y) represents the two-dimensional position coordinates corresponding to any pixel position contained in the reference image; (x i ,y i ) represents the two-dimensional position coordinates of the i-th corner point in the three-dimensional bounding box on the two-dimensional plane corresponding to the reference image; H(x, y, i) represents the heat map value of the pixel position with the two-dimensional position coordinates (x, y) contained in the reference image corresponding to the i-th corner point; the heat map value can be used to specifically represent the degree of proximity between the pixel position with the two-dimensional position coordinates (x, y) contained in the reference image and the position of the i-th corner point; σ represents the standard deviation of the Gaussian distribution.

[0124] In practical applications, the value of σ can be determined based on the size of the target object.

[0125] For example, in one example, the size of the target object can be an integer multiple of the value of σ. For example, the value of σ can be 1 / 10 of the size of the target object. The size of the target object can be represented by the square of the average pixel distance from each corner point of the 3D bounding box to the geometric center of the 3D bounding box. Of course, the size of the target object can also be represented in other ways.

[0126] It should be noted that since the above-mentioned three-dimensional bounding box contains a total of i corner points, the heat map value corresponding to any pixel position contained in the reference image calculated based on the above-mentioned heat map function will usually include a total of i heat map values corresponding to the pixel position corresponding to these i corner points; therefore, in this way, the heat map value finally obtained corresponding to each pixel position contained in the reference image can specifically be a color value with i color channels.

[0127] By adopting a heat map function based on Gaussian distribution, the characteristics of Gaussian distribution can be utilized to make the pixel values of the pixel positions around each corner point in the two-dimensional heat map present a smooth transition feature, thereby avoiding the problem that the pixel values of the pixel positions around each corner point in the two-dimensional heat map change too drastically and hinder the learning effect of the deep learning model.

[0128] For example, if a one-hot heatmap function is used, the heatmap values calculated for each pixel position in the reference image typically contain only two values: 1 and 0. 1 typically indicates that the pixel position is a corner point; 0 indicates that the pixel position is not a corner point. Consequently, in the two-dimensional one-hot heatmap ultimately constructed for each reference image, the pixel values around each corner point typically change from 1 to 0, resulting in an unsmooth transition. This can hinder the learning process of deep learning models.

[0129] If the above-mentioned heat map function based on Gaussian distribution is adopted, the pixel values of the pixel positions around each corner point in the final generated two-dimensional heat map can show a smooth transition characteristic; for example, gradually changing from 1 to 0; compared with the use of one-hot heat map, it is obvious that the problem of excessive changes in the pixel values of the pixel positions around each corner point in the two-dimensional heat map, which hinders the learning effect of the deep learning model, can be avoided.

[0130] It should be noted that the specific type of the above-mentioned deep learning model and the model architecture of the above-mentioned deep learning model are not particularly limited in this specification. In practical applications, they can be flexibly selected based on specific needs.

[0131] In one embodiment shown, the above-mentioned deep learning model can be an end-to-end deep learning model using the Transformer architecture; for example, it can be an end-to-end decoder using the Transformer architecture. The end-to-end model specifically refers to a model in which the entire process from input data to final output result is completed completely autonomously by the model, without the need for manual design of additional feature engineering or intermediate processing steps. This end-to-end model can automatically learn all the feature representations and transformations required to perform the task directly from the raw data, thereby achieving the goals of a specific task,

[0132] At this time, the decoder can be used to decode the image features extracted from the target image with reference to the image features extracted from the reference image and the two-dimensional heat map corresponding to the reference image, so as to obtain the two-dimensional heat map corresponding to the target image. In other words, the image decoding capability of the end-to-end decoder using the Transformer architecture can be utilized to perform image decoding on the image features extracted from the reference image and the two-dimensional heat map corresponding to the reference image, so as to predict the two-dimensional heat map corresponding to the target image.

[0133] In this case, before the image features extracted from each reference image, the two-dimensional heat map corresponding to each reference image, and the image features extracted from the target image are input to the decoder as input data, the image features extracted from each reference image and the image features extracted from the target image can be respectively constructed into the form of feature maps, and then the feature maps corresponding to each reference image are fused with the two-dimensional heat maps corresponding to each reference image to generate a fused image, and then the fused image and the feature maps corresponding to the target image are fused. Figure 1 The images are input into the decoder for image decoding.

[0134] For example, assuming that the reference image set contains a total of N reference images, the feature maps constructed based on the image features extracted from these N reference images can be recorded as F1~F N , and the two-dimensional heat maps corresponding to these N reference images can be recorded as H1~H N , the feature map constructed based on the image features extracted from the target image can be recorded as F q .

[0135] Since the feature map is usually constructed based on high-dimensional image features, and the two-dimensional heat map is usually constructed based on low-dimensional image features, in practical applications, a linear layer can be added to the input layer of the Transformer architecture to perform linear transformation on the two-dimensional heat map to convert the two-dimensional heat map into an image that matches the feature dimension of the feature map. Then, the feature maps F1 to F N and two-dimensional heat map H1~H N Perform one-to-one image fusion; for example, at this time, the feature dimensions of the feature map and the two-dimensional feature map are the same, and the pixel values contained in the feature map F1 can be directly added element by element with the pixel values at the same pixel position contained in the two-dimensional heat map H1 to perform image fusion on the feature map F1 and the two-dimensional heat map H1. Similarly, the same method can be used to perform image fusion on the remaining feature maps and two-dimensional heat maps in turn.

[0136] In addition, in order to adapt to the processing capabilities of the input layer of the Transformer architecture, before the above-mentioned fused image and the feature map corresponding to the target image are used as input data and input into the decoder, the above-mentioned fused image and the feature map corresponding to the target image can be divided into non-overlapping image units according to the maximum processing capability of the input layer of the Transformer architecture, and these image units are spliced into a sequence of image units in a unit-by-unit order, and then the sequence is input into the decoder for image decoding.

[0137] After the decoder decodes the two-dimensional heat map corresponding to the target image based on the input data, the two-dimensional heat map is an image composed of high-dimensional image features. In this case, a linear layer can be added to the output layer of the Transformer architecture to perform linear changes on the two-dimensional heat map, converting the two-dimensional heat map into an image whose feature dimensions match the original two-dimensional feature map corresponding to the reference image, and then outputting the two-dimensional heat map corresponding to the target image.

[0138] It should be emphasized that the training method used when training the above-mentioned deep learning model can be flexibly selected based on needs in practical applications.

[0139] For example, in one embodiment shown, a supervised training method may be used to train the deep learning model. In this case, a complete training sample may specifically be a sample pair consisting of a reference image sample and a target image sample.

[0140] The reference image sample may specifically include a feature map sample constructed based on image features extracted from the reference image sample, a two-dimensional heat map sample corresponding to the reference image sample, and the target image sample may include a two-dimensional heat map sample corresponding to the target image.

[0141] In addition, in supervised training scenarios, the 2D heatmap samples corresponding to the target image can also be pre-labeled as a supervision item. This supervision item typically depends on the optimization objective reflected by the loss function used when training the deep learning model, and is not specifically limited in this specification.

[0142] In one embodiment shown, when the deep learning model is trained in a supervised training manner based on prepared training samples, the model loss used may specifically include two parts: a first model loss and a second model loss.

[0143] The first model loss can be used to represent the first error between the heat map value corresponding to each pixel position in the predicted two-dimensional heat map and the true heat map value corresponding to each pixel position in the two-dimensional heat map. In other words, it represents the error between the heat map value corresponding to each pixel position in the predicted two-dimensional heat map and the true heat map value.

[0144] The second model loss can be used to represent the second error between the 2D position coordinates of the corner points of the 3D bounding box in the predicted 2D heat map and the actual 2D position coordinates of the corner points in the 2D heat map. That is, the error between the 2D position coordinates of the corner points contained in the predicted 2D heat map and the actual 2D position coordinates of the corner points in the 2D heat map.

[0145] At this time, the supervision items pre-labeled for the two-dimensional heat map sample corresponding to the target image may specifically include the true heat map values corresponding to each pixel position contained in the two-dimensional heat map sample, and the true two-dimensional position coordinates of the corner points of the three-dimensional bounding box in the two-dimensional heat map sample.

[0146] In this case, the model loss used to train the deep learning model is expressed as follows:

[0147] L=L1+λL2

[0148]

[0149] Wherein, L represents the model loss of the deep learning model; L1 represents the first model loss; L2 represents the second model loss; λ represents the weighting coefficient; Loss represents the loss function adopted by the model loss; N represents the number of pixel positions contained in the predicted two-dimensional heat map; M represents the number of corner points contained in the three-dimensional bounding box (for example, the value of M can be 8); h i Represents the heat map value corresponding to the i-th pixel position contained in the predicted two-dimensional heat map; h′ i represents the true heat map value (i.e., the pre-marked heat map value) corresponding to the i-th pixel position contained in the predicted two-dimensional heat map; b i b′ represents the two-dimensional position coordinates of the i-th corner point in the three-dimensional bounding box in the predicted two-dimensional heat map; i Represents the true two-dimensional position coordinates (i.e., the pre-marked two-dimensional position coordinates) of the i-th corner point in the three-dimensional bounding box in the predicted two-dimensional heat map.

[0150] It should be emphasized that in practical applications, the type of loss function used in the model loss formula for training the deep learning model can be flexibly selected based on specific needs.

[0151] For example, in one embodiment shown, the loss function adopted by the above-mentioned model loss may include a Smooth L1 loss function.

[0152] Among them, the Smooth L1 loss function combines the advantages of L1 loss and L2 loss, aiming to address the problems of L1 loss gradient discontinuity and L2 loss sensitivity to outliers. Compared with other common loss functions, such as L1 loss function and L2 loss function, the Smooth L1 loss function can reduce the impact of outliers and provide smoother gradients, which helps to accelerate convergence when the error is small and improve the accuracy of the solution.

[0153] When the Smooth L1 loss function is used, the first model loss and the second model loss can be expressed as follows:

[0154]

[0155] The meanings of the parameters in the above two formulas will not be repeated here.

[0156] After obtaining the two-dimensional heat map corresponding to the target image output by the deep learning model, a correspondence can be established between the two-dimensional position coordinates of the corner points of the three-dimensional bounding box in the two-dimensional heat map corresponding to the target image and the three-dimensional position coordinates of the corner points in the three-dimensional boundary map. Then, based on the correspondence, a pose estimation calculation is performed on the target object contained in the target image to obtain the pose information of the target object from the perspective of the target image.

[0157] It should be noted that the posture estimation algorithm used when performing posture estimation calculations on the target object contained in the target image based on the above-mentioned correspondence will no longer be specifically limited in this specification. In actual applications, it can be flexibly selected based on needs.

[0158] For example, in one embodiment shown, a PnP algorithm can be specifically used to calculate the posture information of the target object contained in the target image from the perspective of the target image based on the 2D position coordinates of the corner points of the established three-dimensional bounding box in the two-dimensional heat map corresponding to the target image, and the 2D-3D coordinate correspondence between the 3D position coordinates of the corner points in the three-dimensional boundary map.

[0159] It should be noted that the PnP algorithm is an algorithm that uses known 3D coordinates and their 2D projections in the image (i.e., 2D coordinates) in combination with camera parameters to determine the rotation (R) and translation (t) of the camera coordinates relative to the world coordinate system. The PnP algorithm calculates the pose information of the target object contained in the target image from the perspective of the target image, based on the established 2D-3D coordinate correspondence. The specific process of calculating the rotation (R) and translation (t) of the target object from the perspective of the target image using the PnP algorithm will not be described in detail in this manual.

[0160] In the above technical solution, on the one hand, the position coordinates of the corner points of the three-dimensional bounding box surrounding the target object constructed based on the reference image can be used to estimate the pose of the target object under the new perspective contained in the target image; and the three-dimensional bounding box surrounding the target object can usually be reliably restored by using only a small number of reference images under sparse perspectives; therefore, this pose estimation method no longer relies on reference images under dense perspectives, and only a small number of reference images under sparse perspectives can be used to complete the pose estimation of the target object, thereby improving the versatility and adaptability of pose estimation, and being able to ensure the accuracy of pose estimation when it is impossible to obtain enough reference images under different perspectives.

[0161] On the other hand, whether the target object contained in the reference image is occluded usually does not directly affect the construction of the three-dimensional bounding box surrounding the target object; therefore, in practical applications, even if the target object contained in the reference image is occluded, the three-dimensional bounding box surrounding the target object can still be reliably reconstructed through limited data, and the position coordinates of the corner points of the three-dimensional bounding box can be used to normally complete the pose estimation of the target object under the new perspective, thereby improving the robustness of the pose estimation.

[0162] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0163] Figure 2 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 2At the hardware level, the device includes a processor 202, an internal bus 204, a network interface 206, a memory 208, and a non-volatile memory 210, and may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 202 reading the corresponding computer program from the non-volatile memory 210 into the memory 208 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0164] like Figure 3 As shown, Figure 3 This is a block diagram of a posture estimation device according to an exemplary embodiment of the present specification. The device can be operated in the following manner: Figure 2 In the electronic device shown in FIG. 1 , the technical solution of this specification is implemented. The device 20 includes:

[0165] An acquisition module 301 acquires a reference image set for performing pose estimation on a target object contained in a target image; wherein the reference images in the reference image set respectively contain the target object at different perspectives; and the perspective of the target object contained in the target image is different from the perspective of the target object contained in the reference image.

[0166] A reconstruction module 302 performs three-dimensional reconstruction of the target object based on the reference image set to obtain a three-dimensional model corresponding to the target object;

[0167] A construction module 303 constructs a three-dimensional bounding box surrounding the target object based on the three-dimensional model, and obtains the position coordinates of the corner points of the three-dimensional bounding box;

[0168] The calculation module 304 performs pose estimation calculation on the target object contained in the target image based on the position coordinates of the corner points of the three-dimensional bounding box.

[0169] Accordingly, this specification also provides an electronic device, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps in all the method flows described above.

[0170] Accordingly, this specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0171] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0172] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0173] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0174] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0175] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any particular order.

[0176] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0177] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0181] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0182] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0183] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0185] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0186] The foregoing is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be included within the scope of the claims.

Claims

1. A pose estimation method, comprising: Acquire a reference image set for estimating a pose of a target object contained in a target image; wherein the reference images in the reference image set respectively contain the target object at different perspectives; and the perspective of the target object contained in the target image is different from the perspective of the target object contained in the reference image; Performing three-dimensional reconstruction of the target object based on the reference image set to obtain a three-dimensional model corresponding to the target object; Constructing a three-dimensional bounding box surrounding the target object based on the three-dimensional model, and obtaining position coordinates of corner points of the three-dimensional bounding box; Based on the position coordinates of the corner points of the three-dimensional bounding box, a pose estimation calculation is performed on the target object contained in the target image.

2. The method of claim 1, wherein the three-dimensional model comprises a three-dimensional point cloud; Constructing a three-dimensional bounding box surrounding the target object based on the three-dimensional model includes: Performing target detection on the three-dimensional point cloud to determine coordinate points related to the target object contained in the three-dimensional point cloud; The other coordinate points in the three-dimensional point cloud except the coordinate points related to the target object are deleted, and a three-dimensional bounding box surrounding the target object is constructed based on the remaining coordinate points in the three-dimensional point cloud.

3. The method of claim 1 , wherein performing pose estimation calculation on the target object contained in the target image based on the position coordinates of the corner points of the three-dimensional bounding box comprises: Projecting the corner points of the three-dimensional bounding box onto two-dimensional planes corresponding to respective reference images in the reference image set to obtain two-dimensional position coordinates of the corner points on the two-dimensional planes; Based on the two-dimensional position coordinates of the corner points on the two-dimensional plane, a pose estimation calculation is performed on the target object contained in the target image.

4. The method according to claim 3, performing pose estimation calculation on the target object contained in the target image based on the two-dimensional position coordinates of the corner point on the two-dimensional plane, comprising: Constructing a two-dimensional heat map for each reference image based on the two-dimensional position coordinates; wherein the two-dimensional heat map is used to represent the position distribution of the corner points on the two-dimensional plane corresponding to the reference image; Image features are extracted from each reference image in the reference image set and the target image, and the extracted image features and the two-dimensional heat maps corresponding to each reference image are input into a pre-trained deep learning model; wherein the deep learning model is used to predict a two-dimensional heat map corresponding to the target image based on the input image features extracted from the reference images, the two-dimensional heat maps corresponding to the reference images, and the image features extracted from the target image; the two-dimensional heat map corresponding to the target image is used to represent the position distribution of the corner points on the two-dimensional plane corresponding to the target image; Obtain a two-dimensional heat map corresponding to the target image output by the deep learning model, and establish a correspondence between the two-dimensional position coordinates of the corner points in the two-dimensional heat map corresponding to the target image and the three-dimensional position coordinates of the corner points in the three-dimensional boundary map; Based on the corresponding relationship, a pose estimation calculation is performed on the target object contained in the target image to obtain pose information of the target object under the viewing angle of the target image.

5. The method of claim 4, constructing a two-dimensional heat map for each of the reference images based on the two-dimensional position coordinates, comprising: Based on a preset heat map function, mapping the two-dimensional position coordinates into heat map values corresponding to the respective pixel positions contained in the respective reference images; wherein the heat map values represent the degree of proximity between the respective pixel positions and the positions of the corner points; A two-dimensional heat map is constructed for each of the reference images based on the heat map values corresponding to the respective pixel positions. The method of claim 5 , wherein the heat map function comprises a heat map function based on Gaussian distribution.

7. In the method of claim 4, the deep learning model includes an end-to-end decoder based on a Transformer architecture; the decoder is used to decode the image features extracted from the target image with reference to the image features extracted from the reference image and the two-dimensional heat map corresponding to the reference image to obtain a two-dimensional heat map corresponding to the target image.

8. The method of claim 7, wherein the model loss used in training the deep learning model comprises: a first model loss for representing a first error between a heat map value corresponding to each pixel position included in the predicted two-dimensional heat map and a true heat map value corresponding to each pixel position included in the two-dimensional heat map; as well as, A second model loss is used to represent a second error between the two-dimensional position coordinates of the corner point in the predicted two-dimensional heat map and the true two-dimensional position coordinates of the corner point in the two-dimensional heat map.

9. The method according to claim 4, wherein the step of performing pose estimation calculation on the target object contained in the target image based on the correspondence relationship to obtain pose information of the target object from the perspective of the target image comprises: The PnP algorithm is adopted to calculate the position and posture information of the target object contained in the target image under the viewing angle of the target image based on the corresponding relationship.

10. A posture estimation device, comprising: an acquisition module, configured to acquire a reference image set for performing pose estimation on a target object contained in a target image; wherein the reference images in the reference image set respectively contain the target object at different perspectives; and the perspective of the target object contained in the target image is different from the perspective of the target object contained in the reference image; a reconstruction module, performing three-dimensional reconstruction of the target object based on the reference image set to obtain a three-dimensional model corresponding to the target object; A construction module, which constructs a three-dimensional bounding box surrounding the target object based on the three-dimensional model and obtains the position coordinates of the corner points of the three-dimensional bounding box; A calculation module performs pose estimation calculation on the target object contained in the target image based on the position coordinates of the corner points of the three-dimensional bounding box.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.

Images