Pose estimation method and device, computer device and storage medium

Abstract

The application provides a pose estimation method and device, a computer device and a storage medium. The method comprises: acquiring a monocular image, a target mask image corresponding to a target to be measured in the monocular image, and a three-dimensional model of the target to be measured; sampling a plurality of initial pose data of the target to be measured under a determined view direction; determining initial position data of the target to be measured under each initial pose data based on the target mask image; wherein the initial pose data and the initial position data constitute initial pose data, and the plurality of initial pose data constitute an initial pose estimation set; selecting target quantity candidate pose data from the initial pose estimation set, adjusting the target quantity candidate pose data, and generating a plurality of intermediate pose data; constructing a loss function, and iteratively optimizing each intermediate pose data by using the loss function to generate optimized pose data; and determining pose estimation data corresponding to the target to be measured from the plurality of optimized pose data.

Description

Technical Field

[0001] This application relates to the field of pose estimation technology, and more specifically, to a pose estimation method, apparatus, computer device, and storage medium. Background Technology

[0002] Six-degree-of-freedom (6D) pose estimation is a key technology for estimating the six-degree-of-freedom pose (such as position and orientation) of a target object in space based on image information. It is crucial for fields such as industrial robot grasping, automated assembly, augmented reality, and precision measurement. Compared to two-dimensional detection, 6D pose estimation enables a higher level of spatial understanding, and it has irreplaceable value, especially in tasks involving precise manipulation or interaction with rigid objects.

[0003] Most 6D pose estimation techniques rely on deep learning models, requiring extensive labeled data for training and heavily depending on prior information about the target object and the consistency of the training data distribution during the inference phase. However, in many practical applications, target objects are often diverse and lack labeled data, making it extremely difficult to collect large amounts of training data, especially in environments such as high-altitude operations, extreme scenarios, and unstructured outdoor scenes. Furthermore, existing 6D pose estimation techniques heavily depend on depth information, but the performance of depth sensors degrades significantly under conditions of strong light interference, reflective materials, and long-distance or high-altitude shooting, severely limiting their applicability in complex outdoor or industrial scenarios. Therefore, achieving 6D pose estimation under conditions of insufficient training data and inability to obtain reliable depth information has become one of the key technical challenges urgently needing to be overcome in this field. Summary of the Invention

[0004] In view of this, this application provides a pose estimation method, apparatus, computer device and storage medium, which realizes a complete closed-loop process from the generation of initial pose assumptions to fine optimization based on the pose estimation strategy of geometric principles and image information fusion, without the participation of neural networks. While ensuring the accuracy of pose estimation, it improves the interpretability, versatility and system stability of the method.

[0005] Specifically, this application is implemented through the following technical solution: In a first aspect, embodiments of this application provide a pose estimation method, including: Acquire a monocular image, a target mask image corresponding to the target under test in the monocular image, and a three-dimensional model of the target under test; Multiple viewpoint directions corresponding to the 3D model are determined, and multiple initial pose data of the target under test are sampled under each viewpoint direction; and based on the target mask image, the initial position data of the target under test under each initial pose data is determined; wherein the initial pose data and the initial position data constitute initial pose data, and multiple initial pose data constitute an initial pose estimation set; Select a target number of candidate pose data from the initial pose estimation set, and adjust the target number of candidate pose data to generate multiple intermediate pose data; For each intermediate pose data, after generating the intermediate rendered image and intermediate mask image corresponding to the intermediate pose data using differential rendering technology, a loss function is constructed using the intermediate rendered image, the intermediate mask image, the monocular image and the target mask image, and the intermediate pose data is iteratively optimized using the loss function to generate optimized pose data. The pose estimation data corresponding to the target to be tested is determined from multiple optimized pose data.

[0006] In one optional implementation, determining the initial position data of the target under test based on the target mask image for each initial pose data includes: Determine the coordinate information of the center point of the bounding box of the target to be tested in the target mask image; Based on the coordinate information of the center point of the bounding box of the target under test, the set estimated depth value, and the camera intrinsic parameter matrix, the estimated three-dimensional position of the target under test is determined. For each initial pose data, the estimated three-dimensional position of the target is corrected based on the first diagonal length of the bounding box of the target in the target mask image and the second diagonal length of the bounding box of the target in the rendered image corresponding to the initial pose data, thereby generating the initial position data of the target under the initial pose data.

[0007] In one optional implementation, selecting a target number of candidate pose data from the initial pose estimation set includes: For each initial pose data in the initial pose estimation set, an initial rendered image corresponding to the initial pose data is generated; and feature points are extracted from the monocular image and the initial rendered image using the set target operator, respectively, to obtain multiple first feature points included in the monocular image and multiple second feature points included in the initial rendered image; and the matching distance between the first feature points and the matched second feature points in the feature space is calculated respectively. When the matching distance is less than the set distance threshold, the first feature point and the second feature point are determined to be a valid matching point pair, and the number of valid matching point pairs corresponding to the initial pose data is determined. Based on the number of valid matching point pairs corresponding to the initial pose data, a target number of candidate pose data are selected from the initial pose estimation set.

[0008] In one optional implementation, constructing a loss function using the intermediate rendered image, the intermediate mask image, the monocular image, and the target mask image includes: The monocular image is subtracted element-wise from the intermediate rendered image to obtain a first difference image; and the target mask image is subtracted element-wise from the intermediate mask image to obtain a second difference image. Using the set target region mask image, perform element-wise multiplication with the first difference image and the second difference image respectively to obtain the adjusted first difference image and the adjusted second difference image; A norm factor is calculated for the adjusted first difference image to obtain a first loss value, and a norm factor is calculated for the adjusted second difference image to obtain a second loss value; The loss function is obtained by multiplying the set image color weight factor by the first loss value and the set mask loss weight factor by the second loss value, and then adding the two products together.

[0009] In one optional implementation, determining the pose estimation data corresponding to the target to be tested from the plurality of optimized pose data includes: Determine the loss value corresponding to each of the optimized pose data; From the multiple optimized pose data, the optimized pose data with the smallest loss value is selected as the pose estimation data corresponding to the target to be tested.

[0010] In one optional implementation, the step of iteratively optimizing the intermediate pose data using the loss function to generate optimized pose data includes: The gradient of the loss function with respect to the attitude parameters is used to obtain attitude adjustment information; Based on the preset learning rate and the posture adjustment information, the intermediate pose data is adjusted to generate the adjusted pose data. The adjusted pose data is used as new intermediate pose data. The steps of generating intermediate rendered images and intermediate mask images corresponding to the intermediate pose data are returned until the iteration cutoff condition is met, and optimized pose data is generated.

[0011] In an optional implementation, after determining the pose estimation data corresponding to the target to be measured, the method further includes: Based on the set sampling range and exponential sampling strategy, multiple candidate learning rates are generated. For each candidate learning rate, the pose estimation data is iteratively optimized using the loss function based on the candidate learning rate to generate optimized pose estimation data; The optimized pose estimation data with the smallest loss value among multiple optimized pose estimation data is determined as the target pose data of the target to be tested in the monocular image.

[0012] Secondly, embodiments of this application also provide a pose estimation device, comprising: The acquisition module is used to acquire a monocular image, a target mask image corresponding to the target under test in the monocular image, and a three-dimensional model of the target under test; The first determining module is used to determine multiple view directions corresponding to the three-dimensional model, and sample multiple initial pose data of the target under test in each view direction; and determine the initial position data of the target under test under each initial pose data based on the target mask image; wherein the initial pose data and the initial position data constitute initial pose data, and multiple initial pose data constitute an initial pose estimation set; The adjustment module is used to select a target number of candidate pose data from the initial pose estimation set, and adjust the target number of candidate pose data to generate multiple intermediate pose data. The generation module is used to generate an intermediate rendered image and an intermediate mask image corresponding to each intermediate pose data using differential rendering technology, and then construct a loss function using the intermediate rendered image, the intermediate mask image, the monocular image and the target mask image, and use the loss function to iteratively optimize the intermediate pose data to generate optimized pose data. The second determining module is used to determine the pose estimation data corresponding to the target to be tested from multiple optimized pose data.

[0013] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described pose estimation method.

[0014] This application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described pose estimation method when executing the program.

[0015] The method provided in this application acquires a monocular image, a target mask image corresponding to the target in the monocular image, and a 3D model of the target. An initial pose estimation set is constructed using these data, and candidate pose data are selected from the initial pose estimation set. Several candidate pose data are adjusted to generate multiple intermediate pose data, achieving coarse adjustment of the candidate pose data. For each intermediate pose data, a loss function is constructed using the corresponding intermediate rendered image, intermediate mask image, monocular image, and target mask image. The loss function is then used to iteratively optimize the intermediate pose data, generating optimized pose data, achieving fine adjustment of the pose data. Finally, more accurate pose estimation data corresponding to the target can be determined from multiple optimized pose data.

[0016] It is evident that this application provides a pose estimation strategy based on the fusion of geometric principles and image information, realizing a complete closed-loop process from the generation of initial pose assumptions to fine optimization, without the participation of neural networks. While ensuring the accuracy of pose estimation, it also improves the interpretability, versatility and system stability of the method. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a pose estimation method in an exemplary embodiment of this application; Figure 2A This is a flowchart illustrating a pose estimation method in another exemplary embodiment of this application; Figure 2B Specifically shown Figure 2A A schematic diagram illustrating the determination of initial position data in the middle; Figure 3 This is a schematic diagram of a pose estimation device shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0021] Research shows that most 6D pose estimation techniques rely on deep learning models, which require a large amount of labeled data for training and are highly dependent on the prior information of the target object and the consistency of the distribution of training data during the inference stage.

[0022] For example, one related technology is a six-DOF pose estimation technique for a target object based on a Recurrent Neural Network (RNN) – Pose. This method, based on an initial pose estimation, uses an RNN architecture to iteratively optimize the pose of the RGB image and continuously corrects errors by introducing rendering feedback, achieving high-precision pose regression. Its core idea is: based on obtaining the initial pose of the target, an RNN structure is used to encode the image and the current pose state. Then, a synthetic image in the current pose is generated through a rendering module and compared with the real image to predict the pose update. After multiple rounds of recursive iteration, a precise pose estimation result is finally output. The main advantage of this method is that its optimization framework is applicable to pure RGB input and continuously improves pose accuracy through multiple iterations, making it suitable for situations where the initial estimate is not accurate enough.

[0023] Although RNNPose achieves good pose optimization results without depth information, its core technology heavily relies on deep neural network models, resulting in the following main drawbacks: First, it depends on training data; RNNPose requires training on large-scale, multimodal pose datasets to capture the complex relationships between target shape, texture, and pose changes. Once the target category or image distribution changes, the model performance significantly degrades, making it difficult to generalize to unknown new objects. Second, the model structure is complex, resulting in high computational and training overhead. RNNPose's network model structure and computational methods are relatively complex, making it difficult to meet the training and industrial deployment requirements of low-resource platforms. Therefore, while RNNPose can eliminate its dependence on depth information, its requirements for training data, model complexity, and computing power limit its universal applicability in real-world industrial environments.

[0024] Another related technology is the target object six-DOF pose estimation technique based on OnePose++. OnePose++ is an advanced technique for 6D pose estimation in sparse RGB images for new objects and scenes. This method trains a unified feature encoding network to perform local feature matching and pose estimation on a single image of a new target in a new scene, exhibiting a certain degree of generalization ability. Specifically, OnePose++ uses a pre-trained local feature extractor and a global graph optimization module. Given a model of the target object and a small number of reference images, it performs keypoint matching and PnP solving, and optimizes the final pose output through sparse reconstruction. The main highlight of this method is that it does not require retraining during the testing phase and still shows good accuracy on some unseen targets, making it representative of the generalization direction of 6D pose estimation.

[0025] Despite its progress in generalization performance, OnePose++ still suffers from the following significant shortcomings: First, its generalization ability relies on the training set. Although OnePose++ does not require retraining during the inference phase, its overall framework's generalization ability is built upon training data of large-scale objects and scenes, essentially making it a "training-dependent" method. Second, deploying new objects still requires preprocessing data. In practical applications, OnePose++ still needs to capture targets from multiple angles, construct reference atlases, and perform preprocessing to establish sparse keypoint matching relationships. This process places certain demands on data acquisition and cannot yet achieve truly "plug-and-play" deployment. Therefore, although OnePose++ has overcome its dependence on depth information and improved its adaptability to new objects to some extent, it still cannot achieve direct deployment that is completely training-independent and requires no data preprocessing, making it difficult to meet the dual requirements of efficiency and versatility in industrial settings.

[0026] Analysis reveals that in many practical applications, target objects are often diverse and lack labeled data, making it particularly difficult to collect large amounts of training data, especially in environments such as high-altitude operations, extreme scenarios, and unstructured outdoor environments. Furthermore, existing 6D pose estimation techniques heavily rely on depth information, but the performance of depth sensors degrades significantly under conditions of strong light interference, reflective materials, and long-distance or high-altitude shooting, severely limiting their applicability in complex outdoor or industrial scenarios. Therefore, achieving 6D pose estimation under conditions of insufficient training data and inability to obtain reliable depth information has become one of the key technical challenges urgently needing to be overcome in this field.

[0027] This application acquires a monocular image, a target mask image corresponding to the target in the monocular image, and a 3D model of the target. An initial pose estimation set is constructed using these data, and candidate pose data are selected from this set. Several candidate pose data are then adjusted to generate multiple intermediate pose data, achieving coarse adjustment of the candidate pose data. For each intermediate pose data, a loss function is constructed using the corresponding intermediate rendered image, intermediate mask image, monocular image, and target mask image. This loss function is then used to iteratively optimize the intermediate pose data, generating optimized pose data, thus achieving fine adjustment of the pose data. Finally, more accurate pose estimation data corresponding to the target can be determined from multiple optimized pose data.

[0028] As can be seen, this application proposes a solution for pose estimation without training or reference atlases. On the one hand, it overcomes the dependence on large-scale training data and deep neural networks in related technologies; on the other hand, it overcomes the problem that RGB-based methods still require the acquisition and processing of reference atlases, achieving "plug-and-play" capability that can be directly deployed on unknown new objects without preprocessing; and it also overcomes the problem that depth-based methods are limited in deployment in complex environments. The method proposed in this application does not rely on depth sensors at all and can adapt to deployment needs under conditions where depth information is difficult to obtain.

[0029] Specifically, this application provides a pose estimation strategy based on the fusion of geometric principles and image information, realizing a complete closed-loop process from the generation of initial pose assumptions to fine optimization, without the participation of neural networks. While ensuring the accuracy of pose estimation, it improves the interpretability, versatility and system stability of the method.

[0030] To facilitate understanding of this embodiment, a pose estimation method disclosed in this application will first be described in detail. The subject executing the pose estimation method provided in this application is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a computing device, an embedded device, etc. In some possible implementations, the pose estimation method can be implemented by the processor calling computer-readable instructions stored in memory.

[0031] See Figure 1 The diagram shows a flowchart of a pose estimation method provided in an embodiment of this application. The method includes steps S101 to S105, wherein: S101. Acquire a monocular image, a target mask image corresponding to the target to be tested in the monocular image, and a three-dimensional model of the target to be tested; S102. Determine multiple viewpoint directions corresponding to the three-dimensional model, and sample multiple initial pose data of the target under test in each viewpoint direction; and determine the initial position data of the target under test under each initial pose data based on the target mask image; wherein the initial pose data and the initial position data constitute initial pose data, and multiple initial pose data constitute an initial pose estimation set. S103. Select a target number of candidate pose data from the initial pose estimation set, and adjust the target number of candidate pose data to generate multiple intermediate pose data. S104. For each intermediate pose data, after generating the intermediate rendering image and intermediate mask image corresponding to the intermediate pose data using differential rendering technology, a loss function is constructed using the intermediate rendering image, the intermediate mask image, the monocular image and the target mask image, and the intermediate pose data is iteratively optimized using the loss function to generate optimized pose data. S105. Determine the pose estimation data corresponding to the target to be tested from multiple optimized pose data.

[0032] The pose estimation method proposed in this application does not rely on the training process of deep learning networks or the extraction of depth information. Based on monocular RGB images, it realizes six-degree-of-freedom pose estimation of target objects. It has good deployment versatility and environmental adaptability, and is particularly suitable for complex conditions in real industrial scenarios where targets are diverse, training data is scarce, or depth cameras are difficult to use.

[0033] The following sections will provide specific explanations of S101-S105.

[0034] Regarding S101: In implementation, the monocular image can be an image of the target object acquired from any scene, and the target object can be any object. By detecting the monocular image, a target mask image corresponding to the target object is generated. Furthermore, intelligent devices or design models can be used to generate a 3D model of the target object. For example, an intelligent device can scan the actual target object in the monocular image to generate a 3D model of that target object. Alternatively, if a 3D model of the target object exists in an industrial setting, it can be directly obtained. If other data, such as depth maps or binocular images, can be acquired in the application scenario, the 3D model of the target object can also be generated using these data. The process of determining the 3D model of the target object can be configured according to the actual situation; this is only an example. This allows for subsequent six-degree-of-freedom pose estimation of the target object based on the monocular image, the target mask image, and the 3D model of the target object.

[0035] Regarding S102: Based on monocular RGB images, this application proposes a pose hypothesis generation method that combines bounding box center estimation and image rendering feedback. This method uses the mask region of the target to estimate its initial 3D position and combines multi-view and image plane rotation sampling to generate a complete 6D pose hypothesis set (i.e., initial pose estimation set), which has both coverage and optimizability.

[0036] In practice, this application decomposes the three-dimensional pose into two dimensions: viewpoint and rotation in the image plane. Multiple initial poses are sampled in each of the two dimensions to construct an initial pose estimation set. In this way, it is theoretically possible to achieve full coverage of the initial pose, so as to provide basic data support for subsequently determining a more accurate pose result of the target to be tested.

[0037] Multiple viewpoint directions corresponding to the 3D model of the target object can be determined. For example, multiple vertices of the regular polyhedron surrounding the 3D model can be identified. Based on the coordinate information of the vertices and the center coordinates of the 3D model, the viewpoint direction is determined. This viewpoint direction represents the perspective from which the virtual camera points to the center of the 3D model. Alternatively, multiple viewpoint directions can be sampled using methods such as spherical uniform sampling, pyramid viewpoint strategy, playback of historical experience pose databases, and projection direction estimation based on coarse lighting inference.

[0038] Multiple planar rotation angles of the target under test are sampled in each view direction to obtain multiple initial attitude data, which constitute a candidate set of rotation attitudes. As shown below: in, Indicates the number of viewpoint directions. The number of plane rotation angles downsampled for each viewpoint direction. This represents the initial attitude data sampled.

[0039] For each initial pose data in the candidate set of rotational poses, its corresponding initial position data is determined in order to determine the 6D pose of the target under test, i.e., the initial pose data. For example, the initial position data of the target under test can be estimated based on the target mask image.

[0040] Optionally, the determination of the initial position data of the target under test based on the target mask image in this application specifically includes: Step a1: Determine the coordinate information of the center point of the bounding box of the target to be tested in the target mask image; Step a2: Based on the coordinate information of the center point of the bounding box of the target to be measured, the set estimated depth value, and the camera intrinsic parameter matrix, determine the estimated three-dimensional position of the target to be measured. Step a3: For each initial pose data, based on the first diagonal length of the bounding box of the target under test in the target mask image and the second diagonal length of the bounding box of the target under test in the rendered image corresponding to the initial pose data, the estimated three-dimensional position of the target under test is corrected to generate the initial position data of the target under test under the initial pose data.

[0041] In practice, for the target mask image of the target to be tested, the center coordinates of its minimum bounding rectangle are determined. This refers to the coordinates of the center point of the bounding box of the target in the target mask image, which is defined as the initial image center hypothesis: .

[0042] Considering that this center point is not necessarily equivalent to the actual position of the target object's center in the monocular image, we assume that the actual position of the target object's center in the monocular image is... Furthermore, there is a positional offset between the two. .

[0043] Furthermore, this application defines an estimated approximate depth, namely, a predicted depth value. The estimated depth value can be flexibly set according to the actual application scenario. For example, in an indoor factory scenario, assuming that the depth in the scenario is generally 0.5m to 3m, a depth value can be randomly selected from this range as the estimated depth value.

[0044] Therefore, based on the estimated depth value Camera intrinsic parameter matrix and the coordinates of the center point of the bounding box. Determine the estimated three-dimensional position of the target to be measured. .for example: Due to coordinates Not equivalent to the actual center coordinates However, using the actual center coordinates Calculated three-dimensional position It should be: Therefore, when the depth information is the same (i.e., under the assumed estimated depth value), there is a spatial offset between the estimated 3D position and the 3D position corresponding to the true center. .

[0045] Considering that there is a deviation between the assumed estimated depth value and the actual depth value, a correction ratio can be determined to coarsely correct the estimated three-dimensional position.

[0046] Furthermore, this application can render a rendered image corresponding to the initial pose data based on the initial pose data and the estimated 3D position, and determine the bounding box of the target to be tested in the rendered image corresponding to the initial pose data, so as to determine the length of the first diagonal of the bounding box of the target to be tested in the target mask image. And the length of the second diagonal of the bounding box of the target in the rendered image corresponding to the initial pose data. Determine the correction ratio to utilize the correction ratio and the estimated depth value. Estimate the true depth .

[0047] Furthermore, assume that the center point of the smallest bounding rectangle determined by the mask in the rendered image corresponding to the initial pose data is... The actual center of the 3D model of the target object is located in the rendered image. Since this application estimates the three-dimensional position based on the initial attitude data... The rendered model, and Since this relates to the actual pose of the model's center, it is clear that the association is important. In reality, they are collinear. The ideal center should satisfy the following conditions, derived through rendering constraints: in , , , For camera internal references The components can be obtained Based on the rendered geometry, the conversion relationship between deviation and depth can be obtained: This leads to a more accurate three-dimensional position. for: Assuming the offset is small, its calculation can be approximated as follows: The correction ratio can be obtained by using the lengths of the first and second diagonals. This correction ratio can then be used to estimate the three-dimensional position of the target under test. By performing calibration, more accurate initial position data of the target can be obtained. .

[0048] The above process estimates the initial position data in the camera coordinate system by using the center of the minimum bounding rectangle. In practice, other estimation methods can also be used, such as the geometric center of the target area, depth estimation generated by model reprojection matching, fixed height / distance strategy, etc., to determine the initial position data.

[0049] After determining the initial position corresponding to each initial attitude data, the initial attitude data and its corresponding initial position data constitute the initial pose data, i.e., the six-degree-of-freedom pose.

[0050] An initial pose estimation set is constructed using multiple initial pose data. This results in the final complete set of pose assumptions: The process of generating the initial pose estimation set described above in this application combines image information, camera model, and differentiable rendering feedback, and has clear geometric meaning and good practical adaptability, providing a high-quality initial solution space for subsequent pose selection and optimization.

[0051] Regarding S103: To improve pose adjustment efficiency and more efficiently determine the six-DOF pose of the target under test, a number of candidate pose data can be selected from the initial pose estimation set, such as randomly selecting a number of candidate pose data.

[0052] Optionally, selecting a target number of candidate pose data from the initial pose estimation set includes: Step b1: For each initial pose data in the initial pose estimation set, generate an initial rendering image corresponding to the initial pose data; and use the set target operator to extract feature points from the monocular image and the initial rendering image respectively, to obtain multiple first feature points included in the monocular image and multiple second feature points included in the initial rendering image; calculate the matching distance in the feature space between the first feature points and the matched second feature points respectively. Step b2: When the matching distance is less than the set distance threshold, determine the first feature point and the second feature point as a valid matching point pair, and determine the number of valid matching point pairs corresponding to the initial pose data; Step b3: Select a target number of candidate pose data from the initial pose estimation set based on the number of valid matching point pairs corresponding to the initial pose data.

[0053] In this application, S102 generated the following: × To reduce subsequent computational load while maintaining pose estimation accuracy, this application introduces a self-consistent screening mechanism based on feature consistency to select a target number of candidate pose data from multiple initial pose data for subsequent pose adjustment.

[0054] In practice, for each initial pose data in the initial pose estimation set, an initial rendered image corresponding to the initial pose data is generated. For example, a 3D model under the initial pose data can be rendered to obtain an initial rendered image. Then, using set target operators such as Oriented FAST and Rotated BRIEF (ORB) operators, Scale-Invariant Feature Transform (SIFT) operators, SuperPoint operators, and accelerated KAZE feature operators, the first feature points included in the monocular image and the second feature points included in the initial rendered image are extracted. The first and second feature points are then matched; that is, for each first feature point, the matching distance between the first feature point and the matched second feature point in the feature space is calculated, thus obtaining the matching distance of multiple feature point pairs in high-dimensional space. Each feature point pair includes a first feature point and its matched second feature point.

[0055] When the matching distance is less than the set distance threshold, the first feature point and the second feature point are determined to be a valid matching point pair; conversely, if the matching distance is greater than or equal to the distance threshold, the first feature point and the second feature point are determined to be an invalid matching point pair. The number of valid matching point pairs corresponding to this initial pose data can then be counted, which is equivalent to obtaining the number of valid matching point pairs corresponding to each initial pose data in the initial pose estimation set.

[0056] The number of valid matching point pairs is used as a filtering criterion to select a target number of candidate pose data from the initial pose estimation set. For example, the target number of candidate pose data can be selected from the initial pose estimation set in descending order of the number of valid matching point pairs. This target number of candidate pose data constitutes the candidate set. The target number can be flexibly set according to business needs.

[0057] After selecting the target number of candidate pose data, coarse adjustments can be made to the candidate pose data. For example, Random Sample Consensus (RANSAC) and pose solving algorithms can be used to robustly solve the candidate pose data in the candidate set to eliminate interference from incorrect matching points, thereby adjusting the candidate pose data and obtaining intermediate pose data with higher accuracy. Pose solving algorithms used in implementation include Efficient Perspective-n-Point (EPnP), Direct Linear Transformation (DLS), Uncalibrated Perspective-n-Point (UPnP), and Pose from Orthography and Scaling with Iterations (POSIT).

[0058] The self-consistent selection mechanism based on feature consistency proposed in this application has the following advantages: First, the more effective matching points there are, the higher the similarity between the pose data and the real target to be measured, and the higher the accuracy of the pose data. Therefore, it is possible to select candidate pose data with higher accuracy. Second, random sampling consistency and pose solving algorithms such as RANSAC-EPnP can effectively optimize the pose assumption (i.e., candidate pose data) with large initial errors, thus improving robustness. Specifically, in this application, candidate pose data are selected according to the number of effective matching point pairs. That is, there are more effective matching point pairs (i.e., high-quality matching points) in the candidate pose data. The algorithm has better performance when there are more high-quality matching points. Even if the pose assumption error is large, the error can be optimized away when there are more high-quality matching points. Therefore, after selecting candidate poses using the number of effective matching point pairs, this application can use the above algorithm to achieve more accurate adjustment of the candidate pose data, thus achieving robust and high-precision pose estimation. Finally, the scoring and selection reduces the computational burden of subsequent optimization.

[0059] Regarding S104: After adjusting a target number of candidate pose data to generate a target number of intermediate pose data, the intermediate pose data can be iteratively optimized multiple times to obtain optimized pose data with higher accuracy. For example, differential rendering technology can be used to fine-tune the pose of the intermediate pose data. Specifically, pose parameters can be treated as optimizable variables and introduced into a differentiable renderer to render intermediate rendered images containing corresponding gradient information. The gradients can then be automatically solved based on these intermediate rendered images, thereby achieving a continuously differentiable pose optimization process.

[0060] During implementation, for each intermediate pose data, based on the intermediate pose data... 3D model Camera intrinsic parameter matrix Use a differentiable rendering engine (denoted as ) ), generate the intermediate rendered image and intermediate mask image corresponding to the intermediate pose data, such as The intermediate rendered images generated using differential rendering technology contain corresponding gradient information, which allows for iterative optimization using these intermediate rendered images.

[0061] Using the intermediate rendered image, intermediate mask image, monocular image and target mask image corresponding to the intermediate pose data, a loss function with pose parameters as variable parameters is constructed.

[0062] Optionally, constructing a loss function using the intermediate rendered image, intermediate mask image, monocular image, and target mask image corresponding to the intermediate pose data includes: subtracting the monocular image from the intermediate rendered image element-wise to obtain a first difference image; subtracting the target mask image from the intermediate mask image element-wise to obtain a second difference image; multiplying the set target region mask image element-wise with the first difference image and the second difference image respectively to obtain an adjusted first difference image and an adjusted second difference image; calculating a norm factor for the adjusted first difference image to obtain a first loss value, and calculating a norm factor for the adjusted second difference image to obtain a second loss value; multiplying a set image color weight factor with the first loss value, and multiplying a set mask loss weight factor with the second loss value, and adding the two products to obtain the loss function.

[0063] For example, the constructed loss function As shown below: in, In Indicates attitude parameters, This indicates the target area mask image to be set. The target area mask image can be flexibly set according to business needs. Represents a monocular image. This indicates an intermediate rendered image. Represents the target mask image. Represents the intermediate mask image. This represents the Hadamard product, which is an element-wise multiplication. Represents the image color weighting factor. This represents the mask loss weighting factor, used to balance the importance of both; where The first difference image is obtained. The second difference image is obtained.

[0064] The gradient of the loss function with respect to the pose parameters is calculated, and the pose data is iteratively optimized using the gradient descent method to generate optimized pose data.

[0065] Optionally, the step of iteratively optimizing the intermediate pose data using the loss function to generate optimized pose data includes: calculating the gradient of the loss function with respect to the pose parameters to obtain pose adjustment information; adjusting the intermediate pose data based on a preset learning rate and the pose adjustment information to generate adjusted pose data; using the adjusted pose data as new intermediate pose data, and returning to the step of generating intermediate rendered images and intermediate mask images corresponding to the intermediate pose data, until the iteration cutoff condition is met to generate optimized pose data.

[0066] In implementation, for each intermediate pose data point, a loss function is constructed with pose parameters as variables. The gradient of this loss function with respect to the pose parameters is then calculated to obtain pose adjustment information. The preset learning rate is then multiplied by the pose adjustment information to obtain an intermediate product result. This intermediate pose data is then processed. Subtract the intermediate product results to generate the adjusted pose data. ,Right now , It has a fixed preset learning rate, which can be flexibly set according to business needs.

[0067] The adjusted pose data is used as the new intermediate pose data. The steps for generating the intermediate rendered image and intermediate mask image corresponding to the intermediate pose data are returned. The optimization operation is repeated multiple times until the iteration cutoff condition is met, such as the number of iterations equals the set threshold, and the optimized pose data is generated.

[0068] In practice, the above iterative optimization process can be performed in parallel on multiple intermediate pose data to obtain optimized pose data corresponding to each intermediate pose data. In the initial stage of optimization, this application uses the same small learning rate for each intermediate pose data to ensure the stability and convergence of the optimization process.

[0069] Regarding S105: After obtaining multiple optimized pose data, the pose estimation data corresponding to the target to be tested can be determined from the multiple optimized pose data. For example, the pose data corresponding to the minimum loss can be selected as the pose estimation data of the target to be tested.

[0070] Optionally, determining the pose estimation data corresponding to the target to be tested from multiple optimized pose data includes: determining the loss value corresponding to each optimized pose data; and selecting the optimized pose data with the smallest loss value from multiple optimized pose data as the pose estimation data corresponding to the target to be tested.

[0071] During implementation, the loss value corresponding to each optimized pose data can be determined according to the loss function, and multiple optimized pose data (i.e., forming a candidate pose set) can be selected. In the process, the optimized pose data with the smallest loss value is selected as the pose estimation data corresponding to the target to be tested.

[0072] After multiple iterations, the candidate with the minimum loss is selected as the current optimal pose estimate. .

[0073] Optionally, a specific number of optimized pose data can be selected in ascending order of loss value, or optimized pose data with loss values ​​less than a preset loss threshold can be selected. The selected optimized pose data are then output so that the user can determine the pose estimation data corresponding to the target to be tested from the selected optimized pose data.

[0074] This optimization iterative process is called Multi-Init Pose Optimization (MIPO), which can effectively avoid local optima traps and improve pose convergence.

[0075] While the MIPO optimization iterative process is used to obtain better pose estimation results, this application proposes a Multi-Scale Pose Optimization (MSPO) strategy to further improve the accuracy and robustness of the final pose estimation. Specifically, this application employs an exponential sampling strategy to construct a set of learning rates at different scales. Under different learning rates, the determined pose estimation data is iterated multiple times, and the optimal pose estimation result is selected as the target pose data for the target object.

[0076] Optionally, after determining the pose estimation data corresponding to the target to be tested, the method further includes: generating multiple candidate learning rates based on the set sampling range and exponential sampling strategy; for each candidate learning rate, iteratively optimizing the pose estimation data using the loss function based on the candidate learning rate to generate optimized pose estimation data; and determining the optimized pose estimation data with the smallest loss value among the multiple optimized pose estimation data as the target pose data of the target to be tested in the monocular image.

[0077] The sampling range can be set according to business needs. This application employs an exponential sampling strategy to construct a set of learning rates at different scales. ,for example: Each learning rate used , The sampling range is set.

[0078] pose estimation data As the initial pose, the pose estimation data is processed using each learning rate in the learning rate set. Gradient optimization was performed again, resulting in a set of pose estimation results. (i.e., multiple optimized pose estimation data). In the pose estimation data... When there are multiple learning rates, each learning rate in the set can be used to perform parallel estimations of multiple pose data. Gradient optimization is then performed again. The gradient optimization process involves calculating the gradient of the loss function with respect to the attitude parameters and iteratively optimizing the attitude using the gradient descent method. For details, please refer to the previous explanation, which will not be elaborated here.

[0079] Finally, the optimized pose estimation data with the minimum loss value is selected as the final output pose, which is also the target pose data of the target in the monocular image. : This application combines the MIPO and MSPO strategies, which not only converges quickly to high-quality estimation results among multiple candidate solutions, but also improves the adaptability to perturbations of different scales through multi-scale optimization, thereby ensuring that the attitude estimation results, i.e. the target pose data, have stronger robustness and accuracy.

[0080] The method proposed in this application constructs a pose estimation process based on the geometric relationship between images and 3D models, without involving neural network training and feature learning, nor relying on depth sensor information. Through a three-stage design of initial hypothesis-coarse matching screening-differential fine-tuning, a high-precision, high-efficiency, and robust pose estimation process is achieved, suitable for various rigid object recognition and deployment tasks lacking training data and depth information.

[0081] The advantage of this application lies in its ability to achieve high-precision pose estimation without relying on depth information. However, the proposed method has good scalability and can be extended to application scenarios where depth information exists. In such cases, the center position of the real target point cloud can be directly obtained using the depth map to replace approximate depth estimation; alternatively, the point cloud can be used for initial position correction, error term weighting, 3D feature matching, etc., to further improve accuracy and convergence speed. The method of this application is well-compatible with depth information, suitable for a direct enhancement version under RGBD systems, and retains the advantages of training independence and strong interpretability.

[0082] Optionally, specific steps of the present application's solution can be replaced using a neural network model to extend the solution. For example, feature matching can be used to select candidate pose data, or the learning rate can be adjusted based on the neural network to achieve iterative optimization of pose estimation data using multiple learning rates. Therefore, all other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0083] See Figure 2A , combined Figure 2A The method proposed in this application is illustrated by example. The method includes: Figure 2A The process shown in section a is the attitude hypothesis generation process. For clarity, Figure 2B Specifically shown Figure 2A A schematic diagram illustrating the determination of initial position data. Specifically, multiple viewpoints corresponding to the 3D model can be determined, and multiple initial pose data can be sampled at each viewpoint. For each initial pose data, its corresponding initial position data is determined. Then, based on the initial pose data and the corresponding initial position data, initial pose data can be constructed. Multiple initial pose data constitute a coarse pose hypothesis H1, which is the pose hypothesis set described above. The specific implementation process can be found in the aforementioned detailed explanation of S102.

[0084] like Figure 2A The process shown in section b is the second part: pose selection and coarse optimization. Specifically, after rendering the 3D model based on the initial pose data to obtain the RGB rendered image (i.e., the initial rendered image), the ORB operator can be used to extract feature points from the monocular image and the RGB rendered image, and then match them. If the matching distance is less than the preset matching distance, it is determined to be a valid matching point pair. Using the number of valid matching point pairs as the scoring index, the top K most representative pose hypotheses (i.e., the target number of candidate pose data) are selected from the coarse pose hypotheses H1. For the selected pose hypotheses, the RANSAC-EPnP algorithm is used for coarse pose optimization. The coarsely optimized pose data constitute the candidate pose set H. K .

[0085] like Figure 2A The process shown in Figure c is the third step: fine-tuning of differential rendering. First, the Multi-Init Pose Optimization (MIPO) strategy is used to refine the candidate pose set H. KEach candidate pose data point is iteratively optimized using gradient descent to obtain optimized pose data, achieving fine-tuning of the pose data. The pose data corresponding to the minimum loss value is selected from the K optimized pose data points as the pose estimation data for the target object. Then, using the Multi-Scale Pose Optimization (MSPO) strategy, a set of learning rates at different scales is constructed. Each learning rate is used to iteratively optimize the pose estimation data for the target object, generating multiple optimized pose estimation data points. Finally, the optimized pose estimation data with the minimum loss value is selected as the output pose of this application, i.e., the target pose data for the target object.

[0086] The pose estimation method proposed in this application does not rely on any training data or neural network model. It is based entirely on image masks and geometric relationships for pose estimation, and is applicable to any new object. It can be deployed directly without the need to collect reference atlases in advance. Furthermore, the pose estimation method proposed in this application does not rely on depth sensors and only uses monocular RGB images, making it suitable for complex environments where depth sensors are difficult to use, such as strong light interference, reflective materials, and long distances.

[0087] Furthermore, this application effectively improves the accuracy and robustness of pose estimation through a multi-initial-value and multi-scale optimization strategy, and has good tolerance for initial estimation errors. Moreover, it does not require large-scale training or model customization, the calculation process is relatively simple and efficient, requires less computing resources, has high deployment efficiency, and can be directly integrated into various application scenarios such as robotics, industrial inspection, and augmented reality.

[0088] The key technology of this application lies in proposing a six-DOF pose estimation method based solely on monocular RGB images, requiring no training or depth information. The initial position is determined through a target mask and camera intrinsic parameters, and a multi-view and multi-angle pose sampling strategy is combined to generate a high-coverage candidate pose set. Subsequently, image feature matching and the RANSAC-EPnP algorithm are used for coarse optimization and screening. Based on this, a differentiable rendering mechanism is introduced to finely optimize the pose using gradient descent. Furthermore, the optimization process incorporates multiple initial value optimization (MIPO) and multi-scale optimization (MSPO) strategies, significantly improving the accuracy and stability of the estimation.

[0089] The method presented in this application is applicable to direct deployment in unknown new objects and scenarios, and can be naturally extended to situations where depth information exists, maintaining the method's structure and advantages. Therefore, this application possesses good scalability and general value in practical engineering applications, demonstrating clear technological innovation and application prospects.

[0090] Corresponding to the embodiments of the aforementioned pose estimation method, this application also provides embodiments of a pose estimation device. Figure 3A schematic diagram of the pose estimation device provided in this application specifically includes: The acquisition module 301 is used to acquire a monocular image, a target mask image corresponding to the target to be tested in the monocular image, and a three-dimensional model of the target to be tested; The first determining module 302 is used to determine multiple view directions corresponding to the three-dimensional model, and sample multiple initial pose data of the target under test in each view direction; and determine the initial position data of the target under test under each initial pose data based on the target mask image; wherein the initial pose data and the initial position data constitute initial pose data, and multiple initial pose data constitute an initial pose estimation set. The adjustment module 303 is used to select a target number of candidate pose data from the initial pose estimation set, and adjust the target number of candidate pose data to generate multiple intermediate pose data. The generation module 304 is used to generate an intermediate rendering image and an intermediate mask image corresponding to each intermediate pose data using differential rendering technology, and then construct a loss function using the intermediate rendering image, the intermediate mask image, the monocular image and the target mask image, and use the loss function to iteratively optimize the intermediate pose data to generate optimized pose data. The second determining module 305 is used to determine the pose estimation data corresponding to the target to be tested from multiple optimized pose data.

[0091] In an optional implementation, the first determining module 302, when determining the initial position data of the target under test for each initial pose data based on the target mask image, is used to: Determine the coordinate information of the center point of the bounding box of the target to be tested in the target mask image; Based on the coordinate information of the center point of the bounding box of the target under test, the set estimated depth value, and the camera intrinsic parameter matrix, the estimated three-dimensional position of the target under test is determined. For each initial pose data, the estimated three-dimensional position of the target is corrected based on the first diagonal length of the bounding box of the target in the target mask image and the second diagonal length of the bounding box of the target in the rendered image corresponding to the initial pose data, thereby generating the initial position data of the target under the initial pose data.

[0092] In an optional implementation, the adjustment module 303, when selecting a target number of candidate pose data from the initial pose estimation set, is used to: For each initial pose data in the initial pose estimation set, an initial rendered image corresponding to the initial pose data is generated; and feature points are extracted from the monocular image and the initial rendered image using the set target operator, respectively, to obtain multiple first feature points included in the monocular image and multiple second feature points included in the initial rendered image; and the matching distance between the first feature points and the matched second feature points in the feature space is calculated respectively. When the matching distance is less than the set distance threshold, the first feature point and the second feature point are determined to be a valid matching point pair, and the number of valid matching point pairs corresponding to the initial pose data is determined. Based on the number of valid matching point pairs corresponding to the initial pose data, a target number of candidate pose data are selected from the initial pose estimation set.

[0093] In an optional implementation, the generation module 304, when constructing a loss function using the intermediate rendered image, the intermediate mask image, the monocular image, and the target mask image, is used to: The monocular image is subtracted element-wise from the intermediate rendered image to obtain a first difference image; and the target mask image is subtracted element-wise from the intermediate mask image to obtain a second difference image. Using the set target region mask image, perform element-wise multiplication with the first difference image and the second difference image respectively to obtain the adjusted first difference image and the adjusted second difference image; A norm factor is calculated for the adjusted first difference image to obtain a first loss value, and a norm factor is calculated for the adjusted second difference image to obtain a second loss value; The loss function is obtained by multiplying the set image color weight factor by the first loss value and the set mask loss weight factor by the second loss value, and then adding the two products together.

[0094] In an optional implementation, the second determining module 305, when determining the pose estimation data corresponding to the target to be tested from the plurality of optimized pose data, is used to: Determine the loss value corresponding to each of the optimized pose data; From the multiple optimized pose data, the optimized pose data with the smallest loss value is selected as the pose estimation data corresponding to the target to be tested.

[0095] In one optional implementation, the generation module 304, when iteratively optimizing the intermediate pose data using the loss function to generate optimized pose data, is used to: The gradient of the loss function with respect to the attitude parameters is used to obtain attitude adjustment information; Based on the preset learning rate and the posture adjustment information, the intermediate pose data is adjusted to generate the adjusted pose data. The adjusted pose data is used as new intermediate pose data. The steps of generating intermediate rendered images and intermediate mask images corresponding to the intermediate pose data are returned until the iteration cutoff condition is met, and optimized pose data is generated.

[0096] In an optional implementation, after determining the pose estimation data corresponding to the target to be measured, the device further includes: an optimization module 306, used for: Based on the set sampling range and exponential sampling strategy, multiple candidate learning rates are generated. For each candidate learning rate, the pose estimation data is iteratively optimized using the loss function based on the candidate learning rate to generate optimized pose estimation data; The optimized pose estimation data with the smallest loss value among multiple optimized pose estimation data is determined as the target pose data of the target to be tested in the monocular image.

[0097] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0098] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0099] This application also provides a computer-readable storage medium storing a computer program that can be used to execute the pose estimation method described in the above embodiments.

[0100] This application also provides a computer device, see [link to relevant documentation] Figure 4The diagram shown is a structural schematic of the computer device provided in this application. At the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the pose estimation method described in the above embodiments. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0105] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0106] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, and optical disc read-only memory (ODROM). Computer-readable media may be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transient media such as modulated data signals and carrier waves.

[0107] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0108] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0109] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0110] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A pose estimation method, characterized in that, The method includes: Acquire a monocular image, a target mask image corresponding to the target under test in the monocular image, and a three-dimensional model of the target under test; Multiple viewpoint directions corresponding to the 3D model are determined, and multiple initial pose data of the target under test are sampled under each viewpoint direction; and based on the target mask image, the initial position data of the target under test under each initial pose data is determined; wherein the initial pose data and the initial position data constitute initial pose data, and multiple initial pose data constitute an initial pose estimation set; Select a target number of candidate pose data from the initial pose estimation set, and adjust the target number of candidate pose data to generate multiple intermediate pose data; For each intermediate pose data, after generating the intermediate rendered image and intermediate mask image corresponding to the intermediate pose data using differential rendering technology, a loss function is constructed using the intermediate rendered image, the intermediate mask image, the monocular image and the target mask image, and the intermediate pose data is iteratively optimized using the loss function to generate optimized pose data. The pose estimation data corresponding to the target to be tested is determined from multiple optimized pose data.

2. The method according to claim 1, characterized in that, The step of determining the initial position data of the target under test based on the target mask image for each initial pose data includes: Determine the coordinate information of the center point of the bounding box of the target to be tested in the target mask image; Based on the coordinate information of the center point of the bounding box of the target under test, the set estimated depth value, and the camera intrinsic parameter matrix, the estimated three-dimensional position of the target under test is determined. For each initial pose data, the estimated three-dimensional position of the target is corrected based on the first diagonal length of the bounding box of the target in the target mask image and the second diagonal length of the bounding box of the target in the rendered image corresponding to the initial pose data, thereby generating the initial position data of the target under the initial pose data.

3. The method according to claim 1, characterized in that, The step of selecting a target number of candidate pose data from the initial pose estimation set includes: For each initial pose data in the initial pose estimation set, an initial rendered image corresponding to the initial pose data is generated; and feature points are extracted from the monocular image and the initial rendered image using the set target operator, respectively, to obtain multiple first feature points included in the monocular image and multiple second feature points included in the initial rendered image; and the matching distance between the first feature points and the matched second feature points in the feature space is calculated respectively. When the matching distance is less than the set distance threshold, the first feature point and the second feature point are determined to be a valid matching point pair, and the number of valid matching point pairs corresponding to the initial pose data is determined. Based on the number of valid matching point pairs corresponding to the initial pose data, a target number of candidate pose data are selected from the initial pose estimation set.

4. The method according to claim 1, characterized in that, The step of constructing a loss function using the intermediate rendered image, the intermediate mask image, the monocular image, and the target mask image includes: The monocular image is subtracted element-wise from the intermediate rendered image to obtain a first difference image; and the target mask image is subtracted element-wise from the intermediate mask image to obtain a second difference image. Using the set target region mask image, perform element-wise multiplication with the first difference image and the second difference image respectively to obtain the adjusted first difference image and the adjusted second difference image; A norm factor is calculated for the adjusted first difference image to obtain a first loss value, and a norm factor is calculated for the adjusted second difference image to obtain a second loss value; The loss function is obtained by multiplying the set image color weight factor by the first loss value and the set mask loss weight factor by the second loss value, and then adding the two products together.

5. The method according to claim 1, characterized in that, The step of determining the pose estimation data corresponding to the target to be tested from multiple optimized pose data includes: Determine the loss value corresponding to each of the optimized pose data; From the multiple optimized pose data, the optimized pose data with the smallest loss value is selected as the pose estimation data corresponding to the target to be tested.

6. The method according to claim 1, characterized in that, The step of iteratively optimizing the intermediate pose data using the loss function to generate optimized pose data includes: The gradient of the loss function with respect to the attitude parameters is used to obtain attitude adjustment information; Based on the preset learning rate and the posture adjustment information, the intermediate pose data is adjusted to generate the adjusted pose data. The adjusted pose data is used as new intermediate pose data. The steps of generating intermediate rendered images and intermediate mask images corresponding to the intermediate pose data are returned until the iteration cutoff condition is met, and optimized pose data is generated.

7. The method according to any one of claims 1-6, characterized in that, After determining the pose estimation data corresponding to the target to be measured, the method further includes: Based on the set sampling range and exponential sampling strategy, multiple candidate learning rates are generated. For each candidate learning rate, the pose estimation data is iteratively optimized using the loss function based on the candidate learning rate to generate optimized pose estimation data; The optimized pose estimation data with the smallest loss value among multiple optimized pose estimation data is determined as the target pose data of the target to be tested in the monocular image.

8. A pose estimation device, characterized in that, The device includes: The acquisition module is used to acquire a monocular image, a target mask image corresponding to the target under test in the monocular image, and a three-dimensional model of the target under test; The first determining module is used to determine multiple view directions corresponding to the three-dimensional model, and sample multiple initial pose data of the target under test in each view direction; and determine the initial position data of the target under test under each initial pose data based on the target mask image; wherein the initial pose data and the initial position data constitute initial pose data, and multiple initial pose data constitute an initial pose estimation set; The adjustment module is used to select a target number of candidate pose data from the initial pose estimation set, and adjust the target number of candidate pose data to generate multiple intermediate pose data. The generation module is used to generate an intermediate rendered image and an intermediate mask image corresponding to each intermediate pose data using differential rendering technology, and then construct a loss function using the intermediate rendered image, the intermediate mask image, the monocular image and the target mask image, and use the loss function to iteratively optimize the intermediate pose data to generate optimized pose data. The second determining module is used to determine the pose estimation data corresponding to the target to be tested from multiple optimized pose data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor performs the steps of the method according to any one of claims 1-7.

Images