Multi-view object pose estimation and pose optimization method and electronic device

Abstract

This invention discloses a multi-view object pose estimation and optimization method and electronic device. The method includes: acquiring RGB images from multiple viewpoints; extracting features from the images of each viewpoint using an image encoding module to obtain feature maps for each viewpoint; inputting the feature maps and corresponding camera poses into a multi-view pose estimation module for pose estimation to obtain the initial pose of the object; inputting the pose estimation results, feature maps, and corresponding camera poses into a pose optimization module for pose optimization to obtain the pose residuals of the object; and obtaining the final object pose for each viewpoint based on the initial pose and pose residuals. This invention directly models multi-view images using a deformer network, solving for the object's pose in one step, replacing the previous two-stage solution process. This makes the overall framework simpler and more efficient, achieving a dual optimization of accuracy and efficiency. This invention can be widely applied in the interdisciplinary fields of computer vision and robotics.

Description

Technical Field

[0001] This invention relates to the intersection of computer vision and robotics, and in particular to a method and electronic device for multi-view object pose estimation and optimization. Background Technology

[0002] Current object pose estimation methods mainly include: pose estimation methods based on single-view RGB input and pose estimation methods based on multi-view RGB input. These two types of methods are analyzed in detail below:

[0003] (1) Pose estimation method based on single-view RGB input

[0004] 1) Direct Regression Methods. Object pose estimation aims to determine the rotation and translation parameters of an object in an image relative to the optical center of the camera that took the picture. Methods based on single-view direct regression for pose estimation, such as PoseCNN and GDR-Net, solve for the translation component of the pose by first regressing the 2D projected coordinates and depth value of the object's center, and then solving for the object's translation relative to the camera through backprojection. However, since recovering depth information from a single-view RGB image is inherently an underdetermined problem, this process is practically impossible according to the principles of perspective geometry. Therefore, these methods often overfit to data from specific scenes, lacking a sufficient theoretical explanation for their effectiveness.

[0005] 2) Methods for solving correspondences. Another method for estimating object pose from a single-view RGB image relies on solving the correspondences of points. This involves first determining the positions of the object's 2D keypoints, and then using the PnP algorithm to solve for the object's pose. Representative algorithms in this category include BB8, Keypose, and PVNet. However, for objects with insufficient keypoint features, such as transparent or translucent objects, objects with highly reflective surfaces, and objects with self-occlusion, this method often proves unsatisfactory. The fundamental reason is that accurately identifying and locating these keypoints becomes extremely difficult when their features are not prominent.

[0006] (2) Pose estimation method based on multi-view RGB input

[0007] Multi-view pose estimation refers to the process of recovering an object's pose from images taken from different angles. Existing multi-view RGB pose estimation methods, such as MV6D, MV-Render, and CosyPose, typically follow a two-stage process to solve the pose estimation problem. In the first stage, these methods process the input from each viewpoint independently, using a single-view approach to estimate a preliminary result, which may be the object's initial pose or the position of 2D keypoints. In the second stage, the intermediate results obtained in the first stage are used for joint optimization to finally determine the object's pose. Nevertheless, this two-stage approach still faces certain challenges. On the one hand, in the first stage, because the processing of each viewpoint is independent, it inherits the limitations of the single-view approach and fails to fully utilize the potential correlations and constraints between multiple views. On the other hand, in the iterative optimization process of the second stage, the high computational complexity leads to a long overall solution time, making it difficult to meet the requirements of real-time pose estimation. Summary of the Invention

[0008] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a multi-view object pose estimation and pose optimization method, electronic device and medium.

[0009] The first technical solution adopted in this invention is:

[0010] A method for multi-view object pose estimation and optimization includes the following steps:

[0011] Obtain RGB images from N perspectives

[0012] Images from each viewpoint are processed by an image encoding module. Feature extraction is performed to obtain feature maps {F1, F2, ..., F} for each viewpoint. B};

[0013] The feature map {F1,F2,…,F B The initial pose of the object is obtained by inputting the camera pose corresponding to the feature map into the multi-view pose estimation module.

[0014] The pose estimation results, feature maps, and the corresponding camera poses are input into the pose optimization module for pose optimization to obtain the object's pose residuals.

[0015] Based on the initial attitude and attitude residual Obtain the object pose {P1, P2, ..., P} from each viewpoint. N}; where each object pose P includes a three-dimensional rotation R∈SO(3) from the object coordinate system to the camera coordinate system and a three-dimensional translation.

[0016] Furthermore, the image encoding module is implemented using a deep neural network.

[0017] Furthermore, in the multi-view pose estimation module and pose optimization module, a Transformer network is used to model the intrinsic relationship between different viewpoints, and two learnable pose tokens are introduced for learning rotation and translation.

[0018] Furthermore, the multi-view pose estimation module operates as follows:

[0019] Two-dimensional feature maps F from different perspectives i i Perform feature reduction and flatten it into a shape with K 0 A one-dimensional token sequence of tokens

[0020] Introduce two learnable tokens and The learning of rotation and translation in pose is used, so for each viewpoint, K will eventually be obtained. 0 +2 tokens, that is

[0021] The token sequence from all N perspectives Concatenate along the direction of the sequence and input it into L. 0 Features are updated and interacted in a Transformer network composed of stacked layers, enabling feature fusion between multiple views;

[0022] Two tokens related to posture learning and The values ​​are separated and input into two multilayer perceptron (MLP) networks to regress the rotation of the object at each viewpoint. Peaceful relocation Obtain the initial orientation of the object.

[0023] Furthermore, it also includes the step of encoding camera extrinsic and intrinsic parameters:

[0024] Encoding of camera extrinsic parameters: For each viewpoint, align it to the first viewpoint and calculate the camera relative transformation from each viewpoint to the first viewpoint; then flatten the extrinsic parameter transformation matrix into an extrinsic parameter vector, and use a multilayer perceptron (MLP) to flatten the extrinsic parameter vector to the same dimension as the token D. 0 With the same dimensionality, the extrinsic position encoding for each viewpoint is obtained. Then encode the extrinsic parameter positions. The token sequence is directly added to each viewpoint i. On each token;

[0025] Encoding of camera intrinsics: For each viewpoint, the camera intrinsics are encoded by encoding each token. The corresponding pixel coordinates in the image Perform perspective backprojection transformation to obtain and Then and The data is concatenated and input into a multilayer perceptron (MLP) to be encoded into the token dimension D. 0 With the same dimensional size, the positional encoding of each token with respect to the camera intrinsics is obtained. Finally Added to each token; where C x C y is the principal point coordinate of the camera, and f is the focal length of the camera.

[0026] Furthermore, attitude estimation also includes the following steps:

[0027] The token sequence output from the deformer network is divided into N segments, each segment being the token sequence for each viewpoint; the tokens in each segment are then... The recombined size is H′×W ′ The feature map is then upsampled by 4 times through interpolation, and the mask of the object and the coordinates of key points are estimated from the upsampled feature map.

[0028] Furthermore, the attitude optimization module operates as follows:

[0029] For a given K Δ First, these predefined key points are used to determine the initial pose obtained from the multi-view pose estimation module. The image is transformed to the camera's perspective, then projected onto a 2D image plane, and the coordinates of these 2D pixels are derived from the feature map F. i K was obtained by mid-sampling Δ Key feature tokens

[0030] For each viewpoint, this set of keypoint feature tokens introduces two learnable pose tokens. Together they form a token sequence

[0031] In the pose optimization module, three different forms of position encoding are used: camera extrinsic position encoding. 2D pixel coordinate position encoding Key point coordinate location encoding Keypoint feature tokens obtained by sampling Then, each token is sequentially encoded with three different forms of positional codes to obtain the token sequence input to the Transformer network.

[0032] Learnable pose tokens from the token sequence output by the Transformer network. The pose residuals are separated and obtained through regression using a multilayer perceptron (MLP).

[0033] residual Results from the multi-view pose estimation module Together, they form the final predicted rotation and translation, thus obtaining the final object pose.

[0034] Furthermore, a two-stage training approach is used to train the image coding module, the multi-view pose estimation module, and the pose optimization module:

[0035] The first phase involves jointly training the image coding module and the multi-view pose estimation module. The second phase involves freezing the image coding module and the multi-view pose estimation module, and training the pose optimization module separately.

[0036] Furthermore, in the first stage, the image encoding module and the multi-view pose estimation module are trained by optimizing the following objective function:

[0037]

[0038] in, and These are the rotation, translation, mask, and keypoint coordinates of the i-th viewpoint predicted by the multi-view pose estimation module; with superscripts. The corresponding representation is the actual annotation; for rotation, It is obtained by calculating the distance between two point clouds:

[0039]

[0040] Where O is a series of points sampled from the object's CAD model; for symmetrical objects, a true value rotation set is selected. The loss is calculated based on the result that is closest to the prediction.

[0041]

[0042] For other loss items: translation loss The mask loss is represented by the Euclidean distance between the predicted translation vector and the true translation vector. The keypoint prediction loss is obtained by calculating the binary cross-entropy. Represented by the Euclidean distance between two sets of key points;

[0043] In the second stage, the training weights from the first stage are fixed, and the following objective function training posture optimization module is optimized:

[0044]

[0045] in, Represents the residuals of predicted rotations and translations; with superscript. The symbol represents the actual annotation.

[0046] The second technical solution adopted in this invention is:

[0047] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a multi-view object pose estimation and pose optimization method as described above.

[0048] The third technical solution adopted in this invention is:

[0049] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a multi-view object pose estimation and pose optimization method as described above.

[0050] The fourth technical solution adopted in this invention is:

[0051] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned multi-view object pose estimation and pose optimization method.

[0052] The beneficial effects of this invention are as follows: By directly modeling multi-view images, this invention solves the object's pose in one step, replacing the previous two-stage solution process, making the overall framework simpler and more efficient; it possesses high flexibility and wide applicability, capable of handling inputs from any number of viewpoints and compatible with binocular input methods with left and right parallel configurations. Furthermore, while improving pose estimation accuracy, it significantly accelerates computation speed, achieving a dual optimization of performance and efficiency. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a visualization of the multi-view object pose estimation method and pose optimization method provided in the embodiments of the present invention on the YCB-V dataset;

[0055] Figure 2 This is a visualization of the multi-view object pose estimation method and pose optimization method provided in this embodiment of the invention on the StereOBJ-1M dataset;

[0056] Figure 3 This is an overall flowchart of the multi-view object pose estimation method and pose optimization method provided in the embodiments of the present invention;

[0057] Figure 4 This is a flowchart of the image encoding module in an embodiment of the present invention;

[0058] Figure 5 This is a flowchart of the multi-view pose estimation module in an embodiment of the present invention;

[0059] Figure 6 This is a flowchart of the attitude optimization module in an embodiment of the present invention. Detailed Implementation

[0060] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0062] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0063] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0064] Technical explanation:

[0065] Object pose estimation, an interdisciplinary field between computer vision and robotics, specifically involves techniques from image processing, pattern recognition, and 3D reconstruction. This technology aims to identify and determine the pose of objects in a scene within 3D space, specifically their rotation and translation. Object pose estimation has significant applications in areas such as robot grasping, augmented reality (AR), virtual reality (VR), and autonomous driving.

[0066] (1) Robotic grasping: By recognizing the posture of an object, a robot or robotic arm can accurately grasp and place the object, and it is widely used in industrial logistics, life services and other scenarios.

[0067] (2) Augmented Reality and Virtual Reality: Pose estimation technology can help AR / VR systems better understand the user's surrounding environment, thereby providing a more immersive experience.

[0068] (3) Autonomous driving: Vehicles can identify the position and orientation of obstacles such as pedestrians and other vehicles through object pose estimation, which can be used for obstacle avoidance and path planning and other subsequent processes to improve the safety and efficiency of autonomous driving.

[0069] In summary, object pose estimation is a fundamental and important technology with a wide range of applications, not limited to the fields mentioned above. As technology advances and application scenarios expand, its application areas continue to grow.

[0070] To address the problems of existing technologies, this method aims to recover the six-DOF pose of an object from multi-view RGB images: given RGB images from N viewpoints... Directly predict the object pose {P1,P2,…,P} from each viewpoint. N Each object pose P includes a 3D rotation R∈SO(3) from the object coordinate system to the camera coordinate system and a 3D translation. This invention assumes that the camera extrinsic parameters {C1, C2, ..., C} for each viewpoint are... N The parameters are known and can be obtained directly through camera calibration or robotic systems. After obtaining the camera extrinsic parameters for each viewpoint, the relative pose transformation between any two cameras can be obtained through simple transformations.

[0071] Example 1

[0072] like Figure 3 As shown, this embodiment provides a multi-view object pose estimation method and a pose optimization method, specifically including the following steps:

[0073] A1. Obtain RGB images from N perspectives.

[0074] A2. Images from each viewpoint are encoded using an image encoding module. Feature extraction is performed to obtain feature maps {F1, F2, ..., F} for each viewpoint. N};

[0075] A3. The feature map {F1, F2, ..., F...} N The initial pose of the object is obtained by inputting the object's pose and the corresponding camera pose into the multi-view pose estimation module.

[0076] A4. Input the pose estimation results, feature maps, and corresponding camera poses into the pose optimization module for pose optimization to obtain the object's pose residuals.

[0077] A5. Based on the initial posture and attitude residual Obtain the object pose {P1, P2, ..., P} from each viewpoint. N}

[0078] In the following descriptions of the embodiments of this invention, the superscripts "0" and "Δ" will be used consistently for the introduction of the multi-view pose estimation module and the pose optimization module. The image encoding module, the multi-view pose estimation module, and the pose optimization module will be described in detail below.

[0079] (1) Image encoding module

[0080] See Figure 4 Given RGB images from N viewpoints The function of the image encoding module is to extract feature maps {F1, F2, ..., Fn} from these N viewpoints. N},in H×W and H′×W′ are the height and width of the image and feature map, respectively, and D is the number of channels in the feature map. The image encoding module can be various types of deep neural networks, such as convolutional neural networks (CNNs) or visual deformer networks (ViTs). This invention uses ResNet34 as the structure of the image encoding module.

[0081] (2) Multi-view pose estimation module

[0082] See Figure 5 The multi-view pose estimation module's function is to extract values ​​from the multi-view feature maps {F1, F2, ..., F...} N The initial pose of the object is obtained directly through regression. In this module, the present invention chooses to use a Transformer network to model the intrinsic relationships between different viewpoints due to its strong advantages in modeling long-range relationships. To achieve direct regression, the present invention introduces two learnable pose tokens for learning rotation and translation. This is also the innovation and unique feature of the present invention.

[0083] Specifically, this invention first takes two-dimensional feature maps F from different viewpoints i. i The features are reduced in dimensionality and flattened into a shape with K using a prior layer. 0 A one-dimensional token sequence of tokens Where K 0 =H ′ W ′ , D 0 This involves using a linear layer to reduce the dimensionality of each pixel in the original feature map to a new dimension. Then, this invention introduces two learnable tokens. and The learning of rotation and translation in pose is used, so for each viewpoint, K will eventually be obtained. 0 +2 tokens Next, the token sequences from all N perspectives It will be concatenated along the direction of the sequence and input into L 0 Feature updates and interactions are performed within a layered Transformer network to achieve feature fusion across multiple viewpoints. Finally, two tokens related to pose learning are combined. and The values ​​are separated and input into two multilayer perceptron (MLP) networks to obtain the rotation of the object from each viewpoint. Peaceful relocation In this invention, a 6D representation of rotation is used as the characterization of rotation. After obtaining the 6D representation, the rotation matrix can be obtained through Schmidt orthogonalization.

[0084] In addition to the steps described above, this embodiment of the invention also encodes the camera extrinsic and intrinsic parameters and adds them to each token in the form of positional codes to implicitly establish constraints between viewpoints, thereby enhancing the ability to model intra-view and inter-view information. The specific encoding and implementation process is as follows:

[0085] 1) Encoding of camera extrinsic parameters. For each viewpoint, it is aligned to the first viewpoint, and the camera relative transformation from each viewpoint to the first viewpoint is calculated. This extrinsic parameter transformation matrix is ​​then flattened into a 3×4 extrinsic parameter vector, which is further stretched to the level of the token dimension D using a multilayer perceptron (MLP). 0 The same dimensionality is used to obtain the extrinsic position encoding for each viewpoint. Then encode the extrinsic parameter positions. The token sequence is directly added to each viewpoint i. On each token.

[0086] 2) Encoding of camera intrinsic parameters. For the encoding of camera intrinsic parameters for each viewpoint, this invention encodes each token... The corresponding pixel coordinates in the image Perspective back projection transformation is performed to obtain and Then and The data is concatenated and input into a multilayer perceptron (MLP) to be encoded into the token dimension D. 0 The same dimensional size yields the positional encoding of each token with respect to the camera intrinsics. Finally This is also added to each token (except for the learnable pose token, since these two tokens are additionally introduced and do not exist in the image). Here, C x C y is the principal point coordinate of the camera, and f is the focal length of the camera, which can be obtained by consulting the camera's intrinsic parameter matrix.

[0087] In summary, the final token sequence input into the Transformer network can be represented by the following formula:

[0088]

[0089] To further improve the performance of pose estimation, this invention also extracts the object mask M from the output of the Transformer network. 0 and the three-dimensional position coordinates Q of the key points 0 Prediction was used as auxiliary supervision. Specifically, the token sequence from the deformer network was divided into N segments, each segment representing the token sequence for each viewpoint. The tokens in each segment were then... The recombined size is H′×W ′The feature map is then upsampled by a factor of 4 through interpolation, and the object mask and key point coordinates are estimated from the upsampled feature map.

[0090] (3) Attitude optimization module

[0091] See Figure 6 The purpose of the pose optimization module is to further optimize the initial pose of the object obtained in the previous step. and For the pose optimization module, this embodiment of the invention also uses a Transformer network to process features across multiple views. The difference from the multi-view pose estimation module is that the pose optimization module processes features from the multi-view feature maps {F1, F2, ..., F...}. N Instead of flattening the entire feature map, feature tokens corresponding to keypoints are upsampled. Meanwhile, in the pose optimization module, this invention still uses learnable pose tokens to capture more refined pose features and learn pose residuals from them. The specific steps of the pose optimization module are as follows:

[0092] For a given K Δ First, these predefined key points are used to determine the initial pose obtained from the multi-view pose estimation module. The image is transformed to the camera's perspective, then projected onto a 2D image plane, and the coordinates of these 2D pixels are derived from the feature map F. i K was obtained by mid-sampling Δ Key feature tokens Similarly, this invention introduces two learnable pose tokens for this set of keypoint feature tokens for each viewpoint. Together they form a token sequence

[0093] In the attitude optimization module, this invention uses three different forms of position encoding: camera extrinsic parameter position encoding. 2D pixel coordinate position encoding Key point coordinate location encoding Keypoint feature tokens obtained by sampling Then, each token is sequentially encoded with three different forms of positional codes to obtain the token sequence input to the Transformer network. Specifically, it can be expressed by the following formula:

[0094]

[0095] Among them, camera extrinsic position encoding The acquisition method is the same as that in the multi-view pose estimation module. 2D pixel coordinate position encoding This is achieved by transforming the keypoint coordinates in the object coordinate system according to the initial pose, projecting them onto the image plane, and then encoding them using an MLP (Multi-Level Processing) algorithm. Keypoint coordinate position encoding. It is obtained by encoding the coordinates of key points in the object coordinate system using a multilayer perceptron (MLP).

[0096] Similarly, the learnable pose tokens from the output token sequence of the Transformer network will ultimately be used. The pose residuals are separated and obtained through regression using a multilayer perceptron (MLP). residual Results from the multi-view pose estimation module Together, they form the final predicted rotation and translation.

[0097] (4) Network loss function

[0098] This invention employs a two-stage training approach to train the image encoding module, the multi-view pose estimation module, and the pose optimization module. The first stage involves jointly training the image encoding module and the multi-view pose estimation module. The second stage freezes the image encoding module and the multi-view pose estimation module, and trains the pose optimization module separately.

[0099] In the first stage, this invention trains the image coding module and the multi-view pose estimation module by optimizing the following objective function:

[0100]

[0101] in, and These are the rotation, translation, mask, and keypoint coordinates of the i-th viewpoint predicted by the multi-view pose estimation module. (Indicated by superscript) The corresponding representation is the actual annotation. For rotation, It is obtained by calculating the distance between two point clouds:

[0102]

[0103] Here, O is a series of points sampled from the object's CAD model. For symmetrical objects, a true rotation set is selected. The loss is calculated based on the result that is closest to the prediction.

[0104]

[0105] For other loss items: translation loss The mask loss is represented by the Euclidean distance between the predicted translation vector and the true translation vector. The keypoint prediction loss is obtained by calculating the binary cross-entropy. It is represented by the Euclidean distance between two key point sets.

[0106] In the second stage, the training weights from the first stage are fixed, and the following objective function training posture optimization module is optimized:

[0107]

[0108] in, Represents the residuals of the predicted rotation and translation. (Indicated by superscript) The symbol represents the actual annotation.

[0109] The above method will be further explained below with reference to specific implementation methods.

[0110] This embodiment provides a method for multi-view object pose estimation and pose optimization, including the following steps:

[0111] S1: Acquire multi-view RGB images N can be any viewpoint greater than or equal to 2; generally, a value between 2 and 5 is sufficient for accurate estimation of pose information. After data acquisition, these acquired multi-view images... Calibration was performed to obtain the camera intrinsic parameters K for each viewpoint. i and external parameter C i , 1≤i≤N. Simultaneously, these images need to be labeled to obtain the ground truth rotation R and ground truth translation t of the objects in the images, for use in the model training process. For this step, considering the challenge and high cost of obtaining the rotation and translation of objects in the 3D world, a renderer, such as BlenderProc, can be used to render and generate simulation images and labels to replace manual data collection. This reduces the cost of data collection while ensuring the accuracy of the labels. In addition to collecting and labeling data, a CAD model of the object needs to be prepared beforehand, which can be obtained by scanning the object using reconstruction equipment. After obtaining the CAD model of the object, the farthest point sampling algorithm (FPS) is used to sample K from the CAD model. Δ K is a key point used in subsequent processes. Δ Generally, 32 or 64 is sufficient.

[0112] S2: After collecting and labeling the data, the first stage of model training can begin, which involves training the image encoding module and the multi-view pose estimation module. The specific implementation process is as follows:

[0113] S2.1: Process images from each viewpoint using an object detection model. The 2D bounding box (bbox) of each object in the image is obtained as bbox = {x1, y1, x2, y2}, where x1, y1 represent the coordinates of the top-left corner of the bounding box, and x2, y2 represent the coordinates of the bottom-right corner. The image is then cropped around the center point of the bounding box, with an area 1.25 times the length and width of the bounding box, resulting in a low-resolution image containing only the target object. This low-resolution image is then adjusted to a uniform resolution; the resolution used in this invention is 256×256. For the sake of brevity, this invention still uses... To express it.

[0114] S2.2: Convert low-resolution images of the same target object from different viewpoints. The data are combined into a batch size and input into the feature extraction network, ultimately yielding the feature map F of the target object from each viewpoint. i This invention uses ResNet34 as the feature extraction network, thus ultimately obtaining feature maps of four resolutions: 8×8, 16×16, 32×32, and 64×64. This invention primarily uses the 8×8 feature map because its feature dimension is 512, providing a higher level of semantic information.

[0115] S2.3: For viewpoint i, after obtaining an 8×8 feature map, this invention first uses a simple 1×1 convolution to reduce the dimensionality of the 8×8 feature map from 512 to 256. Then, the feature map is stretched into a sequence of length 64, while introducing two learnable pose tokens. and The tokens placed at the beginning of the sequence together form the token sequence of the i-th perspective. Next, the token sequence Position encoding is added. This includes position encoding related to camera intrinsic parameters and position encoding related to camera extrinsic parameters. The specific encoding method for position encoding has been described in detail in the technical content section of this invention. For the token sequence... After adding positional encoding, these token sequences from all views are concatenated again to obtain the final token sequence. Here, this invention uses 3 views during training, so the length of the final token sequence is (2+64)×3=198. Then, this 198-length token sequence is input into a 6-layer Transformer network for feature aggregation and interaction between multiple views. Finally, the learnable pose tokens for each view are extracted from the output of the Transformer network. and The separated components are then directly regressed through a multilayer perceptron for rotation and translation. The multilayer perceptron used in this invention employs a three-layer structure: "linear layer-activation layer-linear layer-activation layer-linear layer". The activation function used is ReLU.

[0116] S2.4: After obtaining the rotation R and translation t, the loss for each part is calculated according to the formula in the technical content section of this invention. Finally, the image encoding module and the multi-view pose estimation module are trained through backpropagation. The optimizer used in this invention is Adam, with a batch size of 32, an initial learning rate of 1e-4, and a cosine decay rate reduced to 0 throughout the training process.

[0117] S3: After training the image encoding module and the multi-view pose estimation module, freeze the weights of these two modules and continue training the pose optimization module. The specific steps are as follows:

[0118] S3.1: Using the rotation and translation obtained in S2.4 as the initial estimation results, the predefined key points obtained by upsampling the object's CAD model are transformed as follows: Here, X represents the keypoint coordinates in the object coordinate system, transformed to the camera's viewpoint. Then, using the perspective projection formula, the keypoint coordinates from the camera's viewpoint are projected onto the 2D image plane to obtain their corresponding pixel coordinates (u, v). Finally, based on these pixel coordinates, keypoint feature tokens are obtained by sampling from the feature map. This step can be achieved directly by calling PyTorch's `grid_sample` function. Similarly, two learnable pose tokens are introduced. Together they form a token sequence Then, each token is sequentially encoded with three different forms of positional codes to obtain the final token sequence. Similar to the multi-view pose estimation module, the token sequences from N views are concatenated to obtain the final token sequence, which is then input into the Transformer network for feature interaction. Here, N is still set to 3, and the number of layers in the Transformer network is 6. Finally, the learnable pose tokens for each view are extracted from the output of the Transformer network. and The separated rotational residual R is regressed using a multilayer perceptron. Δ Translation residual t Δ .

[0119] S3.2: Add the residual estimation result from S3.1 to the initial estimation result from S2.4 to obtain the final estimation result. Then, calculate the loss function according to the description in the technical content section of this invention, and train the pose optimization module using the backpropagation algorithm.

[0120] S4: After training the entire network, inference on new images can begin. The overall process is basically the same as the training process, except that the steps of calculating the loss function and backpropagation are no longer required, and it can generalize to support any number of input images.

[0121] The above process is the general flow of this invention. Besides supporting input from any N perspectives, this invention is also compatible with binocular input arranged horizontally. Binocular input is more user-friendly for data acquisition and calibration, simplifies the operation, and, after testing, its performance meets the requirements for actual robot grasping.

[0122] In summary, this invention directly models multi-view images to solve the object's pose in one step, effectively addressing the following issues: 1) It solves the problem that object pose estimation methods based on single-view RGB input cannot directly solve for the translation component in the object's pose. 2) It solves the problem of low accuracy in solving object pose using methods based on single-view RGB input. 3) It replaces the traditional multi-view pose estimation framework, solving the problems of its cumbersome process, long time consumption, and inability to meet real-time estimation requirements. Compared with existing technologies, the method of this invention has the following advantages and beneficial effects: 1) This invention is the first to propose using a Transformer network to directly regress the object pose estimation problem under multi-view input to replace the previous two-stage solution process, making the overall framework simpler and more efficient. 2) It has high flexibility and wide applicability, capable of handling inputs from any number of views and compatible with binocular input methods with left and right parallel configurations. 3) While improving the accuracy of pose estimation, it significantly accelerates the calculation speed, achieving a dual optimization of performance and efficiency.

[0123] The method of this invention was compared with existing technical solutions on the same dataset, and the results are as follows:

[0124] (1) Results of the YCB-V dataset

[0125] As shown in Table 1, compared to the single-view method GDR-Net and the multi-view methods MV6D and CosyPose, the method of this invention outperforms previous methods in all three evaluation metrics. Furthermore, compared to CosyPose, the computation time of this invention is reduced from 0.318s to 0.106s, more than doubling the computation speed. The reason this invention achieves both performance and efficiency optimization is that it uses a Transformer network to directly model images from multiple perspectives, realizing information exchange and fusion between multi-view images, and directly regressing the object pose from the aggregated information. This approach is theoretically supported by multi-view geometry theory, making it more reasonable than single-view and multi-stage multi-view methods. Simultaneously, within the overall new framework, this method designs two pose-solving modules based on Transformer networks. Compared to the later fusion process of iterative optimization, this method only requires one forward computation of the network to obtain the pose result, thus significantly optimizing computational efficiency. See also Figure 1 , Figure 1 The image shows the visualization results of the method of the present invention on the YCB-V dataset. The green box in the figure represents the result of projecting the 3D bounding box of the object onto the image plane according to the predicted pose.

[0126] Table 1: Comparison of YCB-V dataset results

[0127]

[0128] (2) Results of the StereOBJ-1M dataset

[0129] As shown in Table 2, the method of this invention outperforms previous methods on the StereoOBJ-1M dataset. The StereoOBJ-1M dataset includes transparent and semi-transparent objects. The lack of salient features in these objects makes keypoint localization extremely challenging, leading to poor performance of keypoint-based pose recovery methods such as KeyPose and PVNet. In contrast, this invention directly regresses the object's pose from the image, eliminating the step of keypoint localization. Therefore, it maintains high pose estimation accuracy even for objects with inconspicuous keypoint features.

[0130] Furthermore, the StereoOBJ-1M dataset is a standard stereo image dataset, while YCB-V is a multi-view dataset from any number of viewpoints. This invention achieves top performance on both datasets, demonstrating its high flexibility and wide applicability; it can handle input from any number of viewpoints and is compatible with left-right parallel stereo input methods. See also Figure 2 , Figure 2 The image shows the visualization results of the method of the present invention on the StereOBJ-1M dataset. The green box in the figure represents the result of projecting the 3D bounding box of the object onto the image plane according to the predicted pose.

[0131] Table 2: Comparison of results from the StereOBJ-1M dataset

[0132]

[0133] Example 2

[0134] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 3 This paper presents a method for multi-view object pose estimation and pose optimization.

[0135] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0136] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0137] Since this electronic device is the electronic device corresponding to the multi-view object pose estimation and pose optimization method of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0138] Example 3

[0139] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 3 This paper presents a method for multi-view object pose estimation and pose optimization.

[0140] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0141] Since this storage medium is the storage medium corresponding to a multi-view object pose estimation and pose optimization method in an embodiment of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0142] Example 4

[0143] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a multi-view object pose estimation and pose optimization method according to various exemplary embodiments of the present application described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0144] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0145] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0146] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for multi-view object pose estimation and pose optimization, characterized in that, Includes the following steps: Obtain RGB images from N perspectives Images from each viewpoint are processed by an image encoding module. Feature extraction is performed to obtain feature maps {F1, F2, ..., F} for each viewpoint. N }; The feature map {F1,F2,…,F N The initial pose of the object is obtained by inputting the object's pose and the corresponding camera pose into the multi-view pose estimation module. The initial pose, feature map, and corresponding camera pose are input into the pose optimization module for pose optimization to obtain the object's pose residual. Based on the initial attitude and attitude residual Obtain the object pose {P1, P2, ..., P} from each viewpoint. N }; In the multi-view pose estimation module and pose optimization module, a deformer network is used to model the intrinsic relationship between different views, and two learnable pose tokens are introduced for learning rotation and translation. The multi-view pose estimation module works as follows: Two-dimensional feature maps F from different perspectives i i Perform feature reduction and flatten it into a shape with K 0 A one-dimensional token sequence of tokens Introduce two learnable tokens and The learning of rotation and translation in pose is used, so for each viewpoint, K will eventually be obtained. 0 +2 tokens, that is The token sequence from all N perspectives Concatenate along the direction of the sequence and input it into L. 0 Features are updated and interacted in a deformer network composed of stacked layers, enabling feature fusion between multiple perspectives; Two tokens related to posture learning and The values ​​are separated and input into two multilayer perceptron networks to regress the rotation of the object at each viewpoint. Peaceful relocation Obtain the initial orientation of the object.

2. The method for multi-view object pose estimation and pose optimization according to claim 1, characterized in that, The image encoding module is implemented using a deep neural network.

3. The method for multi-view object pose estimation and pose optimization according to claim 1, characterized in that, It also includes the step of encoding camera extrinsic and intrinsic parameters: Encoding of camera extrinsic parameters: For each viewpoint, align it to the first viewpoint and calculate the camera relative transformation from each viewpoint to the first viewpoint; then flatten the extrinsic parameter transformation matrix into an extrinsic parameter vector, and use a multilayer perceptron to stretch the extrinsic parameter vector to the level of the token dimension D. 0 With the same dimensionality, the extrinsic position encoding for each viewpoint is obtained. Then encode the extrinsic parameter positions. The token sequence is directly added to each viewpoint i. On each token; Encoding of camera intrinsics: For each viewpoint, the camera intrinsics are encoded by encoding each token. The corresponding pixel coordinates in the image Perform perspective backprojection transformation to obtain and Then and The data is concatenated and input into a multilayer perceptron, encoded into the token dimension D. 0 With the same dimensional size, the positional encoding of each token with respect to the camera intrinsics is obtained. Finally Added to each token; where c x ,c y is the principal point coordinate of the camera, and f is the focal length of the camera.

4. The method for multi-view object pose estimation and pose optimization according to claim 1, characterized in that, Attitude estimation also includes the following steps: The token sequence output from the deformer network is divided into N segments, each of which is the token sequence for each viewpoint. In each segment The recombined size is H′×W ′ The feature map is then upsampled by 4 times through interpolation, and the mask of the object and the coordinates of key points are estimated from the upsampled feature map.

5. The method for multi-view object pose estimation and pose optimization according to claim 1, characterized in that, The attitude optimization module operates as follows: For a given K Δ First, these predefined key points are used to determine the initial pose obtained from the multi-view pose estimation module. The image is transformed to the camera's perspective, then projected onto a 2D image plane, and the coordinates of these 2D pixels are derived from the feature map F. i K was obtained by mid-sampling Δ Key feature tokens For each viewpoint, this set of keypoint feature tokens introduces two learnable pose tokens. Together they form a token sequence In the pose optimization module, three different forms of position encoding are used: camera extrinsic position encoding. 2D pixel coordinate position encoding Key point coordinate location encoding Keypoint feature tokens obtained by sampling Then, each token is sequentially encoded with three different forms of positional codes to obtain the token sequence input to the deformer network. The learnable pose tokens are extracted from the output token sequence of the deformer network, and the pose residuals are obtained by regression through a multilayer perceptron. residual Results from the multi-view pose estimation module Together, they form the final predicted rotation and translation, thus obtaining the final object pose.

6. The method for multi-view object pose estimation and pose optimization according to claim 1, characterized in that, A two-stage training approach is used to train the image coding module, the multi-view pose estimation module, and the pose optimization module: The first phase involves jointly training the image coding module and the multi-view pose estimation module. The second phase involves freezing the image coding module and the multi-view pose estimation module, and training the pose optimization module separately.

7. The method for multi-view object pose estimation and pose optimization according to claim 6, characterized in that, In the first stage, the image encoding module and the multi-view pose estimation module are trained by optimizing the following objective function: in, and These are the rotation, translation, mask, and keypoint coordinates of the i-th viewpoint predicted by the multi-view pose estimation module; with superscripts. The corresponding representation is the actual annotation; for rotation, It is obtained by calculating the distance between two point clouds: Where O is a series of points sampled from the object's CAD model; for symmetrical objects, a true value rotation set is selected. The loss is calculated based on the result that is closest to the prediction. For other loss items: translation loss The mask loss is represented by the Euclidean distance between the predicted translation vector and the true translation vector. The keypoint prediction loss is obtained by calculating the binary cross-entropy. Represented by the Euclidean distance between two sets of key points; In the second stage, the training weights from the first stage are fixed, and the following objective function training posture optimization module is optimized: in, Represents the residuals of predicted rotations and translations; with superscript. The symbol represents the actual annotation.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the method as described in any one of claims 1 to 7.

Images