Multi-view object attitude estimation and attitude optimization method and electronic equipment

Abstract

The invention discloses a multi-view object attitude estimation and attitude optimization method and electronic equipment. The method comprises the following steps: acquiring RGB (Red, Green and Blue) pictures of a plurality of views; performing feature extraction on the picture of each view angle through an image coding module to obtain a feature map of each view angle; inputting the feature map and the corresponding camera attitude into a multi-view attitude estimation module for attitude estimation to obtain an initial attitude of the object; inputting the attitude estimation result, the feature map and the corresponding camera attitude into an attitude optimization module for attitude optimization to obtain an attitude residual error of the object; and obtaining the final object attitude under each visual angle according to the initial attitude and the attitude residual error. According to the method, modeling is directly carried out on the multi-view image through the deformer network, the posture of the object is solved at a time, the previous two-stage solving process is replaced, the whole frame is more concise and efficient, and double optimization of precision and efficiency is achieved. The method can be widely applied to the crossing field of computer vision and robot technology.

Description

Technical Field

[0001] The present invention relates to the intersecting field of computer vision and robotics technology, and in particular to a multi-view object posture estimation and posture optimization method and electronic equipment. Background Art

[0002] The current object posture estimation methods mainly include: posture estimation methods based on single-view RGB input and posture estimation methods based on multi-view RGB input. The specific analysis of these two methods is as follows:

[0003] (1) Posture estimation method based on single-view RGB input

[0004] 1) Direct regression method. Object pose estimation aims to determine the rotation and translation parameters of the object in the image relative to the optical center of the camera taking the photo. Methods based on single-view direct regression to solve pose, such as PoseCNN and GDR-Net, solve the translation component in the pose by first regressing the 2D projection coordinates of the object center and the depth value of the object center, and then solving the translation of the object relative to the camera through back projection. However, since recovering depth information from a single-view RGB image is essentially an underdetermined problem, this process is actually impossible according to the principles of perspective geometry. Therefore, these methods can often only overfit the data of specific scenes, and lack a full theoretical explanation of their effectiveness.

[0005] 2) Methods for solving correspondences. Another method for estimating the pose of an object from a single-view RGB image relies on solving the correspondence of points, that is, first determining the position of the 2D key points of the object, and then solving the pose of the object through the PnP algorithm. Representative algorithms of this type of method are BB8, Keypose, and PVNet. However, for some objects whose key point features are not significant enough, such as transparent or translucent objects, objects with highly reflective surfaces, and objects with self-occlusion, the effect of this method is often unsatisfactory. The fundamental reason is that when the key point features are not obvious, it becomes extremely difficult to accurately identify and locate these key points.

[0006] (2) Posture Estimation Method Based on Multi-view RGB Input

[0007] Multi-view pose estimation refers to the process of recovering the pose of an object from images taken from different angles. Existing multi-view RGB pose estimation methods, such as MV6D, MV-Render, and CosyPose, usually follow a two-stage process to solve the pose estimation problem. In the first stage, these methods process the input of each view independently and use a single-view method to estimate a preliminary result, which may be the initial pose of the object or the position of 2D key points. After entering the second stage, the intermediate results obtained in the first stage are jointly optimized to finally determine the pose of the object. Despite this, this two-stage method still faces certain challenges. On the one hand, in the first stage, since the processing of each view is independent of each other, it inherits the limitations of the single-view method and fails to fully utilize the potential correlations and constraints between multiple views. On the other hand, in the iterative optimization process of the second stage, due to the high computational complexity, the overall solution process takes a long time, making it difficult to meet the requirements of real-time pose estimation. Summary of the invention

[0008] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a multi-view object posture estimation and posture optimization method, electronic device and medium.

[0009] The first technical solution adopted by the present invention is:

[0010] A multi-view object posture estimation and posture optimization method comprises the following steps:

[0011] Get RGB images from N perspectives

[0012] The image encoding module is used to encode the image of each view. Perform feature extraction to obtain the feature map {F1, F2, ..., F B};

[0013] The feature map {F1, F2, ..., F B} and the camera pose corresponding to the feature map are input into the multi-view pose estimation module for pose estimation to obtain the initial pose of the object

[0014] The pose estimation result, feature map and camera pose corresponding to the feature map are input into the pose optimization module for pose optimization to obtain the pose residual of the object.

[0015] According to the initial posture and attitude residual Get the final object posture {P1,P2,…,P N}; where each object pose P includes a three-dimensional rotation R∈SO(3) and a three-dimensional translation from the object coordinate system to the camera coordinate system

[0016] Furthermore, the image encoding module is implemented using a deep neural network.

[0017] Furthermore, in the multi-view pose estimation module and the pose optimization module, a Transformer network is used to model the intrinsic connection between different views, and two learnable pose tokens are introduced for learning rotation and translation.

[0018] Furthermore, the working mode of the multi-view attitude estimation module is as follows:

[0019] The two-dimensional feature maps F from different viewpoints i i Perform feature dimensionality reduction and flatten into K 0 One-dimensional token sequence of tokens

[0020] Introducing 2 learnable tokens and For the learning of rotation and translation in the posture, so for each perspective, K 0 +2 tokens, i.e.

[0021] The token sequences from all N perspectives Splice along the sequence direction and input into the L 0 The features are updated and interacted in the Transformer network composed of stacked layers to achieve feature fusion between multiple perspectives.

[0022] Two tokens related to posture learning and Separate them and input them into two multi-layer perceptron (MLP) networks to regress the rotation of the object at each perspective. and pan Get the initial pose of the object.

[0023] Furthermore, the method further includes the step of encoding the camera external parameters and the camera internal parameters:

[0024] Encoding of camera extrinsic parameters: For each view, align it to the first view, calculate the camera relative transformation from each view to the first view; then straighten the extrinsic parameter transformation matrix into an extrinsic parameter vector, and use a multi-layer perceptron (MLP) to pull the extrinsic parameter vector to the same dimension as the token D. 0 The same dimension size, get the external parameter position encoding for each view Then encode the external parameter position Directly added to the token sequence of each perspective i On each token of;

[0025] Encoding of camera intrinsic parameters: For the encoding of each camera intrinsic parameter, each token The corresponding pixel coordinates in the image Perform perspective back projection transformation and get and Then and concatenated and fed into a multi-layer perceptron (MLP) to encode to the same dimension as the token D 0 The same dimension size, get the position encoding of each token with respect to the camera internal parameters Finally Added to each token; where C x ,C y are the coordinates of the camera's principal point, and f is the camera's focal length.

[0026] Furthermore, the posture estimation also includes the following steps:

[0027] The token sequence output from the deformer network is divided into N segments, each segment is the token sequence of each perspective; The reorganized size is H′×W ′ The feature map is then upsampled by a factor of 4 through interpolation, and the mask and key point coordinates of the object are estimated from the upsampled feature map.

[0028] Furthermore, the working mode of the posture optimization module is:

[0029] For a given K Δ predefined key points, firstly, these key points are calculated according to the initial pose obtained by the multi-view pose estimation module. Transform it to the camera perspective, then project it to the 2D image plane, and extract the pixel coordinates from the feature map F according to these 2D pixel coordinates. i K is obtained by sampling Δ Keypoint feature tokens

[0030] Two learnable pose tokens are introduced for each viewpoint keypoint feature token Together they form a token sequence

[0031] In the attitude optimization module, three different forms of position encoding are used, namely: camera extrinsic position encoding 2D pixel coordinate position encoding And key point coordinate position encoding Sampling to get key point feature tokens After that, the three forms of position encoding are added to each token in turn to finally obtain the token sequence that is input to the Transformer network.

[0032] Transformer network output token sequence to learnable pose tokens Separate it and regress it through a multi-layer perceptron (MLP) to get the residual of the posture

[0033] The residual Results from the multi-view pose estimation module Together they form the final predicted rotation and translation to obtain the final object pose.

[0034] Furthermore, a two-stage training method is used to train the image encoding module, the multi-view pose estimation module, and the pose optimization module:

[0035] In the first stage, the image coding module and the multi-view pose estimation module are trained together. In the second stage, the image coding module and the multi-view pose estimation module are frozen, and the pose optimization module is trained separately.

[0036] Furthermore, in the first stage, the image encoding module and the multi-view pose estimation module are trained by optimizing the following objective function:

[0037]

[0038] in, and They are the rotation, translation, mask and key point coordinates of the i-th view predicted by the multi-view pose estimation module; with superscript The corresponding representation of is the real annotation; for rotation, It is obtained by calculating the distance between two point clouds:

[0039]

[0040] Where O is a set of points sampled from the object CAD model; for symmetrical objects, the true value rotation set is selected The one closest to the prediction is used to calculate the loss:

[0041]

[0042] For other loss terms: translation loss The mask loss is represented by the Euclidean distance between the predicted translation vector and the true translation vector. By calculating the binary cross entropy, the key point prediction loss is obtained. It is represented by the Euclidean distance between two sets of key points;

[0043] In the second stage, the training weights of the first stage are fixed, and the posture optimization module is trained by optimizing the following objective function:

[0044]

[0045] in, Residuals of predicted rotations and translations; with superscript The corresponding real annotation is represented by .

[0046] The second technical solution adopted by the present invention is:

[0047] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-view object posture estimation and posture optimization method as described above.

[0048] The third technical solution adopted by the present invention is:

[0049] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a multi-view object posture estimation and posture optimization method as described above.

[0050] The fourth technical solution adopted by the present invention is:

[0051] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned multi-view object posture estimation and posture optimization method.

[0052] The beneficial effects of the present invention are as follows: the present invention directly models multi-view images and solves the posture of the object at one time, replacing the previous two-stage solving process, making the overall framework more concise and efficient; it has high flexibility and wide applicability, and can handle input from any number of viewpoints, and is also compatible with left-right parallel configuration binocular input. In addition, while improving the accuracy of posture estimation, it significantly speeds up the calculation speed, achieving dual optimization of performance and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 is a visualization result diagram of the multi-view object posture estimation method and posture optimization method provided in the embodiment of the present invention on the YCB-V data set;

[0055] Figure 2 is a visualization result diagram of the multi-view object posture estimation method and posture optimization method provided in the embodiment of the present invention on the StereOBJ-1M data set;

[0056] Figure 3 is an overall flow chart of a multi-view object posture estimation method and a posture optimization method provided in an embodiment of the present invention;

[0057] Figure 4 is a flow chart of an image encoding module in an embodiment of the present invention;

[0058] Figure 5 is a flow chart of a multi-view attitude estimation module in an embodiment of the present invention;

[0059] Figure 6 It is a flow chart of the posture optimization module in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0062] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0063] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0064] Technical explanation:

[0065] Object pose estimation: It is an intersection of computer vision and robotics. Specifically, it involves technologies such as image processing, pattern recognition, and 3D reconstruction. This technology aims to identify and determine the pose of objects in a scene in 3D space, that is, the rotation and translation of objects in 3D space. Object pose estimation has important application value in robot grasping, augmented reality (AR), virtual reality (VR), and autonomous driving.

[0066] (1) Robot grasping: By recognizing the posture of an object, a robot or robotic arm can accurately grasp and place the object. It is widely used in industrial logistics, life services and other scenarios.

[0067] (2) Augmented reality and virtual reality: Posture estimation technology can help AR / VR systems better understand the environment around the user, thereby providing a more immersive experience.

[0068] (3) Autonomous driving: Vehicles can identify the position and orientation of obstacles such as pedestrians and other vehicles through object posture estimation, which is used for subsequent processes such as obstacle avoidance and path planning to improve the safety and efficiency of autonomous driving.

[0069] In short, object posture estimation is a basic and important technology with a wide range of applications, not limited to the above-mentioned fields. With the advancement of technology and the expansion of application scenarios, its application areas are constantly expanding.

[0070] In view of the existing technical problems, this method aims to recover the six-degree-of-freedom posture of the object from multi-view RGB images: given RGB images from N perspectives Directly predict the object pose {P1,P2,…,P N Each object pose P includes a three-dimensional rotation R∈SO(3) from the object coordinate system to the camera coordinate system and a three-dimensional translation The present invention assumes that the camera extrinsic parameters {C1, C2, ..., C N} are known, and they can be directly obtained through camera calibration or robot system. After obtaining the camera extrinsics of each viewpoint, the relative pose transformation between two cameras can be obtained by simple transformation.

[0071] Example 1

[0072] like Figure 3 As shown, this embodiment provides a multi-view object posture estimation method and a posture optimization method, which specifically includes the following steps:

[0073] A1. Get RGB images from N perspectives

[0074] A2. Image encoding module for each view Perform feature extraction to obtain the feature map {F1, F2, ..., F N};

[0075] A3. Convert the feature map {F1, F2, …, F N} and the corresponding camera posture, which are input into the multi-view posture estimation module for posture estimation to obtain the initial posture of the object

[0076] A4. Input the pose estimation result, feature map and corresponding camera pose into the pose optimization module for pose optimization to obtain the pose residual of the object.

[0077] A5. Based on the initial posture and attitude residual Get the final object posture {P1,P2,…,P N}.

[0078] In the subsequent contents of the embodiments of the present invention, the superscripts "0" and "Δ" are uniformly used to introduce the multi-view pose estimation module and the pose optimization module. The following will introduce the three parts of the image encoding module, the multi-view pose estimation module and the pose optimization module in detail.

[0079] (1) Image Coding Module

[0080] See also Figure 4 , given RGB images from N perspectives The image encoding module is used to extract the feature maps {F1, F2, ..., F N},in H×W and H′×W′ are the height and width of the image and feature map, respectively, and D is the number of channels of the feature map. The image encoding module can be various types of deep neural networks, such as convolutional neural networks (CNN) or visual deformer networks (ViT). The present invention uses ResNet34 as the structure of the image encoding module.

[0081] (2) Multi-view pose estimation module

[0082] See also Figure 5 The function of the multi-view pose estimation module is to extract the pose from the multi-view feature map {F1, F2, …, F N} to directly regress and obtain the initial posture of the object In this module, the present invention chooses to use the Transformer network to model the intrinsic connection between different perspectives, because it has a strong advantage in modeling long-range relationships. In order to achieve the purpose of direct regression, the present invention introduces two learnable posture tokens for learning rotation and translation. This is also the innovation and uniqueness of the present invention.

[0083] Specifically, the present invention first transforms the two-dimensional feature maps F from different viewing angles i into i The feature dimension is reduced through the previous layer and flattened into K 0 One-dimensional token sequence of tokens Where K 0 =H ′ W ′ , D 0 It is a new dimension after the feature dimension reduction is performed on each pixel position of the original feature map through the linear layer. Then, the present invention introduces two learnable tokens and For the learning of rotation and translation in the posture, so for each perspective, K 0 +2 tokens, Next, the token sequences from all N perspectives will be concatenated along the sequence direction and input to the L 0 The Transformer network is composed of a stack of layers to update and interact with features, thus achieving feature fusion between multiple perspectives. Finally, the two tokens related to posture learning are and Separate them and input them into two multi-layer perceptron (MLP) networks to regress the rotation of the object at each perspective and pan In the present invention, a 6D representation of rotation is used as a representation of rotation. After obtaining the 6D representation, the rotation matrix can be obtained by Schmidt orthogonalization.

[0084] In addition to the above steps, the embodiment of the present invention also encodes the camera extrinsic parameters and camera intrinsic parameters and adds them to each token in the form of position encoding to implicitly establish constraints between viewpoints, so as to enhance the ability to model the information within and between viewpoints. The specific encoding and implementation process is as follows:

[0085] 1) Encoding of camera extrinsic parameters. For each view, align it to the first view, and calculate the camera relative transformation from each view to the first view. Then straighten the extrinsic parameter transformation matrix into a 3×4 extrinsic parameter vector, and use a multi-layer perceptron (MLP) to pull the extrinsic parameter vector to the same dimension as the token D. 0 The same dimension size obtains the external parameter position encoding of each view Then encode the external parameter position Directly added to the token sequence of each perspective i on each token.

[0086] 2) Encoding of camera intrinsic parameters. For the encoding of camera intrinsic parameters for each viewing angle, the present invention converts each token The corresponding pixel coordinates in the image Perform perspective back projection transformation to obtain and Then and concatenated and fed into a multi-layer perceptron (MLP) to be encoded into the same token dimension D 0 The same dimension size obtains the position encoding of each token with respect to the camera internal parameters Finally The same is added to each token (except the learnable pose token, because these two tokens are introduced additionally and do not exist in the image). Here, C x ,C y is the coordinate of the principal point of the camera, and f is the focal length of the camera, which can be obtained by consulting the intrinsic parameter matrix of the camera.

[0087] In summary, the token sequence that is finally input to the Transformer network can be expressed as follows:

[0088]

[0089] To further improve the performance of posture estimation, the present invention also extracts the mask M of the object from the output of the Transformer network. 0 And the three-dimensional position coordinates Q of the key points 0 The prediction is used as auxiliary supervision. The specific method is as follows: the token sequence from the deformer network is divided into N segments, each segment is the token sequence of each view. The reorganized size is H′×W ′The feature map is then upsampled by 4 times through interpolation, and the mask and key point coordinates of the object are estimated from the upsampled feature map

[0090] (3) Posture Optimization Module

[0091] See also Figure 6 The purpose of the posture optimization module is to further optimize the initial posture of the object obtained in the previous step. and For the posture optimization module, the embodiment of the present invention also uses a transformer network to process the features between multiple views. The difference from the multi-view posture estimation module is that the posture optimization module is based on the multi-view feature map {F1, F2, ..., F N} Upsample the feature tokens corresponding to the key points instead of straightening the entire feature map. At the same time, the present invention still uses learnable posture tokens in the posture optimization module to capture more refined posture features and learn posture residuals from them. The specific steps of the posture optimization module are as follows:

[0092] For a given K Δ predefined key points, firstly, these key points are calculated according to the initial pose obtained by the multi-view pose estimation module. Transform it to the camera perspective, then project it to the 2D image plane, and extract the pixel coordinates from the feature map F according to these 2D pixel coordinates. i K is obtained by sampling Δ Keypoint feature tokens Similarly, the present invention introduces two learnable posture tokens for this set of key point feature tokens of each perspective: Together they form a token sequence

[0093] In the posture optimization module, the present invention uses three different forms of position encoding, namely: camera extrinsic position encoding 2D pixel coordinate position encoding And key point coordinate position encoding Sampling to get key point feature tokens After that, the three forms of position encoding are added to each token in turn to finally obtain the token sequence that is input to the Transformer network. It can be expressed by the following formula:

[0094]

[0095] Among them, the camera extrinsic position encoding The acquisition method is the same as that in the multi-view attitude estimation module. 2D pixel coordinate position encoding It is obtained by transforming the key point coordinates in the object coordinate system according to the initial posture, projecting them onto the image plane, and then encoding them through MLP. Key point coordinate position encoding It is obtained by encoding the key point coordinates in the object coordinate system through a multi-layer perceptron (MLP).

[0096] Similarly, the Transformer network finally outputs the learnable posture tokens in the token sequence Separate it and regress it through a multi-layer perceptron (MLP) to get the residual of the posture Residual Results from the multi-view pose estimation module Together they make up the final predicted rotation and translation.

[0097] (4) Network loss function

[0098] The present invention adopts a two-stage training method to train the image coding module, the multi-view attitude estimation module and the attitude optimization module. In the first stage, the image coding module and the multi-view attitude estimation module are trained together. In the second stage, the image coding module and the multi-view attitude estimation module are frozen, and the attitude optimization module is trained separately.

[0099] In the first stage, the present invention trains the image encoding module and the multi-view pose estimation module by optimizing the following objective function:

[0100]

[0101] in, and are the rotation, translation, mask and key point coordinates of the i-th view predicted by the multi-view pose estimation module. The corresponding representation of is the true annotation. For rotation, It is obtained by calculating the distance between two point clouds:

[0102]

[0103] Where O is a set of points sampled from the object CAD model. For symmetrical objects, select the true value rotation set The one closest to the prediction is used to calculate the loss:

[0104]

[0105] For other loss terms: translation loss The mask loss is represented by the Euclidean distance between the predicted translation vector and the true translation vector. By calculating the binary cross entropy, the key point prediction loss is obtained. It is represented by the Euclidean distance between two key point sets.

[0106] In the second stage, the training weights of the first stage are fixed, and the posture optimization module is trained by optimizing the following objective function:

[0107]

[0108] in, denotes the residuals of the predicted rotation and translation. The corresponding real annotation is represented by .

[0109] The above method is supplemented with the following specific implementations.

[0110] This embodiment provides a multi-view object posture estimation and posture optimization method, including the following steps:

[0111] S1: Collect multi-view RGB images N can be any angle greater than or equal to 2. Generally, 2-5 can accurately estimate the posture information. Calibrate and obtain the camera intrinsic parameter K of the image at each viewing angle i and external reference C i ,1≤i≤N. At the same time, these pictures need to be annotated to obtain the true rotation R and true translation t of the objects in the pictures for the model training process. For this step, considering that obtaining the rotation and translation of objects in the three-dimensional world is challenging and costly. Therefore, during use, you can choose to use a renderer, such as BlenderProc, to render and generate simulated pictures and annotations instead of manually collecting data. This can not only reduce the cost of data collection, but also ensure the accuracy of annotation. In addition to collecting and annotating data, it is also necessary to prepare the CAD model of the object in advance, which can be obtained by scanning the object using some reconstruction equipment. After obtaining the CAD model of the object, the farthest point sampling algorithm (FPS) is used to sample K from the CAD model. Δ Key points are used in the subsequent process, K Δ Generally 32 or 64 is sufficient.

[0112] S2: After collecting and labeling the data, the first stage of model training can be carried out, that is, training the image encoding module and the multi-view pose estimation module. The specific implementation process is as follows:

[0113] S2.1: Use the object detection model to process images from each perspective Get the 2D bounding box bbox = {x1, y1, x2, y2} of each object in the image, where x1, y1 represent the coordinates of the upper left corner of the bounding box, and x2, y2 represent the coordinates of the lower right corner of the bounding box. Crop the image with an area 1.25 times the length and width of the bounding box centered at the center point of the bounding box to obtain a small-resolution image containing only the target object, and then adjust the small-resolution image to a uniform resolution size. The resolution used in this invention is 256×256. For the sake of simplicity, this invention still uses Indicate it.

[0114] S2.2: Small-resolution images of the same target object from different perspectives Combined into a batch size, input into the feature extraction network, and finally the feature map F of the target object at each perspective can be obtained i The present invention uses ResNet34 as the feature extraction network, so that four feature maps with resolution sizes of 8×8, 16×16, 32×32 and 64×64 can be obtained. The present invention mainly uses the 8×8 feature map because its feature dimension is 512 and has a higher level of semantic information.

[0115] S2.3: For view i, after obtaining an 8×8 feature map, the present invention first uses a simple 1×1 convolution to reduce the dimension of the 8×8 feature map from 512 to 256. Then the feature map is straightened into a sequence of length 64, and two learnable posture tokens are introduced at the same time. and Placed at the front of the sequence together constitute the token sequence of the i-th perspective Next, for the token sequence Add position coding. They are position coding related to camera internal parameters and position coding related to camera external parameters. The specific coding method of position coding has been described in detail in the technical content of this invention. After adding the position encoding, these token sequences of all views are concatenated again to obtain the final token sequence. Here, the present invention uses 3 views during the training process, so the length of the final token sequence is (2+64)×3=198. Then, the token sequence of length 198 is input into a 6-layer transformer network for aggregation and interaction of features between multiple views. Finally, the learnable posture tokens of each view are obtained from the output of the transformer network. and Separate and directly regress the rotation and translation through a multi-layer perceptron. The multi-layer perceptron of the present invention adopts a three-layer structure of "linear layer-activation layer-linear layer-activation layer-linear layer". The activation function used is ReLU.

[0116] S2.4: After obtaining the rotation R and translation t, the loss of each part is calculated according to the formula in the technical content of the present invention. Finally, the image encoding module and the multi-view pose estimation module are trained by back propagation. The optimizer used in the present invention is Adam, the batch size is 32, the initial learning rate is 1e-4, and it is reduced to 0 through the cosine attenuator during the entire training process.

[0117] S3: After training the image encoding module and the multi-view pose estimation module, freeze the weights of the two modules and continue training the pose optimization module. The specific steps are as follows:

[0118] S3.1: Using the rotation and translation obtained in S2.4 as the initial estimation result, the predefined key points obtained by upsampling the object CAD model are transformed as follows: Where X is the coordinate of the key point in the object coordinate system, transformed to the camera perspective. Then, the perspective projection formula is used to project the key point coordinates in the camera perspective onto the 2D image plane to obtain its corresponding pixel coordinates (u, v). Then, according to the pixel coordinates, sampling is performed from the feature map to obtain the key point feature token This step can be directly implemented by calling the grid_sample function of pytorch. Similarly, two learnable posture tokens are introduced Together they form a token sequence And add three forms of position encoding to each token in turn to finally get the token sequence Similar to the multi-view multi-view pose estimation module, the token sequences of N views are concatenated to obtain the final token sequence and input into the Transformer network for feature interaction. Here, N is still 3, and the number of layers of the Transformer network is 6. Finally, the learnable pose tokens of each view are obtained from the output of the Transformer network. and Separate it and regress the rotation residual R through a multi-layer perceptron Δ and the translation residual t Δ .

[0119] S3.2: Add the residual estimation result of S3.1 to the initial estimation result of S2.4 to obtain the final estimation result. Calculate the loss function according to the description of the technical content of the present invention, and train the posture optimization module through the back propagation algorithm.

[0120] S4: After the entire network is trained, you can perform inference on new images. The overall process is basically the same as the training process, except that there is no need to calculate the loss function and back propagation steps, and it can be generalized to support any number of input images.

[0121] The above process is the general process of the present invention. In addition to supporting input from any N perspectives, the present invention can also be compatible with the form of binocular input placed in parallel on the left and right. The form of binocular input is more friendly to data collection and calibration, the operation process is simpler, and the performance can also meet the requirements of actual robot grasping after testing.

[0122] In general, the present invention solves the posture of the object at one time by directly modeling the multi-view images, thereby effectively solving the following problems: 1) Solving the problem that the object posture estimation method based on single-view RGB input cannot directly solve the translation component in the object posture. 2) Solving the problem of low accuracy of the object posture solution based on the method of single-view RGB input. 3) Replacing the traditional multi-view posture estimation framework, solving the problem that its process is cumbersome, time-consuming, and cannot meet the requirements of real-time estimation. Compared with the prior art, the method of the present invention includes the following advantages and beneficial effects: 1) The present invention proposes for the first time to use a transformer network to directly regress the object posture estimation problem under multi-view input to replace the previous two-stage solution process, making the overall framework more concise and efficient. 2) It has high flexibility and wide applicability, and can process inputs from any number of perspectives, and is also compatible with the left and right parallel configuration of binocular input. 3) While improving the accuracy of posture estimation, it significantly speeds up the calculation speed, achieving dual optimization of performance and efficiency.

[0123] The method of the present invention and the existing technical solution were experimentally compared in the same data set, and the results are as follows:

[0124] (1) Results of the YCB-V dataset

[0125] From the data in Table 1, it can be seen that, whether compared with the single-view method GDR-Net or the multi-view methods MV6D and CosyPose, the method of the present invention surpasses the previous methods in the three evaluation indicators. At the same time, compared with CosyPose, the calculation time of the method of the present invention is shortened from 0.318s to 0.106s, and the operation speed is increased by more than 2 times. The reason why the method of the present invention can achieve dual optimization of performance and efficiency is that the present invention uses a transformer network to directly model the pictures between multiple perspectives, realizes the information exchange and fusion between multi-view pictures, and directly regresses and solves the object posture from the aggregated information. In principle, this approach is supported by the theory of multi-view geometry, which is more reasonable than the single-view method and the multi-stage multi-view method. At the same time, under the overall new framework, the method designs two posture solving modules based on the transformer network. Compared with the late fusion process of iterative optimization, the method only needs one forward calculation of the network to solve the posture result, so the computational efficiency is also greatly optimized. See Figure 1 , Figure 1 This is the visualization result of the method of the present invention on the YCB-V dataset. The green box in the figure represents the result of projecting the 3D bounding box of the object onto the image plane according to the predicted posture.

[0126] Table 1: Comparison of results on the YCB-V dataset

[0127]

[0128] (2) Results of the StereOBJ-1M dataset

[0129] It can be seen from the data in Table 2 that the method of the present invention also performs better than previous methods on the StereoOBJ-1M dataset. The StereoOBJ-1M dataset covers transparent and semi-transparent objects. The lack of significant features of such objects makes the location of key points extremely challenging, resulting in poor results for methods that recover the pose based on key point correspondences, such as KeyPose and PVNet. In contrast, the present invention directly regresses the pose of the object from the image, eliminating the step of locating key points. Therefore, even when faced with objects with unclear key point features, it can maintain a high pose estimation accuracy.

[0130] In addition, the StereoOBJ-1M dataset is a standard binocular image dataset, while the YCB-V dataset is a multi-view dataset from any viewpoint. The present invention achieves the highest performance on both datasets, which also shows that the present invention has high flexibility and wide applicability, and can process input from any number of viewpoints, and is also compatible with left-right parallel binocular input. Figure 2 , Figure 2 This is the visualization result of the method of the present invention on the StereOBJ-1M dataset. The green box in the figure represents the result of projecting the 3D bounding box of the object onto the image plane according to the predicted posture.

[0131] Table 2: Comparison of results on the StereOBJ-1M dataset

[0132]

[0133] Example 2

[0134] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 3 A multi-view object pose estimation and pose optimization method is shown.

[0135] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0136] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0137] Since the electronic device is an electronic device corresponding to a multi-view object posture estimation and posture optimization method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0138] Example 3

[0139] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 3 A multi-view object pose estimation and pose optimization method is shown.

[0140] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0141] Since the storage medium is a storage medium corresponding to a multi-view object posture estimation and posture optimization method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0142] Example 4

[0143] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a multi-view object posture estimation and posture optimization method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0144] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0145] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0146] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-view object posture estimation and posture optimization method, characterized in that: The following steps are involved: Get RGB images from N perspectives The image encoding module is used to encode the image of each view Perform feature extraction to obtain the feature map {F1, F2, ..., F N }; The feature map {F1, F2, ..., F B } and the corresponding camera pose are input into the multi-view pose estimation module for pose estimation. Get the initial pose of the object The pose estimation result, feature map and corresponding camera pose are input into the pose optimization module for pose optimization to obtain the pose residual of the object. According to the initial posture and attitude residual Get the final object posture {P1,P2,…,P B }.

2. A multi-view object posture estimation and posture optimization method according to claim 1, characterized in that: The image encoding module is implemented using a deep neural network.

3. A multi-view object posture estimation and posture optimization method according to claim 1, characterized in that: In the multi-view pose estimation module and the pose optimization module, a deformer network is used to model the intrinsic connection between different views, and two learnable pose tokens are introduced for learning rotation and translation.

4. A multi-view object posture estimation and posture optimization method according to claim 1 or 3, characterized in that: The working mode of the multi-view attitude estimation module is: The two-dimensional feature maps F from different viewpoints i i Perform feature dimensionality reduction and flatten into K 0 One-dimensional token sequence of tokens Introducing 2 learnable tokens and For the learning of rotation and translation in the posture, so for each perspective, K 0 +2 tokens, i.e. The token sequences from all N perspectives Splice along the sequence direction and input into the L 0 Update and interact features in the deformer network composed of stacked layers to achieve feature fusion between multiple perspectives; Two tokens related to posture learning and Separate them and input them into two multi-layer perceptron networks to regress the rotation of the object at each perspective and pan Get the initial pose of the object.

5. A multi-view object posture estimation and posture optimization method according to claim 4, characterized in that: It also includes the steps of encoding the camera external parameters and camera internal parameters: Encoding of camera extrinsic parameters: For each view, align it to the first view, calculate the camera relative transformation from each view to the first view; then straighten the extrinsic parameter transformation matrix into an extrinsic parameter vector, and use a multi-layer perceptron to pull the extrinsic parameter vector to the same dimension as the token D 0 The same dimension size, get the external parameter position encoding for each view Then encode the external parameter position Directly added to the token sequence of each perspective i On each token of; Encoding of camera intrinsic parameters: For the encoding of each camera intrinsic parameter, each token The corresponding pixel coordinates in the image Perform perspective back projection transformation and get and Then and concatenated and input into a multi-layer perceptron to be encoded into the same token dimension D 0 The same dimension size, get the position encoding of each token with respect to the camera internal parameters Finally Added to each token; where C x ,C y are the coordinates of the camera's principal point, and f is the camera's focal length.

6. A multi-view object posture estimation and posture optimization method according to claim 4, characterized in that: The following steps are also included in the pose estimation: The token sequence output from the deformer network is divided into N segments, each segment is the token sequence of each perspective; The reorganized size is H′×W ′ The feature map is then upsampled by a factor of 4 through interpolation, and the mask and key point coordinates of the object are estimated from the upsampled feature map.

7. A multi-view object posture estimation and posture optimization method according to claim 1, characterized in that: The working mode of the posture optimization module is as follows: For a given K Δ predefined key points, firstly, these key points are calculated according to the initial pose obtained by the multi-view pose estimation module. Transform it to the camera perspective, then project it to the 2D image plane, and extract the pixel coordinates from the feature map F according to these 2D pixel coordinates. i K is obtained by sampling Δ Keypoint feature tokens Two learnable pose tokens are introduced for each viewpoint keypoint feature token Together they form a token sequence In the attitude optimization module, three different forms of position encoding are used, namely: camera extrinsic position encoding 2D pixel coordinate position encoding And key point coordinate position encoding Sampling to get key point feature tokens Finally, the three forms of position encoding are added to each token in turn to finally obtain the token sequence that is input to the deformer network Transform the deformer network output token sequence into learnable pose tokens Separate it and get the residual of the posture through multi-layer perceptron regression The residual Results from the multi-view pose estimation module Together they form the final predicted rotation and translation to obtain the final object pose.

8. The multi-view object posture estimation and posture optimization method according to claim 1, characterized in that: A two-stage training method is used to train the image encoding module, multi-view pose estimation module, and pose optimization module: In the first stage, the image coding module and the multi-view pose estimation module are trained together. In the second stage, the image coding module and the multi-view pose estimation module are frozen, and the pose optimization module is trained separately.

9. A multi-view object posture estimation and posture optimization method according to claim 8, characterized in that: In the first stage, the image encoding module and the multi-view pose estimation module are trained by optimizing the following objective function: in, and They are the rotation, translation, mask and key point coordinates of the i-th view predicted by the multi-view pose estimation module; with superscript The corresponding representation of is the real annotation; for rotation, It is obtained by calculating the distance between two point clouds: Where O is a set of points sampled from the object CAD model; for symmetrical objects, the true value rotation set is selected The one closest to the prediction is used to calculate the loss: For other loss terms: translation loss The mask loss is represented by the Euclidean distance between the predicted translation vector and the true translation vector. By calculating the binary cross entropy, the key point prediction loss is obtained. It is represented by the Euclidean distance between two sets of key points; In the second stage, the training weights of the first stage are fixed, and the posture optimization module is trained by optimizing the following objective function: in, Residuals of predicted rotations and translations; with superscript The corresponding real annotation is represented by .

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 9.

Images