Binocular dense pose estimation method and device and robot

Abstract

The application discloses a binocular dense pose estimation method and device and a robot, and relates to the field of image processing. The method comprises the following steps: acquiring a disparity map, a disparity confidence and binocular geometric features output by a binocular branch, and monocular semantic features output by a monocular branch based on a reference image; fusing the binocular geometric features and the monocular semantic features to obtain fused features; outputting an instance mask, a dense standard object coordinate field and a coordinate uncertainty by a task head based on the fused features; screening target pixels according to the instance mask, back-projecting to obtain camera coordinate points, back-normalizing to obtain object coordinate points, and determining corresponding weights according to the instance mask, the coordinate uncertainty and the disparity confidence; and performing weighted pose solving according to the camera coordinate points, the object coordinate points and the corresponding weights to obtain pose data of a target object relative to a reference camera. The application improves the accuracy and robustness of pose estimation under the conditions of occlusion, weak texture and invisible local key points.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a binocular dense pose estimation method, apparatus and robot. Background Technology

[0002] Existing six-DOF pose estimation methods typically first predict several 2D keypoints, and then use PnP to solve for rotation and translation based on the known 3D keypoints and camera intrinsic parameters. When keypoints are occluded, fall outside the target object's contour, or have large positioning errors, the solution results are prone to significant fluctuations. For weakly textured target objects, these fluctuations are even more pronounced.

[0003] In binocular vision scenarios, some methods utilize disparity information between the left and right images to assist pose estimation. These methods typically require fusing binocular geometric features with monocular semantic features to predict instance masks and dense correspondences. However, under conditions of occlusion, weak texture, and invisible local keypoints, disparity information is prone to errors at object boundaries, reflective areas, and textureless regions. Directly using the erroneous disparity for backprojection and applying the backprojection results to pose estimation can easily introduce unreliable 3D correspondences. Furthermore, existing methods often struggle to simultaneously ensure the accuracy of instance masks, the reliability of dense correspondences, and the confidence level of disparity information during the inference phase, resulting in insufficient accuracy and robustness in target object pose estimation relative to the reference camera. Summary of the Invention

[0004] This application provides a binocular dense pose estimation method, apparatus, and robot, which can solve the problems of insufficient accuracy and robustness in the pose estimation of a target object relative to a reference camera in the prior art. The technical solution is as follows: In a first aspect, embodiments of this application provide a binocular dense pose estimation method, including: Acquire the left and right images captured by the binocular camera, the camera intrinsic parameters and baseline of the binocular camera, and the three-dimensional model parameters of the target object; The left and right images are input into a pre-trained multi-task neural network; the multi-task neural network includes a binocular branch, a monocular branch, and a task head. Obtain the disparity map, disparity confidence, and binocular geometric features output by the binocular branch based on the left and right images; Obtain the monocular semantic features output by the monocular branch based on the reference image, wherein the reference image is the left image or the right image, and the reference camera is the camera in the binocular camera that acquires the reference image; The binocular geometric features are fused with the monocular semantic features to obtain fused features; Obtain the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fusion features, and align the instance mask, the dense standard object coordinate field, and the coordinate uncertainty to the pixel coordinate system of the reference image; In the pixel coordinate system, target pixels are filtered according to the instance mask, and the target pixels are back-projected to obtain camera coordinate points according to the disparity map, the camera intrinsic parameters corresponding to the reference camera, and the baseline. The object coordinates are obtained by inverse normalization of the dense standard object coordinate field and the three-dimensional model parameters, and the corresponding weight of each target pixel is determined by the instance mask, the coordinate uncertainty and the disparity confidence. The pose data of the target object relative to the reference camera is obtained by performing a weighted pose calculation based on the camera coordinates, the object coordinates, and the corresponding weights.

[0005] Secondly, embodiments of this application provide a binocular dense pose estimation device, the device comprising: The acquisition module is used to acquire the left and right images captured by the binocular camera, the camera intrinsic parameters and baseline of the binocular camera, and the three-dimensional model parameters of the target object; An inference module is used to input the left image and the right image into a pre-trained multi-task neural network; the multi-task neural network includes a binocular branch, a monocular branch, and a task head; The inference module is also used to obtain the disparity map, disparity confidence and binocular geometric features output by the binocular branch based on the left image and the right image; The reasoning module is also used to obtain the monocular semantic features output by the monocular branch based on the reference image, wherein the reference image is the left image or the right image, and the reference camera is the camera in the binocular camera that acquires the reference image; The reasoning module is also used to fuse the binocular geometric features with the monocular semantic features to obtain fused features; The alignment module is used to obtain the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fusion features, and to align the instance mask, the dense standard object coordinate field, and the coordinate uncertainty to the pixel coordinate system of the reference image; The calculation module is used to filter target pixels in the pixel coordinate system according to the instance mask, and back-project the target pixels to obtain camera coordinate points according to the disparity map, the camera intrinsic parameters corresponding to the reference camera and the baseline; The determination module is used to obtain object coordinate points based on the dense standard object coordinate field and the three-dimensional model parameters through inverse normalization, and to determine the corresponding weight of each target pixel based on the instance mask, the coordinate uncertainty and the disparity confidence. The estimation module is used to perform weighted pose calculation based on the camera coordinates, the object coordinates, and the corresponding weights to obtain the pose data of the target object relative to the reference camera.

[0006] Thirdly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0007] Fourthly, embodiments of this application provide a robot that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0008] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following: This application outputs a dense standard object coordinate field through the task header, predicting 3D standard object coordinates for multiple pixels in the target region, replacing a limited number of 2D keypoints. Even with occlusion, weak texture, or large keypoint localization errors, valid correspondences can still be constructed using remaining pixels, reducing solution fluctuations. Correspondence weights are determined based on instance masks, coordinate uncertainties, and disparity confidence, reducing the impact of unreliable pixels in weighted pose calculation. The fusion of binocular geometric features and monocular semantic features improves the prediction accuracy of instance masks, dense standard object coordinate fields, and coordinate uncertainties. All prediction results are aligned to the pixel coordinate system of the reference image, ensuring that backprojection, inverse normalization, and weighted pose calculation are performed in the same coordinate system, avoiding correspondence errors. Finally, weighted pose calculation is performed using camera coordinates, object coordinates, and correspondence weights, improving the accuracy and robustness of target object pose estimation relative to the reference camera under conditions of occlusion, weak texture, and invisible local keypoints. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a scenario architecture diagram provided in the embodiments of this application; Figure 2This is a flowchart illustrating a binocular dense pose estimation method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a binocular dense pose estimation device provided in this application; Figure 4 This is a schematic diagram of a computer storage medium provided in this application; Figure 5 This is a structural schematic diagram of a robot provided in this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0012] See Figure 1 , Figure 1 This is a scene architecture diagram provided for an embodiment of this application. The scene architecture includes at least one server 1 and a robot 2. Figure 1 The device marked as 1 is server 1. Server 1 and robot 2 can establish a communication connection via a wired or wireless network. Server 1 is used to train a pre-trained multi-task neural network. Robot 2 is used to perform inference. Server 1 and robot 2 cooperate to execute the method provided in this application.

[0013] In one possible implementation, robot 2 includes a stereo camera, a processor, and a memory. The stereo camera is used to acquire left and right images. The memory is used to store the camera intrinsic parameters and baseline of the stereo camera, the 3D model parameters of the target object, and the network parameters of a pre-trained multi-task neural network sent by server 1. The processor is used to execute the stereo dense pose estimation method provided in this application.

[0014] Specifically, the processor acquires the left and right images captured by the stereo camera, the camera intrinsics and baseline of the stereo camera, and the 3D model parameters of the target object. The processor inputs the left and right images into a pre-trained multi-task neural network. The pre-trained multi-task neural network includes a stereo branch, a monocular branch, and a task head. The processor acquires the disparity map, disparity confidence, and stereo geometric features output by the stereo branch based on the left and right images. The processor acquires the monocular semantic features output by the monocular branch based on a reference image. The reference image is either the left or right image. The reference camera is the camera in the stereo camera that acquired the reference image. The processor fuses the stereo geometric features with the monocular semantic features to obtain fused features. The processor acquires the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fused features, and aligns the instance mask, dense standard object coordinate field, and coordinate uncertainty to the pixel coordinate system of the reference image. The processor filters target pixels in the pixel coordinate system according to the instance mask, and backprojects the target pixels to obtain camera coordinate points based on the disparity map, the camera intrinsics of the reference camera, and the baseline. The processor obtains the object coordinates based on the dense standard object coordinate field and the 3D model parameters through inverse normalization, and determines the corresponding weight for each target pixel based on the instance mask, coordinate uncertainty, and disparity confidence. The processor then performs a weighted pose calculation based on the camera coordinates, object coordinates, and corresponding weights to obtain the pose data of the target object relative to the reference camera.

[0015] In one possible implementation, after server 1 trains a pre-trained multi-task neural network, it sends the network parameters of the pre-trained multi-task neural network to robot 2. Robot 2's processor loads these network parameters and performs inference using the left and right images captured by the binocular camera. Robot 2 may also send at least one of the left and right images captured during the inference process, as well as the resulting pose data, to server 1. Server 1 updates the training data based on the data sent by robot 2 and retrains the pre-trained multi-task neural network. Server 1 then sends the retrained network parameters of the pre-trained multi-task neural network back to robot 2.

[0016] In a possible embodiment of this application, the process of server 1 training a multi-task neural network is described below.

[0017] Server 1 constructs a multi-task neural network to be trained. The multi-task neural network includes a binocular branch, a monocular branch, and a task head. The binocular branch includes a binocular backbone network, a cost volume construction module, a regularization network, a disparity refinement network, and a geometric feature extraction module. The monocular branch includes a monocular backbone network. The task head includes a masking head and a coordinate field head. Server 1 sets the network parameters for the multi-task neural network to be trained. The network parameters include binocular branch parameters, monocular branch parameters, task head parameters, and learnable surrogate feature parameters. Server 1 trains the multi-task neural network to be trained, obtaining a pre-trained multi-task neural network after training. Server 1 sends the network parameters of the pre-trained multi-task neural network to robot 2.

[0018] Server 1 acquires training data. The training data includes the left and right images of the b-th training sample. Server 1 acquires the camera intrinsics and baseline of the stereo camera. Server 1 acquires the 3D model parameters of the target object. The 3D model parameters include the minimum bounding box value and side length vector of the target object in its own standard object coordinate system. The minimum bounding box value is denoted as... The side length vector is denoted as Server 1 acquires the supervision labels. The supervision labels include the target bounding box, instance mask, effective disparity, dense standard object coordinate field, and ground truth pose. The ground truth pose includes the ground truth rotation matrix and the ground truth translation vector. When the absolute size of the target object is unknown, the ground truth pose also includes the ground truth scale.

[0019] Server 1 generates dense standard object coordinate field labels based on the 3D model parameters of the target object. For points on the 3D model surface... Server 1 calculates the coordinates of the standard object. The standard object coordinates satisfy... , This represents a tiny constant to prevent division by zero. The numerical range of standard object coordinates is 0 to 1. Server 1 generates three-channel standard object coordinate labels for each visible pixel within the target mask by rendering the visible surface of the 3D model in its ground truth pose or by inverse depth transformation. Server 1 uses these three-channel standard object coordinate labels as dense standard object coordinate field labels.

[0020] Server 1 performs spatial and photometric augmentation on the training samples. Spatial augmentation includes random rotation, asymmetric filling, asymmetric cropping, scaling, and perspective transformation. Photometric augmentation includes brightness transformation, contrast transformation, and color transformation. Server 1 records the augmentation operation and parameters for each training sample. Server 1 performs the same 2D resampling on the dense standard object coordinate field labels as on the reference image. The reference image is either the left or right image. The reference camera is the camera in a stereo camera that acquired the reference image. 2D resampling changes the pixel position of the dense standard object coordinate field labels. 2D resampling does not change the coordinate values ​​of the dense standard object coordinate field labels.

[0021] Server 1 determines the geometric compatibility identifier. Server 1 uses a preset geometric compatibility function for each enhancement operation. Server 1 calculates the compatibility based on at least one of the following: camera parameters before and after enhancement, fundamental matrix, and effective disparity labels. The geometric compatibility identifier for the b-th training sample is denoted as... When all enhancements maintain the epipolar geometry and disparity geometry required for binocular branching, =1. When at least one incompatible enhancement exists. =0. Server 1 can also use a batch-level geometric compatibility identifier. The batch-level geometric compatibility identifier is denoted as... , B represents the number of training samples in the training batch. Server 1 can use a sample-level geometric compatibility identifier. .

[0022] Server 1 inputs the enhanced left and right images into the stereo branch. For For training samples with a resolution of 1, server 1 obtains left and right feature maps through a binocular backbone network. Server 1 constructs a cost volume based on the left and right feature maps. Server 1 regularizes the cost volume to obtain an initial disparity. Server 1 refines the initial disparity to obtain a disparity map. Server 1 determines the disparity confidence based on the cost volume or the initial disparity. Server 1 extracts binocular geometric features from the cost volume or the regularized intermediate tensor. Server 1 transforms the binocular geometric features to the same spatial resolution as the monocular semantic features. For training samples with a value of 0, server 1 skips the forward computation of the binocular branch.

[0023] Server 1 inputs a reference image into the monocular branch. Server 1 extracts features from the reference image through a monocular backbone network, obtaining monocular semantic features at multiple scales. These monocular semantic features at multiple scales include those downsampled by 8x, 16x, and 32x.

[0024] Server 1 sets trainable vectors for each fusion scale. 'l' represents the fusion scale. Server 1 will use the trainable vectors... Expanding along the batch and spatial dimensions yields learnable substitute features. , Repeat() indicates that the expansion operation is repeated. Indicates the spatial height of the fusion scale l. This represents the spatial width of the fusion scale l.

[0025] Server 1 selects the stereo geometric features to be injected based on the geometric compatibility identifier. The stereo geometric features to be injected are denoted as follows: The stereo geometric features to be injected satisfy... , This represents the binocular geometric features at fusion scale l. When When =1, server 1 will display the true binocular geometric features. As the stereo geometric features to be injected. When When =0, server 1 will be able to learn alternative features. As the stereo geometric features to be injected.

[0026] Server 1 fuses the binocular geometric features to be injected with the monocular semantic features. For each fusion scale, Server 1 concatenates the binocular geometric features to be injected with the monocular semantic features to obtain concatenated features. Server 1 performs convolution, normalization, gated linear unit (LMU) operations, or attention operations on the concatenated features to obtain the fused features for that fusion scale. Server 1 obtains the fused features based on the fused features from multiple fusion scales.

[0027] Server 1 inputs the fused features into the task header. Server 1 predicts the fused features using the mask header to obtain the instance mask. Server 1 also predicts the fused features using the coordinate field header to obtain the dense standard object coordinate field and coordinate uncertainty. Server 1 can also predict the fused features using the bounding box header to obtain the target bounding box.

[0028] Server 1 solves for the predicted pose based on the instance mask, disparity map, dense standard object coordinate field, coordinate uncertainty, camera intrinsics, baseline, and 3D model parameters. Server 1 filters target pixels in the pixel coordinate system of the reference image using the instance mask. Server 1 backprojects the target pixels to obtain camera coordinate points based on the disparity map, the camera intrinsics of the reference camera, and the baseline. Server 1 obtains object coordinate points by inverse normalization using the dense standard object coordinate field and 3D model parameters. Server 1 determines the corresponding weight for each target pixel based on the instance mask, coordinate uncertainty, and disparity confidence. Server 1 performs a weighted pose solution based on the camera coordinate points, object coordinate points, and corresponding weights to obtain the predicted pose. The predicted pose includes the predicted rotation matrix and the predicted translation vector. When the absolute size of the target object is unknown, the predicted pose also includes the predicted scale.

[0029] Server 1 calculates the joint loss. The joint loss includes the bounding box loss. Mask loss Coordinate field loss Parallax loss and pose loss The joint loss satisfies , Indicates the bounding box loss weights. This indicates the mask loss weights. Indicates the coordinate field loss weight. This represents the weight of the enhanced equivariant consistency loss. This represents binocular correlated loss. Binocular correlated loss satisfies... , Indicates the parallax loss weight. Indicates the pose loss weights. This represents the disparity loss of the b-th training sample. This represents the pose loss of the b-th training sample.

[0030] Server 1 calculates the monocular task loss for all training samples. The monocular task loss includes bounding box loss, mask loss, and coordinate field loss. Server 1 only calculates the loss for g. b Calculate disparity loss and pose loss for training samples with a value of 1. When g... b When the value is 0, server 1 does not backpropagate gradients to the stereo branch. The learnable surrogate feature parameters are still updated based on the monocular task loss. Server 1 applies L2 regularization, random dropout, or cross-batch consistency constraints to the learnable surrogate feature parameters.

[0031] Server 1 performs backpropagation based on the joint loss. Server 1 updates the monocular branch parameters, task header parameters, and learnable alternative feature parameters. The stereo branch only receives g. b Gradient and state updates for training samples with a value of 1. Server 1 skips the forward computation and parameter updates of the binocular branch when the geometry compatibility flag for the entire batch is 0. For batch normalized units, server 1 updates g... b When the value is 0, the running mean and running variance updates for that path are frozen or skipped.

[0032] Server 1 repeats the above training process until the preset training termination conditions are met. These conditions include reaching a preset number of training epochs, the joint loss being less than a preset loss threshold, or the validation set metric no longer improving. Server 1 uses the trained multi-task neural network as a pre-trained multi-task neural network. Server 1 sends the network parameters of the pre-trained multi-task neural network to Robot 2. Robot 2's processor loads these network parameters and performs inference.

[0033] Through the aforementioned training process, Server 1 enables the multi-task neural network to still train monocular branches and task heads even when spatial augmentation disrupts binocular geometry, and maintains the trainability of the fusion interface and feature distribution by utilizing learnable alternative features. Server 1 synchronously switches between feature, loss, and gradient levels using geometrically compatible identifiers, avoiding the pollution of shared monocular tasks by information from failure cost volumes. Server 1 improves the accuracy of pose estimation for target objects under conditions of occlusion, weak texture, and invisible local keypoints by training pose estimation using a weighted 3D correspondence formed by a dense standard object coordinate field and disparity backprojection.

[0034] Please see Figure 2 The diagram below illustrates a flowchart of a binocular dense pose estimation method as provided in this embodiment of the application. Figure 2 As shown, the method in this application embodiment may include the following steps: S1. Acquire the left and right images captured by the binocular camera, the camera intrinsic parameters and baseline of the binocular camera, and the 3D model parameters of the target object.

[0035] The robot's processor acquires left and right images simultaneously via a binocular camera interface. The processor reads the camera intrinsics and baseline from memory. The camera intrinsics include the focal length and principal point coordinates of the reference camera. The baseline represents the distance between the optical centers of the left and right cameras. The processor also reads the 3D model parameters of the target object from memory. These parameters include the minimum bounding box value and side length vector in the target object's standard object coordinate system. The minimum bounding box value is denoted as... The side length vector is denoted as The 3D model parameters are used to subsequently inverse normalize the dense standard object coordinate field to the target object's metric object coordinates.

[0036] In one example, the binocular cameras acquire left and right images at a resolution of 640×480. The reference camera is the left camera. The focal length is the camera intrinsic parameter of the left camera. The target object is a hopper, with a resolution of 800 pixels and principal point coordinates of 320 and 240. The baseline B is 0.12 meters. The minimum bounding box value is required in the 3D model parameters. for[ 0.1, [0.05, 0], side length vector [0.2, 0.1, 0.15].

[0037] S2. Input the left and right images into the pre-trained multi-task neural network.

[0038] The multi-task neural network includes a stereo branch, a monocular branch, and a task head. The robot's processor loads the network parameters of the pre-trained multi-task neural network from memory. These parameters include stereo branch parameters, monocular branch parameters, and task head parameters. The robot's processor inputs the left and right images into the stereo branch. The robot's processor inputs a reference image into the monocular branch. The reference image can be either the left or right image. The reference camera is the camera in the stereo camera that acquired the reference image. During the inference phase, random rotations and asymmetric padding that disrupt the stereo geometry are not performed. The robot's processor sets the geometry-compatible flag to a valid state. The stereo branch outputs true disparity features. The monocular branch outputs monocular semantic features. The task head receives the fused features and outputs the prediction result.

[0039] In one example, the left image is selected as the reference image. The reference camera is the left camera. The robot's processor inputs the left and right images into the stereo branch and the left image into the monocular branch. The network parameters of the pre-trained multi-task neural network are stored in the robot's memory after training by the server. The robot's processor loads these network parameters and performs forward computation.

[0040] S3. Obtain the disparity map, disparity confidence, and binocular geometric features output by the binocular branch based on the left and right images.

[0041] The binocular branch includes a binocular backbone network, a cost volume construction module, a regularization network, a disparity refinement network, and a geometric feature extraction module. The robot's processor inputs the left and right images into the binocular backbone network, obtaining left and right feature maps. The robot's processor constructs a cost volume based on the left and right feature maps. The robot's processor regularizes the cost volume to obtain an initial disparity. The robot's processor refines the initial disparity to obtain a disparity map. The robot's processor determines the disparity confidence based on the cost volume or the initial disparity. The robot's processor extracts binocular geometric features from the cost volume or the regularized intermediate tensor. The robot's processor transforms the binocular geometric features to the same spatial resolution as the monocular semantic features.

[0042] A disparity map is an image output by the binocular branch after stereo matching based on the left and right images. It represents the pixel-wise disparity values ​​of the horizontal position differences between corresponding points in the reference image and the corresponding points in the other image. Disparity confidence is a numerical value determined by the binocular branch based on the cost volume or initial disparity, corresponding one-to-one with the pixels in the disparity map, and used to represent the reliability of the disparity value for that pixel. Binocular geometric features are multi-scale features extracted by the binocular branch from the cost volume or regularized intermediate tensors. These features characterize the stereo matching geometric relationship between the left and right images and are transformed to the same spatial resolution as monocular semantic features.

[0043] S4. Obtain the monocular semantic features of the monocular branch based on the reference image output. The reference image is either the left or right image, and the reference camera is the camera that acquires the reference image in a stereo camera.

[0044] The monocular branch includes a monocular backbone network. The robot's processor inputs a reference image into the monocular backbone network. The robot's processor extracts features from the reference image through the monocular backbone network, obtaining monocular semantic features at multiple scales. These multiple scales of monocular semantic features correspond to spatial resolutions of 8x, 16x, and 32x downsampling of the reference image, respectively. Monocular semantic features are used to provide semantic information, boundary information, and local geometric information of the target object.

[0045] S5. The binocular geometric features and monocular semantic features are fused to obtain the fused features.

[0046] For each fusion scale, the robot's processor concatenates the binocular geometric features with the monocular semantic features to obtain the concatenated features for that scale. The processor then performs convolution, normalization, gated linear unit (LMU) operations, or attention operations on these concatenated features to obtain the fused features for that scale. Based on the fused features from multiple fusion scales, the processor derives the final fused features. During the inference phase, geometric compatibility is flagged as a valid state. The processor then injects the true binocular geometric features output from the binocular branch into the fusion process. The processor does not use learnable surrogate features.

[0047] S6. Obtain the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fusion features, and align the instance mask, dense standard object coordinate field, and coordinate uncertainty to the pixel coordinate system of the reference image.

[0048] The task head includes a masking head and a coordinate field head. The robot's processor inputs the fused features into the masking head. The robot's processor predicts the fused features using the masking head to obtain an instance mask. The robot's processor inputs the fused features into the coordinate field head. The robot's processor predicts the fused features using the coordinate field head to obtain a dense standard object coordinate field and coordinate uncertainties. The dense standard object coordinate field includes the 3D standard object coordinates of each pixel in the target region. The coordinate uncertainties include the logarithmic uncertainty of each pixel in the target region along the three coordinate axes. The robot's processor aligns the instance mask, dense standard object coordinate field, and coordinate uncertainties to the pixel coordinate system of the reference image. Alignment methods include interpolation or resampling.

[0049] An instance mask is a mask output by the task head based on fused features, corresponding one-to-one with pixels in the reference image and used to represent the probability that a pixel belongs to the target object. A dense standard object coordinate field is a coordinate field output by the task head based on fused features, predicting the three-dimensional standard object coordinates in the target object's own standard object coordinate system for each pixel in the target region of the reference image. Coordinate uncertainty is the logarithmic uncertainty output by the task head based on fused features, corresponding one-to-one with the three-dimensional standard object coordinates of each pixel in the dense standard object coordinate field, and used to represent the reliability of the prediction of the three-dimensional standard object coordinates in the three coordinate axes. The pixel coordinate system of the reference image is a two-dimensional coordinate system established based on the reference image, representing the pixel positions in the reference image using pixel coordinates. The reference image can be the left or right image, and the reference camera is the camera in a stereo camera that acquires the reference image.

[0050] S7. In the pixel coordinate system, filter the target pixels according to the instance mask, and back-project the target pixels to obtain the camera coordinate points according to the disparity map, the camera intrinsic parameters corresponding to the reference camera and the baseline.

[0051] The robot's processor obtains the mask probability of the instance mask in the pixel coordinate system of the reference image. The robot's processor identifies pixels with mask probabilities greater than a preset threshold as target pixels. The robot's processor obtains the disparity value of the target pixel from the disparity map. The robot's processor calculates the depth value of the target pixel based on the disparity value, the focal length in the camera intrinsics of the reference camera, and the baseline. The robot's processor back-projects the target pixel based on the depth value and the camera intrinsics of the reference camera to obtain the camera coordinate point. The depth value satisfies... The camera coordinates satisfy... .in, Represents the depth value of the target pixel. Indicates the focal length of the reference camera. Indicates the baseline. This represents the disparity value of the target pixel. This represents a small constant used to prevent division by zero. This represents the camera intrinsic parameter matrix of the reference camera. and This represents the coordinates of the target pixel in the reference image pixel coordinate system. This represents the camera coordinate point. The camera coordinate point is a three-dimensional coordinate point obtained by back-projecting the target pixel onto the reference camera coordinate system based on the disparity value of the target pixel in the disparity map, the camera intrinsic parameters of the reference camera, and the baseline. It is used to characterize the three-dimensional position of the target pixel in the reference camera coordinate system.

[0052] S8. Obtain the object coordinate points by inverse normalization of the dense standard object coordinate field and 3D model parameters, and determine the corresponding weight of each target pixel based on the instance mask, coordinate uncertainty and disparity confidence.

[0053] The robot's processor obtains the minimum bounding box value from the 3D model parameters of the target object. and side length vector The robot's processor obtains the three-dimensional standard object coordinates of the target pixel from a dense standard object coordinate field. The robot's processor will use the coordinates of a three-dimensional standard object. with side length vector Element-wise multiplication yields the scaled coordinates. The robot's processor then compares these scaled coordinates with the minimum bounding box value. Add them together to get the object coordinates corresponding to the target pixel. The object's coordinates satisfy... .in, This represents element-wise multiplication. The robot's processor obtains the mask probability of the target pixel from the instance mask. The robot's processor takes the negative exponent of the coordinate uncertainty to obtain the uncertainty factor of the target pixel. The robot's processor also obtains the confidence value of the target pixel in the disparity confidence score. The robot's processor will mask the probability. Uncertainty factor and confidence value Multiplying them yields the corresponding weights for the target pixels. The corresponding weights satisfy... .in, , , This represents the coordinate uncertainty in the three coordinate axes. This indicates the optional visibility probability. If no visibility probability is set, then... Take 1.

[0054] S9. Perform weighted pose calculation based on camera coordinates, object coordinates, and corresponding weights to obtain the pose data of the target object relative to the reference camera.

[0055] The robot's processor calculates a weighted average of the object's coordinates based on corresponding weights to obtain the object's centroid. The processor then calculates a weighted average of the camera's coordinates based on corresponding weights to obtain the camera's centroid. The processor then calculates a weighted covariance matrix using the object's coordinates, camera's coordinates, object's centroid, camera's centroid, and corresponding weights. The processor performs singular value decomposition on the weighted covariance matrix to obtain a rotation matrix. The processor then calculates a translation vector using the rotation matrix, camera's centroid, and object's centroid. The processor uses the rotation matrix and translation vector as pose data. The processor can also update the corresponding weights based on the residuals and repeat the weighted pose calculation 1 to K times, where K is an integer greater than 1. The processor calculates the similarity transformation scale when the absolute size of the target object is unknown.

[0056] The object's centroid satisfies The camera's centroid satisfies... The weighted covariance matrix satisfies Singular value decomposition satisfies The rotation matrix satisfies `diag()` represents the diagonal matrix construction operation; that is, `diag(a, b, c)` represents a diagonal matrix whose diagonal elements are a, b, c, and whose off-diagonal elements are all 0. `det()` represents the determinant operation of a matrix; that is, `det(A)` represents the determinant of a square matrix A. Translation vectors satisfy... The residual satisfies The weights are updated using Huber weights, Tukey weights, or truncation weights based on the residuals, and the weighted pose is solved 1 to K times. K is an integer greater than or equal to 1.

[0057] In one example, the object centroid and camera centroid are calculated based on the object coordinates, camera coordinates, and corresponding weights of multiple target pixels. The robot's processor calculates the weighted covariance matrix. The processor performs singular value decomposition on the weighted covariance matrix to obtain the rotation matrix R and translation vector t. The processor updates the corresponding weights based on the residuals and repeats the weighted pose calculation three times. The processor outputs the pose data of the target object relative to the reference camera. The pose data includes the rotation matrix R and the translation vector t. The processor generates robot control commands based on the pose data.

[0058] In one possible implementation, obtaining the disparity map, disparity confidence, and binocular geometric features output by the binocular branch based on the left and right images includes: S31. Input the left image and the right image into the binocular backbone network of the binocular branch to obtain the left feature map and the right feature map.

[0059] The robot's processor inputs the left and right images into the binocular backbone network of the binocular branch, respectively. The binocular backbone network performs shared-weight feature extraction on the left and right images. The binocular backbone network includes multiple downsampling stages. Each downsampling stage outputs left and right feature maps at the corresponding scale. The left and right feature maps have the same number of channels and the same spatial resolution at the same scale. The left and right feature maps are used to construct the cost volume later. The binocular backbone network can employ a convolutional neural network or a Transformer network. During the inference stage, the robot's processor loads the binocular branch parameters trained by the server.

[0060] In one example, both the left and right images have a resolution of 640×480. The binocular backbone network outputs left and right feature maps downsampled by 4x, 8x, and 16x. The left and right feature maps downsampled by 4x have 64 channels. The left and right feature maps downsampled by 8x have 128 channels. The left and right feature maps downsampled by 16x have 256 channels. The robot's processor selects the left and right feature maps downsampled by 4x for constructing the cost volume.

[0061] S32. Construct a cost body based on the left feature map and the right feature map.

[0062] The robot's processor inputs the left and right feature maps into the correlation layer. The correlation layer calculates the matching cost between the left and right feature maps along multiple disparity candidate directions. The disparity candidate range is denoted as D. For each disparity candidate d, the robot's processor calculates the similarity between the left and right feature maps after a horizontal offset of d. Similarity can be calculated using dot product, cosine similarity, or group correlation. The robot's processor constructs a cost volume based on the matching cost. The dimensions of the cost volume include the number of disparity candidates, spatial height, spatial width, and number of channels. The matching cost satisfies... , Represents the left feature map. This represents the right feature map, and shift indicates the horizontal offset. This indicates similarity calculation.

[0063] In one example, the robot's processor constructs a cost volume at a 4x downsampling scale. The disparity candidate range D is 0 to 192, with a stride of 1. The spatial height of the 4x downsampling feature map is 120, the spatial width is 160, and the number of channels is 64. The cost volume dimensions are 193 × 120 × 160 × 64. The robot's processor calculates the matching cost corresponding to each disparity candidate through a correlation layer and arranges the matching costs in disparity candidate order to obtain the cost volume.

[0064] S33. Regularize the cost volume to obtain the initial disparity.

[0065] The robot's processor inputs the cost volume into a regularization network. The regularization network performs regularization on the cost volume. This network can be a 3D or 2D convolutional network. The 3D convolutional network performs convolution operations along the disparity, height, and width dimensions, aggregating contextual information from the cost volume. The regularization network outputs a matching probability volume. The robot's processor determines the initial disparity for each pixel based on this matching probability volume. This determination can be achieved using a winner-take-all or softargmin approach. The initial disparity satisfies... P represents the matching probability volume. The initial disparity is represented by softargmin(), which is a differentiable soft minimum position calculation operation. That is, the cost volume is first normalized along the disparity candidate direction to obtain the matching probability volume, and then the matching probability is weighted and summed with each disparity candidate value as a weight to obtain the continuous initial disparity.

[0066] In one example, the regularization network contains multiple 3D convolutional layers and skip connections. The robot's processor inputs a cost volume of size 193×120×160×64 into the regularization network. The regularization network outputs a matching probability volume of size 193×120×160. For each pixel, the robot's processor selects the disparity candidate with the highest matching probability within the disparity candidate range as the initial disparity. The resolution of the initial disparity is downsampled by a factor of 4.

[0067] S34. Refine the initial disparity to obtain the disparity map.

[0068] The robot's processor inputs the initial disparity and a reference image into a disparity refinement network. The disparity refinement network corrects the initial disparity based on edge, texture, and contextual information from the reference image. The disparity refinement network can employ a contextual network or a residual network. The disparity refinement network outputs a disparity map. The disparity map is aligned with the pixel coordinate system of the reference image. The resolution of the disparity map is the same as the resolution of the reference image.

[0069] In one example, the robot's processor downsamples the initial disparity by a factor of 4 and upsamples it to the reference image resolution using bilinear interpolation. The robot's processor then inputs the upsampled initial disparity and the reference image into a context network. The context network contains multiple residual blocks. The context network outputs residual disparity. The robot's processor adds the upsampled initial disparity to the residual disparity to obtain a disparity map. The disparity map has a resolution of 640×480. Each pixel in the disparity map corresponds to a pixel at the same location in the reference image.

[0070] S35. Determine the disparity confidence level based on the cost body or the initial disparity.

[0071] When the robot's processor determines the disparity confidence based on the cost volume, it obtains the matching probability of each pixel at the optimal disparity candidate from the matching probability volume. The robot's processor uses this matching probability as the disparity confidence. When determining the disparity confidence based on the initial disparity, the robot's processor calculates the difference between the initial disparity and the disparity map. The robot's processor uses the negative exponent of this difference as the disparity confidence. There is a one-to-one correspondence between the disparity confidence and the pixels in the disparity map. The disparity confidence satisfies... , Indicates the parallax confidence level. This represents the matching probability of disparity candidate d.

[0072] In one example, the robot's processor obtains a matching probability of 0.9 for a pixel at an optimal disparity candidate from the matching probability volume. The robot's processor determines the disparity confidence of that pixel to be 0.9. The robot's processor obtains a matching probability of 0.4 for another pixel at an optimal disparity candidate. The robot's processor determines the disparity confidence of that pixel to be 0.4. The disparity confidence is used to subsequently determine the corresponding weight.

[0073] S36. Extract the binocular geometric features from the cost volume or regularized intermediate tensor, and transform the binocular geometric features to the same spatial resolution as the monocular semantic features.

[0074] The robot's processor extracts binocular geometric features from the cost volume or regularized intermediate tensor. Extraction methods include convolution, deconvolution, pooling, or interpolation. The binocular geometric features encompass multiple scales. The robot's processor transforms the binocular geometric features to the same spatial resolution as the monocular semantic features. Monocular semantic features include features downsampled by 8x, 16x, and 32x. The robot's processor transforms the binocular geometric features to these downsampled features using convolution, deconvolution, or interpolation. The transformed binocular geometric features have the same spatial resolution as the monocular semantic features at the same scale. The transformed binocular geometric features are then used for fusion with the monocular semantic features.

[0075] In one example, the robot's processor extracts binocular geometric features downsampled by 4x, 8x, and 16x from a regularized intermediate tensor. The processor then transforms the 4x downsampled binocular geometric features to 8x downsampled features using deconvolution. Next, it transforms the 16x downsampled binocular geometric features to 32x downsampled features using interpolation. This results in binocular geometric features downsampled by 8x, 16x, and 32x. The 8x downsampled binocular geometric features have 64 channels. The 16x downsampled binocular geometric features have 128 channels. The 32x downsampled binocular geometric features have 256 channels. Monocular semantic features also include downsampled features at 8x, 16x, and 32x downsampled levels. Both have the same spatial resolution at the same scale, enabling fusion.

[0076] In one possible implementation, obtaining the monocular semantic features of the monocular branch based on the reference image output, wherein the reference image is the left image or the right image, and the reference camera is the camera in the binocular camera that acquires the reference image, includes: S41. Input the reference image into the monocular backbone network of the monocular branch.

[0077] The robot's processor determines a reference image from the left and right images acquired by the binocular cameras. The reference image is either the left or right image. The reference camera is the one in the binocular cameras that acquired the reference image. The robot's processor loads the network parameters of the monocular branch's monocular backbone network from memory. The robot's processor inputs the reference image into the monocular backbone network. The monocular backbone network can be a convolutional neural network, a Transformer network, a visual base model, or a lightweight network. The reference image maintains the same corrected state as the binocular branch input before input. Random rotations and asymmetric padding that disrupt binocular geometry are not performed during the inference phase. The monocular backbone network is used to perform feature extraction on the reference image. The input to the monocular backbone network is the reference image, and the output is feature maps at multiple scales. These feature maps at multiple scales are used as monocular semantic features in subsequent steps.

[0078] In one example, the left image is selected as the reference image. The reference camera is the left camera. The resolution of the left image is 640×480. The monocular backbone network is a lightweight convolutional network. The robot's processor loads the monocular branch parameters trained on the server from memory. The robot's processor normalizes the left image and inputs it into the monocular backbone network. The monocular backbone network outputs feature maps downsampled by 8x, 16x, and 32x. The feature map with an 8x downsampled size has a spatial size of 80×60. The feature map with a 16x downsampled size has a spatial size of 40×30. The feature map with a 32x downsampled size has a spatial size of 20×15.

[0079] S42. The reference image is used to extract features through the monocular backbone network to obtain monocular semantic features at multiple scales.

[0080] The monocular backbone network comprises multiple downsampling stages. The robot's processor performs hierarchical feature extraction on the reference image through these downsampling stages. Each downsampling stage outputs monocular semantic features at a corresponding scale. These multi-scale monocular semantic features correspond to spatial resolutions of 8x, 16x, and 32x downsampling of the reference image, respectively. The monocular semantic features at different scales have different numbers of channels. The 8x downsampling monocular semantic features provide edge, texture, and local geometric information. The 16x downsampling monocular semantic features provide part-level semantic information. The 32x downsampling monocular semantic features provide target-level semantic information and global contextual information. The multi-scale monocular semantic features satisfy... , Indicates a reference image. This represents the feature extraction stage corresponding to the downsampling factor l in a monocular backbone network. The monocular semantic feature represents the downsampling factor l, where l takes values ​​of 8, 16, or 32.

[0081] In one example, the monocular backbone network comprises five downsampling stages. The robot's processor inputs the left image into the monocular backbone network. The monocular backbone network outputs monocular semantic features downsampled by 8x, 16x, and 32x. The monocular semantic features downsampled by 8x have 128 channels. The monocular semantic features downsampled by 16x have 256 channels. The monocular semantic features downsampled by 32x have 512 channels. The monocular semantic features downsampled by 8x preserve the boundaries and local textures of the target object. The monocular semantic features downsampled by 32x preserve the overall semantic category and global location of the target object.

[0082] S43. The monocular semantic features at the multiple scales are used as the monocular semantic features.

[0083] The robot's processor combines features from multiple scales output by the monocular backbone network into a single monocular semantic feature. This monocular semantic feature is a set of features from multiple scales. Specifically, it includes monocular semantic features downsampled by 8x, 16x, and 32x. During subsequent fusion, the robot's processor fuses the monocular semantic features at each scale with binocular geometric features of the same scale. Monocular semantic features and binocular geometric features of the same scale have the same spatial resolution. The robot's processor maintains the spatial resolution correspondence during the fusion process in this way. The robot's processor does not mix monocular semantic features of different scales into a single-scale feature.

[0084] In one example, the robot's processor combines monocular semantic features downsampled by 8 times, 16 times, and 32 times as its monocular semantic features. During subsequent fusion, the monocular semantic features downsampled by 8 times are concatenated with the binocular geometric features downsampled by 8 times. Similarly, the monocular semantic features downsampled by 16 times are concatenated with the binocular geometric features downsampled by 16 times. Finally, the monocular semantic features downsampled by 32 times are concatenated with the binocular geometric features downsampled by 32 times. The robot's processor obtains multi-scale fused features in this manner.

[0085] Using a reference image as input, monocular semantic features at multiple scales are extracted through a monocular backbone network. This enables the monocular semantic features to provide semantic information, boundary information, and local geometric information at multiple spatial resolutions, and provides a basis for subsequent fusion with binocular geometric features at the same scale. This improves the prediction accuracy of instance masks, dense standard object coordinate fields, and coordinate uncertainties, as well as the stability of pose estimation.

[0086] In one possible implementation, fusing the binocular geometric features with the monocular semantic features to obtain fused features includes: S51. For each fusion scale, the binocular geometric features are concatenated with the monocular semantic features to obtain the concatenated features for that fusion scale.

[0087] The robot's processor acquires multiple fusion scales. These fusion scales include downsampling by 8x, 16x, and 32x. For each fusion scale, the robot's processor acquires the binocular geometric features and monocular semantic features for that scale. The binocular geometric features and monocular semantic features for that scale have the same spatial resolution. The robot's processor concatenates the binocular geometric features and monocular semantic features along the channel dimension. The concatenated feature is denoted as the concatenated feature for that fusion scale. The number of channels in the concatenated feature is equal to the sum of the number of channels in the binocular geometric features and the number of channels in the monocular semantic features. The spatial resolution of the concatenated feature is the same as the spatial resolution of the binocular geometric features and monocular semantic features for that fusion scale. The robot's processor performs concatenation separately for each fusion scale. The robot's processor does not mix features from different fusion scales. The concatenated features satisfy... , Indicates the fusion scale. Indicates fusion scale Binocular geometric features, Indicates fusion scale Monocular semantic features Indicates fusion scale The splicing characteristics.

[0088] In one example, the fusion scale *l* is downsampled by 8 times. The spatial size of the binocular geometric features at this fusion scale is 80×60, with 64 channels. The spatial size of the monocular semantic features at this fusion scale is 80×60, with 128 channels. The robot's processor concatenates the binocular geometric features and monocular semantic features along the channel dimension. The spatial size of the concatenated feature is 80×60, with 192 channels. The robot's processor performs the same operation at fusion scales downsampled by 16 times and downsampled by 32 times. The concatenated feature at the fusion scale downsampled by 16 times has 384 channels. The concatenated feature at the fusion scale downsampled by 32 times has 768 channels.

[0089] S52. Perform convolution, normalization, gated linear unit operation or attention operation on the spliced ​​features to obtain the fused features at the fusion scale.

[0090] Feature transformation operations are performed on the stitched features at this fusion scale. Feature transformation operations include convolution, normalization, gated linear unit (LUnit) operations, or attention operations. The robot's processor can execute convolution and normalization operations sequentially. Alternatively, the robot's processor can execute convolution, normalization, and LUnit operations sequentially. The robot's processor can also execute attention operations. Convolution operations are used to fuse the binocular geometric features and monocular semantic features at this fusion scale. Normalization operations are used to stabilize the numerical distribution of the stitched features at this fusion scale. Gated LUnit operations are used to control the information transfer ratio between the binocular geometric features and monocular semantic features at this fusion scale. Attention operations are used to weight the binocular geometric features at this fusion scale based on the monocular semantic features. The robot's processor obtains the fusion features at this fusion scale through feature transformation operations. The spatial resolution of the fusion features at this fusion scale is the same as the spatial resolution of the stitched features at this fusion scale. The number of channels in the fusion features at this fusion scale can be set according to the network design. The fusion features at this fusion scale satisfy... , Indicates fusion scale Feature transformation operations, Indicates fusion scale The fusion characteristics.

[0091] In one example, the fusion scale *l* is downsampled by 8 times. The spatial size of the stitched feature is 80×60, and the number of channels is 192. The robot's processor performs a 1×1 convolution operation, a batch normalization operation, and a gated linear unit operation on the stitched feature in sequence. The 1×1 convolution operation transforms the number of channels from 192 to 128. The batch normalization operation normalizes the convolutional feature. The gated linear unit operation weights the normalized feature channel by channel. The robot's processor obtains a fused feature downsampled by 8 times. This fused feature has a spatial size of 80×60 and the number of channels is 128. The robot's processor performs the same feature transformation operation on fused features downsampled by 16 times and fused features downsampled by 32 times. The fused feature downsampled by 16 times has 256 channels. The fused feature downsampled by 32 times has 512 channels.

[0092] S53. The fusion features are obtained based on the fusion features of multiple fusion scales.

[0093] The robot's processor acquires fusion features at each fusion scale. These fusion features at multiple fusion scales include those downsampled by 8x, 16x, and 32x. The robot's processor combines these fusion features from multiple scales into a single fusion feature. The fusion feature is a set of fusion features from multiple scales. The robot's processor inputs the fusion feature into the task head. The task head includes a masking head and a coordinate field head. The masking head and coordinate field head can select one or more fusion features from different scales based on task requirements. The robot's processor maintains the spatial resolution and channel number correspondence of the fusion features from multiple scales. The robot's processor does not merge fusion features from multiple scales into a single scale feature. The fusion feature satisfies... F represents the fusion feature, and F8 represents the fusion feature downsampled by 8 times the fusion scale. 16 F represents the fusion feature downsampled by 16 times the fusion scale. 32 This indicates the fusion feature downsampled by 32 times the fusion scale.

[0094] While maintaining the spatial resolution correspondence between binocular geometric features and monocular semantic features, the fused features simultaneously contain both binocular stereo matching geometric information and monocular semantic information, and provide multi-scale input for the task head, thereby improving the prediction accuracy of instance masks, dense standard object coordinate fields and coordinate uncertainties, as well as the stability of pose estimation.

[0095] In one possible implementation, obtaining the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fused features includes: S61. Input the fusion feature into the masking terminal in the task header.

[0096] The robot's processor determines the features of the input masking terminal from the fused features. The fused features include features at multiple fusion scales. The robot's processor inputs the fused features into the masking terminal in the task header. The masking terminal receives fused features at one or more fusion scales. The masking terminal includes convolutional layers, normalization layers, and upsampling layers. The fused features of the input masking terminal maintain the correspondence with pixels in the reference image. The masking terminal is used to predict the instance mask of the target object. During the inference phase, the robot's processor loads the masking terminal parameters trained by the server.

[0097] In one example, the fused features include fused features downsampled at 8x, fused features downsampled at 16x, and fused features downsampled at 32x. The fused feature downsampled at 8x has a spatial size of 80×60 and 128 channels. The robot's processor inputs these three fused features into a masking terminal. The masking terminal fuses the fused features from multiple fused scales by skipping connections.

[0098] S62. The fusion feature is predicted using the masking terminal to obtain the instance mask.

[0099] The robot's processor performs feature transformation on the fused features using a masking terminal. The masking terminal performs channel transformation on the fused features using convolutional layers. The masking terminal stabilizes the feature value distribution using normalization layers. The masking terminal transforms the feature map to the spatial resolution of the reference image using upsampling layers. The masking terminal outputs an instance mask through an output layer. The instance mask includes the probability that each pixel in the reference image belongs to the target object. The instance mask is aligned with the pixel coordinate system of the reference image. The instance mask satisfies... , Indicates the instance mask. "F" indicates the merging feature.

[0100] In one example, the robot's processor inputs fused features downsampled by 8 times the fusion scale into a masking terminal. The masking terminal transforms the number of channels from 128 to 64 using a 3×3 convolution. The masking terminal normalizes the convolutional features using a batch normalization layer. The masking terminal upsamples the feature map from 80×60 to 640×480 using bilinear interpolation. The masking terminal outputs a mask probability map with 1 channel using a 1×1 convolution. Each pixel in the mask probability map corresponds to a pixel at the same location in the reference image. The mask probability ranges from 0 to 1. The robot's processor uses the mask probability map as an instance mask.

[0101] S63. Input the fusion feature into the coordinate field head in the task header.

[0102] The robot's processor determines the features of the input coordinate field head from the fused features. The robot's processor inputs the fused features into the coordinate field head in the task head. The coordinate field head receives fused features at one or more fused scales. The coordinate field head includes convolutional layers, normalization layers, upsampling layers, and an output layer. The fused features of the input coordinate field head maintain the correspondence with pixels in the reference image. The coordinate field head is used to predict the 3D standard object coordinates and coordinate uncertainties for each pixel in the target region. During the inference phase, the robot's processor loads the coordinate field head parameters trained by the server.

[0103] In one example, the robot's processor inputs fused features downsampled at 8x, 16x, and 32x scales into the coordinate field head. The fused feature downsampled at 8x scale has a spatial size of 80×60 and 128 channels. The fused feature downsampled at 16x scale has a spatial size of 40×30 and 256 channels. The fused feature downsampled at 32x scale has a spatial size of 20×15 and 512 channels. The coordinate field head fuses the fused features from multiple scales by skipping connections.

[0104] S64. The fusion feature is predicted by the coordinate field head to obtain the dense standard object coordinate field and the coordinate uncertainty.

[0105] The robot's processor performs feature transformation on the fused features through a coordinate field head. The coordinate field head performs channel transformation on the fused features through convolutional layers. The coordinate field head stabilizes the feature value distribution through a normalization layer. The coordinate field head transforms the feature map to the spatial resolution of the reference image through an upsampling layer. The coordinate field head outputs a dense standard object coordinate field and coordinate uncertainty through an output layer. The dense standard object coordinate field includes the three-dimensional standard object coordinates of each pixel in the target region of the reference image. The three-dimensional standard object coordinates include components along the three coordinate axes. The coordinate uncertainty includes the logarithmic uncertainty of each pixel in the target region of the reference image along the three coordinate axes. The dense standard object coordinate field is aligned with the pixel coordinate system of the reference image. The coordinate uncertainty is also aligned with the pixel coordinate system of the reference image. The dense standard object coordinate field satisfies... The coordinate uncertainty satisfies , Represents the coordinate field of a dense standard object. Indicates coordinate uncertainty. This represents the output branch in the coordinate field head used to predict the coordinate field of a dense standard object, and F represents the fusion feature.

[0106] In one example, the robot's processor inputs fused features downsampled by 8 times the fusion scale into the coordinate field head. The coordinate field head transforms the number of channels from 128 to 64 using a 3×3 convolution. The coordinate field head normalizes the convolutional features using a batch normalization layer. The coordinate field head upsamples the feature map from 80×60 to 640×480 using bilinear interpolation. The coordinate field head outputs a coordinate field prediction result with 6 channels using a 1×1 convolution. The first 3 channels represent the 3D standard object coordinates. The last 3 channels represent the logarithmic uncertainty in the three coordinate axes. The numerical range of the 3D standard object coordinates is 0 to 1. The robot's processor uses the first 3 channels as a dense standard object coordinate field. The robot's processor uses the last 3 channels as coordinate uncertainties.

[0107] By predicting instance masks from fused features using the masking head and predicting dense standard object coordinate fields and coordinate uncertainties from fused features using the coordinate field head, the task head can simultaneously output the target object's region information and pixel-by-pixel 3D correspondence information. This provides a basis for subsequent selection of target pixels, inverse normalization of object coordinate points, and determination of corresponding weights, thereby improving the accuracy and robustness of pose estimation.

[0108] In one possible implementation, filtering target pixels according to the instance mask in the pixel coordinate system includes: S71. Obtain the mask probability of the instance mask in the pixel coordinate system.

[0109] The robot's processor obtains the instance mask in the pixel coordinate system of the reference image. Each instance mask corresponds one-to-one with a pixel in the reference image. Each pixel in the instance mask corresponds to a mask probability. The mask probability represents the probability that the pixel belongs to the target object. The robot's processor reads the mask probability of each pixel in the instance mask. The mask probability ranges from 0 to 1. The mask probability is denoted as m. i Where i represents a pixel in the pixel coordinate system of the reference image. The robot's processor does not change the spatial resolution of the instance mask. The robot's processor does not change the correspondence between the instance mask and the pixels of the reference image.

[0110] In one example, the reference image has a resolution of 640×480. The instance mask also has a resolution of 640×480. The mask probability at pixel coordinates 320,240 is 0.9. The mask probability at pixel coordinates 100,100 is 0.2. The robot's processor reads the mask probabilities for these two pixels.

[0111] S72. Pixels with a mask probability greater than a preset threshold are identified as candidate pixels.

[0112] The robot's processor acquires a preset threshold, denoted as T. The robot's processor then calculates the mask probability m for each pixel in the instance mask.i Compare with a preset threshold T. If the pixel's mask probability m i If the pixel's mask probability is greater than a preset threshold T, the robot's processor identifies the pixel as a candidate pixel. i If the pixel is less than or equal to a preset threshold T, the robot's processor will not classify it as a candidate pixel. A candidate pixel satisfies... The robot's processor performs the above comparison on all pixels in the instance mask. The robot's processor then obtains a set of candidate pixels.

[0113] In one example, the preset threshold T is 0.55. The mask probability at pixel coordinates 320, 240. The probability is 0.9. 0.9 is greater than 0.5. The robot's processor identifies the pixel at coordinates 320, 240 as a candidate pixel. The mask probability at pixel coordinates 100, 100 is... The value is 0.2. 0.2 is less than or equal to 0.5. The robot's processor does not identify the pixel at pixel coordinates 100,100 as a candidate pixel.

[0114] S73. The candidate pixel is determined as the target pixel.

[0115] The robot's processor identifies all candidate pixels in the candidate pixel set as target pixels. These target pixels form the target pixel set. The target pixel set satisfies... Let P represent the target pixel set. Target pixels are used for subsequent backprojection to obtain camera coordinates. Target pixels are also used for subsequent inverse normalization to obtain object coordinates. Target pixels are further used to determine corresponding weights. The robot's processor does not identify pixels with a mask probability less than or equal to a preset threshold as target pixels. The robot's processor excludes background pixels and low-confidence pixels using the above methods.

[0116] In one example, there are 10,000 pixels in the instance mask with a mask probability greater than 0.5. The robot's processor identifies these 10,000 pixels as target pixels. The robot's processor uses these 10,000 pixels for subsequent backprojection and inverse normalization. Pixels with a mask probability less than or equal to 0.5 are not involved in subsequent backprojection and inverse normalization.

[0117] In one possible implementation, the step of back-projecting the target pixel to obtain camera coordinates based on the disparity map, the camera intrinsic parameters corresponding to the reference camera, and the baseline includes: S74. Obtain the disparity value of the target pixel from the disparity map.

[0118] The robot's processor acquires the target pixel in the pixel coordinate system of the reference image. The target pixel is determined by step S73. The robot's processor reads the disparity values ​​with the same pixel coordinates as the target pixel from the disparity map. The disparity map is aligned with the pixel coordinate system of the reference image. The disparity map is aligned with the instance mask, the dense standard object coordinate field, and the coordinate uncertainty to the same pixel coordinate system. The disparity value is denoted as... , This represents the target pixel. The disparity value is a non-negative real number. If the disparity value is 0, the robot's processor uses a small constant. Correction. The robot's processor reads the disparity value for each target pixel individually. The robot's processor uses the disparity values ​​to subsequently calculate the depth value.

[0119] In one example, the target pixel's pixel coordinates are 320,240. The robot's processor reads the disparity value d of that target pixel from the disparity map. i The value is 40 pixels. The pixel coordinates of another target pixel are 100,100. The robot's processor reads the disparity value d of this target pixel from the disparity map. i The value is 0 pixels. The robot's processor uses a tiny constant for this disparity value. Correction.

[0120] S75. Calculate the depth value of the target pixel based on the disparity value, the focal length in the camera intrinsic parameters corresponding to the reference camera, and the baseline.

[0121] The robot's processor obtains the focal length from the intrinsic parameters of the reference camera. The focal length in the intrinsic parameters of the reference camera is denoted as... The robot's processor acquires a baseline. The baseline is denoted as... The robot's processor uses parallax values. ,focal length The depth value of the target pixel is calculated using baseline B. The depth value is denoted as... The depth value satisfies , This represents a tiny constant to prevent division by zero. The depth value represents the depth of the target object's surface point corresponding to the target pixel in the reference camera coordinate system. The unit of the depth value is meters. The robot's processor calculates the depth value for each target pixel. The robot's processor uses the depth value for subsequent back-projection.

[0122] In one example, referencing the camera's focal length 800 pixels. Baseline. The disparity value of the target pixel is 0.12 meters. 40 pixels. Tiny constant. The value is 0.000001. The robot's processor calculates the depth value. The distance is 2.4 meters. The disparity value of another target pixel... It is 0 pixels. The robot's processor uses tiny constants. The depth value was calculated after correction.

[0123] S76. Based on the depth value and the camera intrinsic parameters corresponding to the reference camera, back-project the target pixel to obtain the camera coordinate point.

[0124] The robot's processor acquires the camera intrinsic parameter matrix of the reference camera. The camera intrinsic parameter matrix of the reference camera is denoted as... The robot's processor obtains the pixel coordinates of the target pixel in the pixel coordinate system of the reference image. The pixel coordinates of the target pixel are denoted as... and The robot's processor uses depth values. and camera intrinsic parameter matrix Back-projection is performed on the target pixels. The camera coordinates are denoted as... The camera coordinates satisfy... , This represents the inverse of the camera intrinsic parameter matrix. Camera coordinates represent the 3D coordinates of the target object surface point corresponding to the target pixel in the reference camera coordinate system. The robot's processor calculates the camera coordinates for each target pixel. The robot's processor uses the camera coordinates for subsequent weighted pose calculation.

[0125] The disparity value of the target pixel is obtained from the disparity map. The depth value of the target pixel is calculated based on the focal length and baseline of the reference camera. The target pixel is then back-projected using the depth value and the camera intrinsic parameters of the reference camera to obtain the camera coordinates. This process transforms the disparity map into 3D camera coordinates in the reference camera coordinate system, providing reliable 3D correspondence data for subsequent weighted pose estimation with object coordinates and corresponding weights, thereby improving the accuracy and robustness of pose estimation.

[0126] In one possible implementation, obtaining the object coordinate points based on the dense standard object coordinate field and the three-dimensional model parameters through inverse normalization includes: S81. Obtain the minimum bounding box value and side length vector from the three-dimensional model parameters.

[0127] The robot's processor reads the 3D model parameters of the target object from memory. These parameters include the minimum bounding box value and side length vector of the target object in its own standard object coordinate system. The minimum bounding box value is denoted as... The minimum bounding box value represents the minimum coordinates of the target object's 3D model along the three coordinate axes. The side length vector is denoted as... The side length vector represents the length of the target object's 3D model along the three coordinate axes. The robot's processor uses the bounding box minimum and side length vector for subsequent inverse normalization. The bounding box minimum and side length vector remain unchanged during the inference phase. The robot's processor does not modify the 3D model parameters. The robot's processor uses the same bounding box minimum and side length vector for each target pixel.

[0128] In one example, the target object is a bin. Minimum bounding box value in the 3D model parameters. for[ 0.1, [0.05, 0]. Side length vector The bounding box minimum values ​​are [0.2, 0.1, 0.15]. The three values ​​in the bounding box minimum value represent the minimum coordinates of the bin along the three coordinate axes in the standard object coordinate system. The three values ​​in the side length vector represent the lengths of the bin along the three coordinate axes. The robot's processor reads these parameters from memory and uses them for subsequent inverse normalization.

[0129] S82. Obtain the three-dimensional standard object coordinates of the target pixel from the dense standard object coordinate field.

[0130] The robot's processor acquires the target pixel in the pixel coordinate system of the reference image. The target pixel is determined by step S73. The robot's processor reads the coordinates of a 3D standard object that has the same pixel coordinates as the target pixel from a dense standard object coordinate field. The dense standard object coordinate field is aligned with the pixel coordinate system of the reference image. The dense standard object coordinate field is aligned with the instance mask and coordinate uncertainty to the same pixel coordinate system. The 3D standard object coordinates are denoted as... Here, 'i' represents the target pixel. The 3D standard object coordinates include components along three coordinate axes. The numerical range of the 3D standard object coordinates is 0 to 1. The robot's processor reads the 3D standard object coordinates for each target pixel. The robot's processor uses the 3D standard object coordinates for subsequent inverse normalization.

[0131] In one example, the target pixel's pixel coordinates are 320,240. The robot's processor reads the target pixel's three-dimensional standard object coordinates from a dense standard object coordinate field. The coordinates of the target pixel are [0.5, 0.5, 0.5]. The pixel coordinates of the other target pixel are 100, 100. The robot's processor reads the three-dimensional standard object coordinates of the target pixel [0.2, 0.3, 0.4] from the dense standard object coordinate field. The numerical range of the three-dimensional standard object coordinates is 0 to 1.

[0132] S83. Multiply the coordinates of the three-dimensional standard object element by element with the side length vector to obtain the scaling coordinates.

[0133] The robot's processor obtains the three-dimensional standard object coordinates of the target pixel. The robot's processor obtains the side length vector of the target object. The robot's processor will use the coordinates of a three-dimensional standard object. with side length vector Element-wise multiplication represents multiplying the components along each of the three coordinate axes. The robot's processor obtains the scaled coordinates. The scaled coordinates are denoted as... , This indicates element-wise multiplication. The scaling coordinates represent the offset of the target pixel's corresponding point on the object's surface relative to the minimum bounding box value along the three coordinate axes. The robot's processor calculates the scaling coordinates for each target pixel. The robot's processor then uses these scaling coordinates to add to the minimum bounding box value in subsequent operations.

[0134] In one example, the three-dimensional standard object coordinates of the target pixel The side length vector is [0.5, 0.5, 0.5]. The result is [0.2, 0.1, 0.15]. and Element-wise multiplication. Scaling coordinates are [0.5×0.2, 0.5×0.1, 0.5×0.15]. Scaling coordinates are [0.1, 0.05, 0.075]. The three values ​​in the scaling coordinates represent the offset of the target pixel's corresponding object surface point relative to the minimum value of the bounding box in the three coordinate axes.

[0135] S84. Add the scaling coordinates to the minimum value of the bounding box to obtain the object coordinate point corresponding to the target pixel.

[0136] Obtain the scaling coordinates of the target pixel. The robot's processor obtains the minimum bounding box value of the target object. Scaling coordinates and minimum bounding box value Addition. Addition means adding the components along the three coordinate axes separately. The robot's processor obtains the object coordinates corresponding to the target pixel. The object coordinates are denoted as... The object's coordinates satisfy The object coordinates represent the three-dimensional coordinates of the point on the surface of the target object corresponding to the target pixel in the target object's own standard object coordinate system. The object coordinates are metric coordinates. The unit of the object coordinates is meters. The robot's processor calculates the object coordinates for each target pixel. The robot's processor uses the object coordinates for subsequent weighted pose calculations.

[0137] In one example, the scaling coordinates of the target pixel are [0.1, 0.05, 0.075]. The minimum bounding box value p min for[ 0.1, 0.05, 0]. The robot's processor adds the scaled coordinates to the minimum bounding box value. The object's coordinates are [ 0.1+0.1, [0.05+0.05, 0+0.075]. The object coordinates are [0, 0, 0.075]. The object coordinates represent the three-dimensional coordinates of the point on the surface of the hopper corresponding to the target pixel in the hopper's own standard object coordinate system. The robot's processor calculates the object coordinates for each target pixel. The object coordinates of all target pixels constitute the set of object coordinates.

[0138] The normalized 3D standard object coordinates are converted into metric object coordinates in the target object's own standard object coordinate system, providing reliable 3D object coordinate data for subsequent weighted pose estimation with camera coordinates and corresponding weights, thereby improving the accuracy and robustness of pose estimation.

[0139] In one possible implementation, determining the corresponding weight for each target pixel based on the instance mask, the coordinate uncertainty, and the disparity confidence includes: S85. Obtain the mask probability of the target pixel from the instance mask.

[0140] The target pixel is obtained in the pixel coordinate system of the reference image. The target pixel is determined by step S73. The robot's processor reads the mask probability with the same pixel coordinates as the target pixel from the instance mask. The instance mask is aligned with the pixel coordinate system of the reference image. The instance mask is aligned with the dense standard object coordinate field, coordinate uncertainty, and disparity confidence to the same pixel coordinate system. The mask probability is denoted as m. i Where i represents the target pixel. The mask probability represents the probability that a target pixel belongs to the target object. The mask probability ranges from 0 to 1. The robot's processor reads the mask probability for each target pixel. The robot's processor uses the mask probability to calculate the corresponding weights in subsequent calculations.

[0141] In one example, the target pixel's pixel coordinates are 320, 240. The robot's processor reads the mask probability m of that target pixel from the instance mask. i The value is 0.9. The pixel coordinates of another target pixel are 100,100. The robot's processor reads the mask probability m of this target pixel from the instance mask. i The value is 0.6. The numerical range of the mask probability is 0 to 1.

[0142] S86. Take the negative exponent of the coordinate uncertainty to obtain the uncertainty factor of the target pixel.

[0143] The robot's processor obtains the coordinate uncertainty of the target pixel in the three coordinate axes from the coordinate uncertainty. The coordinate uncertainties in the three coordinate axes are denoted as follows: , and The coordinate uncertainty is a logarithmic uncertainty. The robot's processor calculates the average of the three coordinate uncertainties. The robot's processor then takes the exponentiation of the negative value of the average. The robot's processor obtains the uncertainty factor of the target pixel. The uncertainty factor is denoted as... , exp() represents an exponential function with the natural constant e as its base, i.e., exp(x) equals e raised to the power of x, where e is approximately 2.71828. The uncertainty factor represents the reliability of the predicted 3D standard object coordinates of the target pixel. The smaller the coordinate uncertainty, the larger the uncertainty factor. The robot's processor calculates the uncertainty factor for each target pixel. The robot's processor uses the uncertainty factor in subsequent calculations of the corresponding weights.

[0144] In one example, the coordinate uncertainty of the target pixel in the three coordinate axes. , and All are 0.1. The robot's processor calculates the average of the three coordinate uncertainties as 0.1. The robot's processor then exponentiation of the negative of the average. Uncertainty factor The uncertainty factor is exp(-0.1). The uncertainty factor is approximately 0.9048. The coordinate uncertainty of the other target pixel is 0.5 in all three coordinate axes. The robot's processor calculates the uncertainty factor. For exp( 0.5), approximately 0.6065.

[0145] S87. Obtain the confidence value of the target pixel in the disparity confidence.

[0146] The robot's processor obtains the confidence values ​​from the disparity confidence values ​​that have the same pixel coordinates as the target pixel. There is a one-to-one correspondence between the disparity confidence values ​​and the pixels in the disparity map. The disparity confidence values ​​are aligned with the pixel coordinate system of the reference image. The disparity confidence values ​​are also aligned with the instance mask, the dense standard object coordinate field, and the coordinate uncertainty to the same pixel coordinate system. The confidence value is denoted as... Here, i represents the target pixel. The confidence value indicates the reliability of the disparity value of the target pixel. The confidence value ranges from 0 to 1. The robot's processor reads the confidence value for each target pixel. The robot's processor uses the confidence value to calculate the corresponding weight in subsequent calculations.

[0147] In one example, the target pixel's pixel coordinates are 320,240. The robot's processor reads the confidence value of that target pixel from the disparity confidence. The confidence level is 0.8. The pixel coordinates of the other target pixel are 100,100. The robot's processor reads the confidence value of this target pixel from the disparity confidence score. The confidence level is 0.4. The range of confidence values ​​is 0 to 1.

[0148] S88. Multiply the mask probability, the uncertainty factor and the confidence value to obtain the corresponding weight of the target pixel.

[0149] The robot's processor obtains the mask probability of the target pixel. The robot's processor obtains the uncertainty factor of the target pixel. The robot's processor obtains the confidence value of the target pixel. The robot's processor multiplies the mask probability, uncertainty factor, and confidence value to obtain the corresponding weight of the target pixel. , The corresponding weights can be further multiplied by the optional visibility probability. If the visibility probability is not set, then The value is set to 1. The corresponding weight represents the contribution of the target pixel in the weighted pose solution. The robot's processor calculates the corresponding weight for each target pixel. The robot's processor then uses these weights for subsequent weighted pose solutions.

[0150] In one example, the mask probability of the target pixel The uncertainty factor is 0.9. The confidence level is 0.9048. The value is 0.8. The robot's processor multiplies these three values ​​together. The corresponding weights are... The value is 0.9 × 0.9048 × 0.8, approximately 0.6515. This is the mask probability of the other target pixel. The uncertainty factor is 0.6. The confidence level is 0.6065. The weight is 0.4. The robot's processor calculates the corresponding weight. The value is approximately 0.1456. The robot's processor calculates the corresponding weight for each target pixel. The corresponding weights of all target pixels constitute a set of corresponding weights.

[0151] By considering the probability that a target pixel belongs to the target object, the reliability of the 3D standard object coordinates, and the reliability of the disparity value, the corresponding weights can reflect the credibility of the target pixel in the weighted pose solution. This reduces the impact of unreliable target pixels in occluded boundaries, reflective areas, and textureless areas on the pose solution, thereby improving the accuracy and robustness of pose estimation.

[0152] In one possible implementation, the step of performing weighted pose calculation based on the camera coordinates, the object coordinates, and the corresponding weights to obtain the pose data of the target object relative to the reference camera includes: S91. According to the corresponding weights, the object coordinate points are weighted and averaged to obtain the object coordinate centroid.

[0153] The robot's processor acquires the object coordinates and corresponding weight of each target pixel in the target pixel set. The processor calculates the sum of the corresponding weights for all target pixels. For each target pixel, the processor multiplies its object coordinates by its corresponding weight to obtain the weighted object coordinates. The processor sums the weighted object coordinates of all target pixels. The processor divides the sum by the sum of the corresponding weights to obtain the object centroid. The object centroid is denoted as […]. The object's centroid satisfies... , where i represents the index of the target pixel. This represents the corresponding weight of the target pixel. This represents the object's coordinates at the target pixel. The robot's processor uses the object's centroid coordinates for subsequent calculations of the weighted covariance matrix.

[0154] In one example, the target pixel set includes 3 target pixels. The object coordinates of the first target pixel are [0, 0, 0.075], with a weight of 0.6. The object coordinates of the second target pixel are [0.1, 0, 0.075], with a weight of 0.3. The object coordinates of the third target pixel are [0, 0.05, 0.075], with a weight of 0.1. The sum of the corresponding weights is 1.0. The robot's processor calculates the weighted sum of the object coordinates as [0×0.6+0.1×0.3+0×0.1,0×0.6+0×0.3+0.05×0.1,0.075×0.6+0.075×0.3+0.075×0.1], which is [0.03,0.005,0.075]. The robot's processor divides the weighted sum of the object coordinates by the sum of the corresponding weights, 1.0, to obtain the centroid of the object coordinates. The result is [0.03, 0.005, 0.075].

[0155] S92. According to the corresponding weights, perform a weighted average on the camera coordinate points to obtain the camera coordinate centroid.

[0156] The robot's processor acquires the camera coordinates and corresponding weight of each target pixel in the target pixel set. The processor calculates the sum of the corresponding weights for all target pixels. For each target pixel, the processor multiplies its camera coordinates by its corresponding weight to obtain the weighted camera coordinates. The processor sums the weighted camera coordinates of all target pixels. The processor divides the sum by the sum of the corresponding weights to obtain the camera centroid. The camera centroid is denoted as […]. The camera's centroid satisfies... , where i represents the index of the target pixel. This represents the corresponding weight of the target pixel. This represents the camera coordinates of the target pixel. The robot's processor uses the camera centroid for subsequent calculations of the weighted covariance matrix.

[0157] In one example, the target pixel set includes 3 target pixels. The camera coordinates of the first target pixel are [0, 0, 2.4], with a weight of 0.6. The camera coordinates of the second target pixel are [0.1, 0, 2.4], with a weight of 0.3. The camera coordinates of the third target pixel are [0, 0.05, 2.4], with a weight of 0.1. The sum of the corresponding weights is 1.0. The robot's processor calculates the weighted sum of the camera coordinates as [0×0.6+0.1×0.3+0×0.1, 0×0.6+0×0.3+0.05×0.1, 2.4×0.6+2.4×0.3+2.4×0.1], which is [0.03, 0.005, 2.4]. The robot's processor divides the weighted sum of the camera coordinates by the sum of the corresponding weights, 1.0, to obtain the centroid of the camera coordinates. The result is [0.03, 0.005, 2.4].

[0158] S93. Calculate the weighted covariance matrix based on the object coordinates, the camera coordinates, the object centroid, the camera centroid, and the corresponding weights.

[0159] The robot's processor acquires the object coordinates, camera coordinates, object centroid, camera centroid, and corresponding weights for each target pixel. For each target pixel, the processor calculates the difference between its object coordinates and centroid, obtaining the object coordinate difference value. It then calculates the difference between its camera coordinates and centroid, obtaining the camera coordinate difference value. Next, the processor calculates the transpose of the object coordinate difference and camera coordinate difference, obtaining the coordinate difference product for that target pixel. Finally, the processor multiplies this coordinate difference product by the corresponding weight of the target pixel, obtaining the weighted coordinate difference product. Finally, the processor sums the weighted coordinate difference products of all target pixels to obtain the weighted covariance matrix. The weighted covariance matrix is ​​denoted as... The weighted covariance matrix satisfies , where i represents the index of the target pixel. This represents the corresponding weight of the target pixel. The object coordinates representing the target pixel. Represents the centroid of the object's coordinates. The camera coordinates of the target pixel. This represents the centroid of the camera coordinates. The weighted covariance matrix is ​​a 3x3 matrix. The robot's processor uses the weighted covariance matrix for subsequent singular value decomposition.

[0160] In one example, the target pixel set includes 3 target pixels. The object coordinate difference of the first target pixel is [ 0.03, [0.005,0], the camera coordinate difference is [ 0.03, The object coordinate difference between the second target pixel and the first target pixel is [0.07, 0], corresponding to a weight of 0.6. The camera coordinate difference is [0.005, 0], and the difference between the camera coordinates is [0.07, 0]. [0.005, 0], corresponding to a weight of 0.3. The object coordinate difference of the 3rd target pixel is [ The camera coordinate difference is [0.03, 0.045, 0]. The values ​​[0.03, 0.045, 0] correspond to a weight of 0.1. The robot's processor calculates the coordinate difference product for each target pixel and multiplies it by the corresponding weight. The robot's processor then sums all the weighted coordinate difference products to obtain the weighted covariance matrix H.

[0161] S94. Perform singular value decomposition on the weighted covariance matrix to obtain the rotation matrix.

[0162] The robot's processor acquires the weighted covariance matrix. The processor then performs singular value decomposition on the weighted covariance matrix. The singular value decomposition satisfies... H represents the weighted covariance matrix, and U represents the left singular vector matrix. Represents a singular value diagonal matrix. Let R represent the transpose of the right singular vector matrix. The robot's processor determines the rotation matrix based on the singular value decomposition results. The rotation matrix is ​​denoted as R. The rotation matrix satisfies... , diag means to construct a diagonal matrix, det means the determinant of the matrix, This represents the transpose of the left singular vector matrix. The robot's processor corrects the reflection using determinant constraints, ensuring the determinant of the rotation matrix is ​​1. The robot's processor then uses the rotation matrix to calculate the translation vector.

[0163] S95. Calculate the translation vector based on the rotation matrix, the camera coordinate centroid, and the object coordinate centroid.

[0164] The robot's processor acquires the rotation matrix, the camera's centroid, and the object's centroid. The processor multiplies the rotation matrix by the object's centroid to obtain the rotated object centroid. The processor then subtracts the rotated object centroid from the camera's centroid to obtain the translation vector. This translation vector is denoted as t and satisfies the following condition: , R represents the centroid of the camera coordinate system, and R represents the rotation matrix. The centroid of the object's coordinates is represented by the translation vector. The translation vector represents the translation of the target object relative to the reference camera. The robot's processor uses the translation vector as pose data in subsequent steps.

[0165] In one example, the rotation matrix R is the identity matrix. (Camera coordinate centroid) The centroid of the object is [0.03, 0.005, 2.4]. The value is [0.03, 0.005, 0.075]. The robot's processor calculates the centroid of the rotated object as [0.03, 0.005, 0.075]. The robot's processor subtracts the centroid of the rotated object from the centroid of the camera coordinates to obtain the translation vector t as [0, 0, 2.325]. The translation vector represents the translation of the target object relative to the reference camera in the three coordinate axes.

[0166] S96. Use the rotation matrix and the translation vector as the pose data.

[0167] The robot's processor acquires the rotation matrix and translation vector. The processor uses these as the pose data of the target object relative to the reference camera. The pose data includes the rotation matrix R and the translation vector t. This pose data represents the six-degree-of-freedom pose of the target object in the reference camera coordinate system. The processor can also update the corresponding weights based on the residuals and repeat the weighted pose calculation 1 to K times. The processor calculates the similarity transformation scale when the absolute size of the target object is unknown. The similarity transformation scale is denoted as s. The similarity transformation scale satisfies... tr represents the trace of the matrix. The robot's processor updates the translation vector according to the similarity transformation scale, and the updated translation vector satisfies... The robot's processor uses the rotation matrix, translation vector, and similarity transformation scale as pose data. Based on this pose data, the robot's processor generates robot control commands.

[0168] In one example, the robot's processor obtains a rotation matrix R as an identity matrix and a translation vector [0,0,2.325]. The robot's processor uses the rotation matrix R and the translation vector t as pose data. If the absolute size of the target object is unknown, the robot's processor solves for a similarity transformation scale s of 1.0. The robot's processor outputs the pose data. The pose data is used for robot gripping, assembly, bin picking, loading / unloading, quality inspection, or vision-guided positioning.

[0169] By using weighted pose estimation, the corresponding weight of each target pixel is fully utilized, reducing the impact of unreliable target pixels on the pose estimation. Furthermore, the similarity transformation scale is calculated when the absolute size of the target object is unknown, thereby improving the accuracy and robustness of the pose estimation of the target object relative to the reference camera.

[0170] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0171] Please see Figure 3 This illustration shows a schematic diagram of a binocular dense pose estimation device provided in an exemplary embodiment of this application, hereinafter referred to as device 3. Device 3 can be implemented as all or part of a robot through software, hardware, or a combination of both. Device 3 includes: The acquisition module 301 is used to acquire the left and right images captured by the binocular camera, the camera intrinsic parameters and baseline of the binocular camera, and the three-dimensional model parameters of the target object. Inference module 302 is used to input the left image and the right image into a pre-trained multi-task neural network; the multi-task neural network includes a binocular branch, a monocular branch, and a task head; The inference module 302 is also used to obtain the disparity map, disparity confidence and binocular geometric features output by the binocular branch based on the left image and the right image; The reasoning module 302 is also used to obtain the monocular semantic features output by the monocular branch based on the reference image, wherein the reference image is the left image or the right image, and the reference camera is the camera in the binocular camera that acquires the reference image; The reasoning module 302 is further configured to fuse the binocular geometric features with the monocular semantic features to obtain fused features; Alignment module 303 is used to obtain the instance mask, dense standard object coordinate field and coordinate uncertainty output by the task head based on the fusion feature, and align the instance mask, the dense standard object coordinate field and the coordinate uncertainty to the pixel coordinate system of the reference image; The calculation module 304 is used to filter target pixels in the pixel coordinate system according to the instance mask, and back-project the target pixels to obtain camera coordinate points according to the disparity map, the camera intrinsic parameters corresponding to the reference camera and the baseline. The determination module 305 is used to obtain the object coordinate points based on the dense standard object coordinate field and the three-dimensional model parameters through inverse normalization, and to determine the corresponding weight of each target pixel based on the instance mask, the coordinate uncertainty and the disparity confidence. The estimation module 306 is used to perform weighted pose calculation based on the camera coordinate points, the object coordinate points and the corresponding weights to obtain the pose data of the target object relative to the reference camera.

[0172] For other details regarding the implementation of the above technical solutions by each module in the above-mentioned binocular dense pose estimation device, please refer to the description of the binocular dense pose estimation method provided in the above-mentioned embodiments of the invention, which will not be repeated here.

[0173] It should be noted that the device 3 provided in the above embodiments is only illustrated by the division of the above functional modules when performing the binocular dense pose estimation method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the above functions. In addition, the binocular dense pose estimation device and the binocular dense pose estimation method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0174] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0175] See Figure 4The diagram shown is a schematic of a computer storage medium provided in an embodiment of this application. The computer storage medium 4 can store multiple instructions (i.e., Figure 4 The computer program shown above), the instructions are adapted to be loaded and executed by a processor as described above. Figure 2 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figure 2 The specific details of the illustrated embodiments will not be elaborated here.

[0176] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor to implement the binocular dense pose estimation method as described in the above embodiments.

[0177] Please see Figure 5 This provides a schematic diagram of the robot's structure as an embodiment of this application. Figure 5 As shown, the robot 500 may include: at least one processor 501, at least one network interface 504, user interface 503, memory 505, and at least one communication bus 502.

[0178] The communication bus 502 is used to enable communication between these components.

[0179] The user interface 503 may include input units such as a mouse and keyboard.

[0180] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0181] The processor 501 may include one or more processing cores. The processor 501 connects to various parts within the robot 500 using various interfaces and lines, and performs various functions and processes data of the robot 500 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and by calling data stored in the memory 505. Optionally, the processor 501 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.

[0182] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs.

[0183] exist Figure 5In the robot 500 shown, the user interface 503 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 501 can be used to call the application program stored in the memory 505 and specifically execute, such as Figure 2 The method shown can be referred to for details. Figure 2 As shown, it will not be elaborated further here.

[0184] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0185] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A binocular dense pose estimation method, characterized in that, include: Acquire the left and right images captured by the binocular camera, the camera intrinsic parameters and baseline of the binocular camera, and the three-dimensional model parameters of the target object; The left and right images are input into a pre-trained multi-task neural network; the multi-task neural network includes a binocular branch, a monocular branch, and a task head. Obtain the disparity map, disparity confidence, and binocular geometric features output by the binocular branch based on the left and right images; Obtain the monocular semantic features output by the monocular branch based on the reference image, wherein the reference image is the left image or the right image, and the reference camera is the camera in the binocular camera that acquires the reference image; The binocular geometric features are fused with the monocular semantic features to obtain fused features; Obtain the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fusion features, and align the instance mask, the dense standard object coordinate field, and the coordinate uncertainty to the pixel coordinate system of the reference image; In the pixel coordinate system, target pixels are filtered according to the instance mask, and the target pixels are back-projected to obtain camera coordinate points according to the disparity map, the camera intrinsic parameters corresponding to the reference camera, and the baseline. The object coordinates are obtained by inverse normalization of the dense standard object coordinate field and the three-dimensional model parameters, and the corresponding weight of each target pixel is determined by the instance mask, the coordinate uncertainty and the disparity confidence. The pose data of the target object relative to the reference camera is obtained by performing a weighted pose calculation based on the camera coordinates, the object coordinates, and the corresponding weights.

2. The method according to claim 1, characterized in that, The step of obtaining the disparity map, disparity confidence, and binocular geometric features output by the binocular branch based on the left and right images includes: The left image and the right image are input into the binocular backbone network of the binocular branch to obtain the left feature map and the right feature map; Construct a cost body based on the left and right feature maps; The cost volume is regularized to obtain the initial disparity; The initial disparity is refined to obtain the disparity map; The disparity confidence level is determined based on the cost volume or the initial disparity. The binocular geometric features are extracted from the cost volume or regularized intermediate tensor, and the binocular geometric features are transformed to the same spatial resolution as the monocular semantic features.

3. The method according to claim 2, characterized in that, The construction of the cost body based on the left feature map and the right feature map includes: The left and right feature maps are input into the correlation layer; The matching cost between the left and right feature maps is calculated along multiple disparity candidate directions through the relevant layer. The cost body is constructed based on the matching cost.

4. The method according to claim 1, characterized in that, The step of obtaining the monocular semantic features of the monocular branch based on the reference image, wherein the reference image is the left image or the right image, and the reference camera is the camera in the binocular camera that acquires the reference image, includes: The reference image is input into the monocular backbone network of the monocular branch; The reference image is used to extract features through the monocular backbone network to obtain monocular semantic features at multiple scales. The monocular semantic features at the multiple scales are used as the monocular semantic features.

5. The method according to claim 1, characterized in that, The process of fusing the binocular geometric features with the monocular semantic features to obtain fused features includes: For each fusion scale, the binocular geometric features are concatenated with the monocular semantic features to obtain the concatenated features for that fusion scale. Perform convolution, normalization, gated linear unit operation or attention operation on the spliced ​​features to obtain the fused features at the fusion scale; The fusion features are obtained based on the fusion features of multiple fusion scales.

6. The method according to claim 1, characterized in that, The process of obtaining the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fused features includes: The fusion feature is input into the masking terminal in the task header; The instance mask is obtained by predicting the fused features using the masking tool; The fusion features are input into the coordinate field header in the task header; The fused features are predicted using the coordinate field head to obtain the dense standard object coordinate field and the coordinate uncertainty.

7. The method according to claim 1, characterized in that, The step of filtering target pixels according to the instance mask in the pixel coordinate system includes: Obtain the mask probability of the instance mask in the pixel coordinate system; Pixels whose mask probability is greater than a preset threshold are identified as candidate pixels; The candidate pixel is determined as the target pixel.

8. The method according to claim 1, characterized in that, The step of back-projecting the target pixel to obtain camera coordinates based on the disparity map, the camera intrinsic parameters corresponding to the reference camera, and the baseline includes: Obtain the disparity value of the target pixel from the disparity map; The depth value of the target pixel is calculated based on the disparity value, the focal length in the camera intrinsic parameters corresponding to the reference camera, and the baseline. Based on the depth value and the camera intrinsic parameters corresponding to the reference camera, the target pixel is back-projected to obtain the camera coordinate point.

9. The method according to claim 8, characterized in that, The step of calculating the depth value of the target pixel based on the disparity value, the focal length in the camera intrinsic parameters corresponding to the reference camera, and the baseline includes: Obtain the focal length from the camera intrinsic parameters corresponding to the reference camera; Obtain the baseline; Calculate the product of the focal length and the baseline; Divide the product by the disparity value to obtain the depth value of the target pixel.

10. The method according to claim 1, characterized in that, The process of obtaining object coordinate points based on the dense standard object coordinate field and the three-dimensional model parameters through inverse normalization includes: Obtain the minimum bounding box value and side length vector from the three-dimensional model parameters; Obtain the three-dimensional standard object coordinates of the target pixel from the dense standard object coordinate field; The scaling coordinates are obtained by multiplying the three-dimensional standard object coordinates element by element with the side length vector; The scaling coordinates are added to the minimum value of the bounding box to obtain the object coordinates corresponding to the target pixel.

11. The method according to claim 1, characterized in that, The step of determining the corresponding weight of each target pixel based on the instance mask, the coordinate uncertainty, and the disparity confidence includes: Obtain the mask probability of the target pixel from the instance mask; Taking the negative exponent of the coordinate uncertainty yields the uncertainty factor of the target pixel; Obtain the confidence value of the target pixel in the disparity confidence; The weight of the target pixel is obtained by multiplying the mask probability, the uncertainty factor, and the confidence value.

12. The method according to claim 1, characterized in that, The step of performing weighted pose calculation based on the camera coordinates, the object coordinates, and the corresponding weights to obtain the pose data of the target object relative to the reference camera includes: Based on the corresponding weights, the object coordinate points are weighted and averaged to obtain the object's centroid. Based on the corresponding weights, the camera coordinate points are weighted and averaged to obtain the camera coordinate centroid. Calculate the weighted covariance matrix based on the object coordinates, the camera coordinates, the object centroid, the camera centroid, and the corresponding weights; Singular value decomposition is performed on the weighted covariance matrix to obtain the rotation matrix; Calculate the translation vector based on the rotation matrix, the camera coordinate centroid, and the object coordinate centroid; The rotation matrix and the translation vector are used as the pose data.

13. A binocular dense pose estimation device, characterized in that, include: The acquisition module is used to acquire the left and right images captured by the binocular camera, the camera intrinsic parameters and baseline of the binocular camera, and the three-dimensional model parameters of the target object; An inference module is used to input the left image and the right image into a pre-trained multi-task neural network; the multi-task neural network includes a binocular branch, a monocular branch, and a task head; The inference module is also used to obtain the disparity map, disparity confidence and binocular geometric features output by the binocular branch based on the left image and the right image; The reasoning module is also used to obtain the monocular semantic features output by the monocular branch based on the reference image, wherein the reference image is the left image or the right image, and the reference camera is the camera in the binocular camera that acquires the reference image; The reasoning module is also used to fuse the binocular geometric features with the monocular semantic features to obtain fused features; The alignment module is used to obtain the instance mask, dense standard object coordinate field, and coordinate uncertainty output by the task head based on the fusion features, and to align the instance mask, the dense standard object coordinate field, and the coordinate uncertainty to the pixel coordinate system of the reference image; The calculation module is used to filter target pixels in the pixel coordinate system according to the instance mask, and back-project the target pixels to obtain camera coordinate points according to the disparity map, the camera intrinsic parameters corresponding to the reference camera and the baseline; The determination module is used to obtain object coordinate points based on the dense standard object coordinate field and the three-dimensional model parameters through inverse normalization, and to determine the corresponding weight of each target pixel based on the instance mask, the coordinate uncertainty and the disparity confidence. The estimation module is used to perform weighted pose calculation based on the camera coordinates, the object coordinates, and the corresponding weights to obtain the pose data of the target object relative to the reference camera.

14. A robot, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the binocular dense pose estimation method as described in any one of claims 1 to 12.

Images