Neural network-based three-dimensional key point pose estimation method and robot

Abstract

The application discloses a three-dimensional key point posture estimation method based on a neural network and a robot. The method comprises the following steps: collecting a left-eye image and a right-eye image through a binocular camera; inputting the left-eye image and the right-eye image into a target key point detection model respectively to obtain left-eye two-dimensional key points and right-eye two-dimensional key points, wherein the target key point detection model is obtained by training an initial key point detection model by using multiple sets of training data and a target training loss, and each set of training data in the multiple sets of training data comprises a synthetic binocular image and corresponding key point true values; performing triangulation calculation based on the left-eye two-dimensional key points, the right-eye two-dimensional key points and calibration parameters of the binocular camera to obtain three-dimensional key point coordinates of a target object in a left-eye coordinate system. The application solves the technical problems of complex hardware deployment and poor simulation migration in the related art when performing three-dimensional key point posture estimation.

Description

Technical Field

[0001] This application relates to the fields of computer vision and robot perception technology, and more specifically, to a three-dimensional keypoint pose estimation method based on neural networks and a robot. Background Technology

[0002] 3D keypoint pose estimation for humanoid robots is fundamental to motion control and safe interaction. Most existing solutions rely on optical motion capture, inertial kits, or multi-camera calibration fields to obtain ground truth values, or employ simplified mesh simulation training. However, these motion capture and multi-camera solutions are costly in hardware and complex to deploy, making them difficult to adapt to real-world scenarios with only a head-mounted binocular camera. Furthermore, the significant difference between simplified simulations and real-world appearance makes it difficult to transfer simulations to reality.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a neural network-based method and robot for 3D keypoint pose estimation, which at least solves the technical problems of complex hardware deployment and poor simulation transferability in related technologies when performing 3D keypoint pose estimation.

[0005] According to one aspect of the embodiments of this application, a method for estimating the pose of three-dimensional keypoints based on neural networks is provided, comprising: acquiring left-eye and right-eye images through a binocular camera; inputting the left-eye and right-eye images into a target keypoint detection model respectively to obtain left-eye two-dimensional keypoints and right-eye two-dimensional keypoints, wherein the target keypoint detection model is trained on an initial keypoint detection model using multiple sets of training data and a target training loss, each set of training data including: a synthesized binocular image and the corresponding ground truth value of keypoints, and the target training loss being obtained by weighted calculation based on two-dimensional supervision loss, reprojection consistency loss and left-right consistency loss; performing triangulation calculation based on the left-eye two-dimensional keypoints, the right-eye two-dimensional keypoints and the calibration parameters of the binocular camera to obtain the three-dimensional keypoint coordinates of the target object in the left-eye coordinate system, wherein the calibration parameters of the binocular camera include at least one of the following: focal length, baseline length and principal point coordinates.

[0006] Optionally, the 3D keypoint pose estimation method further includes: sparse reconstruction of multi-view images of the target object to obtain sparse 3D points; pixel-level depth estimation of the multi-view images to obtain pixel-level depth; scale offset calibration of the pixel-level depth based on the projected depth of the sparse 3D points to obtain a calibration depth; determining the 3D representation data of the target object based on the calibration depth, and registering and aligning the 3D representation data of the target object with a preset simulation model to obtain a component-level 3D asset; updating the joint state of the component-level 3D asset through a simulation engine, rendering and generating a synthetic stereo image and the corresponding ground truth of keypoints; and training an initial keypoint detection model based on the synthetic stereo image and the ground truth of keypoints to obtain a target keypoint detection model.

[0007] Optionally, using the projected depth of sparse 3D points as a reference, scale offset calibration is performed on the pixel-level depth to obtain the calibration depth. This includes: constructing a depth calibration optimization objective based on the projected depth of sparse 3D points, the pixel-level depth, and geometric constraint weights, where the depth calibration optimization objective represents the minimum value of the weighted residual sum of squares; solving the depth calibration optimization objective to obtain the target scale coefficient and the target offset; and using the target scale coefficient and the target offset, performing a linear transformation on the pixel-level depth of all pixels in the image to obtain the calibration depth.

[0008] Optionally, registering and aligning the 3D representation data of the target object with a preset simulation model to obtain component-level 3D assets includes: downsampling the 3D representation data to obtain target point cloud data, and downsampling the point cloud data of the preset simulation model to obtain simulation point cloud data; aligning the target point cloud data and the simulation point cloud data to obtain point cloud alignment results; mapping the joint component labels in the preset simulation model to the 3D representation data based on the point cloud alignment results to obtain component label allocation results; and generating component-level 3D assets based on the component label allocation results.

[0009] Optionally, training the initial keypoint detection model based on the synthetic binocular image and the ground truth of keypoints to obtain the target keypoint detection model includes: using the initial keypoint detection model to perform prediction processing on the synthetic binocular image to obtain two-dimensional predicted keypoints, wherein the two-dimensional predicted keypoints include: two-dimensional predicted keypoints for the left eye and two-dimensional predicted keypoints for the right eye; determining the target training loss based on the two-dimensional predicted keypoints and the ground truth of keypoints; and updating the model parameters of the initial keypoint detection model using the target training loss to obtain the target keypoint detection model.

[0010] Optionally, determining the target training loss based on the two-dimensional predicted keypoints and the ground truth of the keypoints includes: determining the two-dimensional supervision loss and the reprojection consistency loss based on the two-dimensional predicted keypoints and the ground truth of the keypoints; determining the left and right eye consistency loss based on the geometric constraint relationship between the left and right eye two-dimensional predicted keypoints; and performing a weighted operation on the two-dimensional supervision loss, the reprojection consistency loss, and the left and right eye consistency loss to obtain the target training loss.

[0011] Optionally, determining the reprojection consistency loss based on the two-dimensional predicted key points and the true values ​​of the key points includes: performing triangulation calculation on the two-dimensional predicted key points to obtain the three-dimensional predicted key points; reprojecting the three-dimensional predicted key points to obtain the reprojection result; and determining the reprojection consistency loss based on the reprojection result and the true values ​​of the key points.

[0012] Optionally, the 3D keypoint pose estimation method further includes randomizing at least one of the following during the rendering of the synthetic binocular image: illumination intensity, light source direction, background scene, and image noise.

[0013] Optionally, the 3D keypoint pose estimation method further includes: obtaining the reprojection error and detection confidence corresponding to the 3D keypoint coordinates; outputting the 3D keypoint coordinates in response to the reprojection error and detection confidence meeting the preset quality conditions; and downgrading the 3D keypoint coordinates in response to the reprojection error and detection confidence not meeting the preset quality conditions, and outputting a structured result with downgraded labels.

[0014] According to another aspect of the embodiments of this application, a robot is also provided, comprising: a torso; two upper limbs and two lower limbs respectively connected to the torso; a binocular camera disposed on the head of the torso; at least one processor; and at least one storage medium, wherein the at least one storage medium stores instructions, which, when executed by the at least one processor, cause the robot to: acquire left and right eye images through the binocular camera; input the left and right eye images into a target keypoint detection model respectively to obtain left-eye two-dimensional keypoints and right-eye two-dimensional keypoints, wherein the target keypoint detection model is obtained by training an initial keypoint detection model using multiple sets of training data and a target training loss, each set of training data including: a synthesized binocular image and the corresponding ground truth value of keypoints, and the target training loss is obtained by weighted calculation based on two-dimensional supervision loss, reprojection consistency loss and left-right eye consistency loss; and perform triangulation calculation based on the left-eye two-dimensional keypoints, the right-eye two-dimensional keypoints and the calibration parameters of the binocular camera to obtain the three-dimensional keypoint coordinates of the target object in the left-eye coordinate system, wherein the calibration parameters of the binocular camera include at least one of the following: focal length, baseline length and principal point coordinates.

[0015] According to another aspect of the embodiments of this application, a three-dimensional keypoint pose estimation device based on a neural network is provided, comprising: an acquisition module for acquiring left-eye and right-eye images through a binocular camera; a detection module for inputting the left-eye and right-eye images into a target keypoint detection model to obtain left-eye two-dimensional keypoints and right-eye two-dimensional keypoints, wherein the target keypoint detection model is obtained by training an initial keypoint detection model using multiple sets of training data and a target training loss, each set of training data including: a synthesized binocular image and the corresponding ground truth value of keypoints, and the target training loss is obtained by weighted calculation based on two-dimensional supervision loss, reprojection consistency loss and left-right consistency loss; and a calculation module for performing triangulation calculation based on the left-eye two-dimensional keypoints, the right-eye two-dimensional keypoints and the calibration parameters of the binocular camera to obtain the three-dimensional keypoint coordinates of the target object in the left-eye coordinate system, wherein the calibration parameters of the binocular camera include at least one of the following: focal length, baseline length and principal point coordinates.

[0016] Optionally, the 3D keypoint pose estimation device further includes: a reconstruction module for sparse reconstruction of multi-view images of the target object to obtain sparse 3D points; an estimation module for pixel-level depth estimation of multi-view images to obtain pixel-level depth; a calibration module for scale offset calibration of the pixel-level depth based on the projected depth of the sparse 3D points to obtain a calibration depth; an alignment module for determining the 3D representation data of the target object based on the calibration depth, and registering and aligning the 3D representation data of the target object with a preset simulation model to obtain a component-level 3D asset; a rendering module for updating the joint state of the component-level 3D asset through a simulation engine, and rendering and generating a synthetic stereo image and corresponding keypoint ground truth; and a training module for training an initial keypoint detection model based on the synthetic stereo image and keypoint ground truth to obtain a target keypoint detection model.

[0017] Optionally, the calibration module is also used to: construct a depth calibration optimization target based on the projected depth, pixel-level depth, and geometric constraint weights of sparse 3D points, wherein the depth calibration optimization target is used to represent the minimum value of the weighted sum of squared residuals; solve the depth calibration optimization target to obtain the target scale coefficient and target offset; and use the target scale coefficient and target offset to perform a linear transformation on the pixel-level depth of all pixels in the image to obtain the calibration depth.

[0018] Optionally, the alignment module is also used to: perform downsampling processing on the 3D representation data to obtain target point cloud data, and perform downsampling processing on the point cloud data of the preset simulation model to obtain simulation point cloud data; perform alignment processing on the target point cloud data and the simulation point cloud data to obtain point cloud alignment results; based on the point cloud alignment results, map the joint component labels in the preset simulation model to the 3D representation data to obtain component label allocation results; and generate component-level 3D assets based on the component label allocation results.

[0019] Optionally, the training module is further configured to: perform prediction processing on the synthetic binocular image using the initial keypoint detection model to obtain two-dimensional predicted keypoints, wherein the two-dimensional predicted keypoints include: two-dimensional predicted keypoints for the left eye and two-dimensional predicted keypoints for the right eye; determine the target training loss based on the two-dimensional predicted keypoints and the ground truth values ​​of the keypoints; and update the model parameters of the initial keypoint detection model using the target training loss to obtain the target keypoint detection model.

[0020] Optionally, the training module is also used to: determine the two-dimensional supervision loss and reprojection consistency loss based on the two-dimensional predicted keypoints and the ground truth of the keypoints; determine the left and right eye consistency loss based on the geometric constraint relationship between the left and right eye two-dimensional predicted keypoints; and perform a weighted operation on the two-dimensional supervision loss, reprojection consistency loss and left and right eye consistency loss to obtain the target training loss.

[0021] Optionally, the training module is also used to: perform triangulation calculation on the two-dimensional predicted key points to obtain the three-dimensional predicted key points; perform reprojection on the three-dimensional predicted key points to obtain the reprojection result; and determine the reprojection consistency loss based on the reprojection result and the ground truth value of the key points.

[0022] Optionally, the 3D keypoint pose estimation device further includes a processing module for randomizing at least one of illumination intensity, light source direction, background scene, and image noise during the rendering of a synthetic binocular image.

[0023] Optionally, the 3D keypoint pose estimation device further includes: an acquisition module for acquiring the reprojection error and detection confidence corresponding to the 3D keypoint coordinates; a first output module for outputting the 3D keypoint coordinates in response to the reprojection error and detection confidence meeting a preset quality condition; and a second output module for downgrading the 3D keypoint coordinates in response to the reprojection error and detection confidence not meeting the preset quality condition, and outputting a structured result with downgrade markers.

[0024] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0025] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0026] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0027] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0028] In this embodiment, by using a binocular camera to acquire left and right eye images, inputting the target key point detection model trained with synthetic data to obtain two-dimensional key points, and combining calibration parameters for triangulation calculation, the head-mounted binocular camera can output the three-dimensional key point pose in the left eye coordinate system without relying on external motion capture equipment, multi-view camera arrays, and special markers. This avoids high hardware costs and complex deployment limitations, realizes the combination of realistic appearance and simulated structure, improves simulation transferability, and solves the technical problems of complex hardware deployment and poor simulation transferability in related technologies when performing three-dimensional key point pose estimation. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 This is a schematic diagram of an optional neural network-based 3D keypoint pose estimation system according to an embodiment of this application;

[0031] Figure 2 This is a flowchart of an optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application;

[0032] Figure 3 This is a schematic diagram of an optional robot according to an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of an optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application;

[0034] Figure 5This is a schematic diagram of another optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of another optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of another optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application;

[0037] Figure 8 This is a structural block diagram of an optional neural network-based 3D keypoint pose estimation device according to an embodiment of this application. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] 3D keypoint pose estimation for humanoid robots is a crucial technological foundation for achieving complex motion control, environmental perception, and safe interaction. In related technological systems, obtaining high-precision 3D pose values ​​typically relies heavily on external auxiliary facilities, such as optical motion capture systems, inertial measurement unit (IMU) kits, external depth camera arrays, or multi-camera calibration fields. While these solutions provide accurate reference data, they generally suffer from drawbacks such as high hardware costs, complex on-site deployment, and stringent environmental requirements, making them difficult to directly transfer to real-world scenarios for standalone humanoid robots equipped only with dual-lens head-mounted optical cameras.

[0041] On the other hand, at the algorithm training level, to reduce data acquisition costs, related technologies often employ simplified mesh simulations or fixed background rendering to generate training data. However, this simulated environment differs significantly from the visual appearance of the real physical world, leading to a severe "simulation-to-reality" domain offset problem. Especially when training neural rendering techniques such as 3D Gaussian Splatting using only red, green, blue (RGB) images, the model often struggles to handle weakly textured regions and highly reflective surfaces, resulting in geometric drift, floating Gaussian points, and artifacts such as inconsistent depth scales. These problems directly lead to low-quality generated joint assets and insufficient accuracy of synthetic annotations, thus limiting the generalization ability and robustness of keypoint detection networks in real-world complex scenes.

[0042] According to an embodiment of this application, a method embodiment for three-dimensional keypoint pose estimation based on neural networks is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] Figure 1 This is a schematic diagram of an optional neural network-based 3D keypoint pose estimation system according to an embodiment of this application, as shown below. Figure 1 As shown, the system comprises four modules: a depth-constrained reconstruction module, a component asset and synthetic data module, a binocular keypoint estimation module, and a deployment output and quality gating module. The system begins with the depth-constrained reconstruction module, which receives multi-view RGB images of the real scene, camera poses and sparse 3D points generated by COLMAP, and a monocular depth map. It calibrates the monocular depth through scale-offset and applies geometric constraints to the 3D Gaussian rendering depth using the calibrated depth, outputting a high-fidelity 3D Gaussian point cloud and camera parameters. COLMAP (Structure from Motion, Multi-View Stereo and Parameter Estimation) is an open-source, general-purpose photogrammetry software package primarily used for reconstructing 3D scene structures from unordered image sets.

[0044] Subsequently, the data flows into the component asset and synthetic data module. This module registers and aligns the output Gaussian point cloud with the simulation model point cloud with joint labels, propagates the simulation joint labels to Gaussian points to construct component-level 3D Gaussian assets, and uses a physical simulation engine to drive changes in the joint state of the assets, rendering and generating synthetic binocular images and corresponding ground truth values ​​of key points, thereby constructing a high-quality synthetic training dataset.

[0045] Next, the synthetic data enters the binocular keypoint estimation module. During the training phase, this module uses the synthetic data to optimize the network parameters. During the deployment phase, it receives the left and right RGB images captured by the binocular camera on the front of the real device, uses the frozen model parameters to perform forward inference to obtain two-dimensional keypoints, and outputs the three-dimensional keypoint coordinates and confidence scores in the left-eye coordinate system through triangulation.

[0046] Finally, the deployment output and quality gating module receives 3D key points, reprojection errors, and synchronization quality indicators. Based on the preset quality threshold, it judges the availability of data. When normal, it publishes structured results with timestamps and coordinate systems. When abnormal, it executes a degradation strategy, ultimately providing a reliable attitude reference for downstream punch control, scoring, or safety modules.

[0047] In an optional embodiment, this application divides the overall process of the neural network-based 3D keypoint pose estimation method into two timelines: an offline training phase and an online deployment phase. The offline training phase encompasses multi-view image acquisition, sparse reconstruction, depth calibration, 3D Gaussian training, registration and label propagation, physics engine-driven rendering, and keypoint network training, after which the model parameters are frozen. The online deployment phase only performs head-mounted binocular synchronous acquisition, network forward detection, and triangulation calculation; sparse reconstruction or updating of 3D assets is no longer performed. This division keeps the computational load and hardware requirements on the deployment side within the limits of the binocular camera and onboard processor.

[0048] Figure 2 This is a flowchart of an optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0049] Step S201: Acquire left and right eye images using a binocular camera;

[0050] Step S202: Input the left and right eye images into the target keypoint detection model to obtain the left eye 2D keypoints and the right eye 2D keypoints. The target keypoint detection model is obtained by training the initial keypoint detection model using multiple sets of training data and target training loss. Each set of training data includes: the synthesized binocular image and the corresponding ground truth keypoints. The target training loss is obtained by weighting the 2D supervision loss, reprojection consistency loss and left and right eye consistency loss.

[0051] Step S203: Triangulation calculation is performed based on the left-eye 2D key points, the right-eye 2D key points, and the calibration parameters of the binocular camera to obtain the 3D key point coordinates of the target object in the left-eye coordinate system. The calibration parameters of the binocular camera include at least one of the following: focal length, baseline length, and principal point coordinates.

[0052] A binocular camera is a visual sensor component mounted on the head of a humanoid robot to simultaneously capture the left and right views of a scene. The binocular camera is mounted on the torso head, and the baseline length between the centers of the left and right eyes is calibrated and stored at the factory. The intrinsic parameter matrix and distortion coefficients are also pre-calibrated. During deployment, the system reads the calibration parameters from the storage medium for triangulation calculations. Synchronous acquisition by the binocular camera is ensured by hardware triggering or software timestamp alignment; subsequent detection and triangulation can only proceed if the synchronization error is within acceptable limits.

[0053] The left-eye image can be two-dimensional image data acquired by the left lens of the stereo camera. The right-eye image can be two-dimensional image data acquired by the right lens of the stereo camera.

[0054] During the online deployment phase, the binocular cameras on the humanoid robot's head simultaneously acquire the left and right RGB images of the scene at the current moment. This step obtains the raw visual input data for subsequent pose estimation, and this acquisition process does not rely on external multi-view camera arrays or motion capture equipment, but is completed solely by the binocular sensors built into the head.

[0055] After acquiring the left and right RGB images, the two images are forward-propagated through a target keypoint detection model with frozen parameters, outputting the 2D pixel coordinates and detection confidence of each keypoint. The target keypoint detection model is obtained during offline training by using synthetic binocular images generated based on real-world Gaussian assets and simulated joint labels, along with their corresponding ground truth keypoint values, as supervision signals to optimize the initial keypoint detection model through backpropagation. Furthermore, by minimizing the target training loss, which is a combination of 2D supervision loss, reprojection consistency loss, and left / right eye consistency loss, the model learns the ability to accurately detect 2D keypoints in real-world scenes.

[0056] The initial keypoint detection model can be a neural network infrastructure that has not been trained on specific synthetic data or has only undergone basic pre-training. The synthetic stereo images can be paired left and right eye image data generated by rendering simulated joint states using a physics simulation engine-driven component-level 3D Gaussian assets. The ground truth keypoints can be accurate 3D or 2D keypoint coordinate data provided by the simulation model, corresponding to the synthetic stereo images.

[0057] Subsequently, triangulation calculations were performed on the corresponding keypoints of the left and right eyes using the pre-calibrated offline binocular camera calibration parameters. The 3D position of each keypoint in space was calculated using geometric triangulation principles. The calculation results were uniformly transformed to the coordinate system of the left eye camera, and the output was the 3D keypoint coordinates in the left eye coordinate system. The calibration parameters can be a set of parameters describing the internal optical characteristics of the binocular camera and the relative positional relationship between the left and right cameras, including focal length, principal point coordinates, baseline length, and rotation and translation transformation matrices between the left and right cameras. The left eye coordinate system can be a 3D Cartesian coordinate system established with the optical center of the left eye camera as the origin, used to describe the spatial position of the 3D keypoints in this coordinate system.

[0058] The formula for triangulation is:

[0059]

[0060]

[0061]

[0062] Where (X, Y, Z) are the three-dimensional coordinates of the key point in the left eye camera coordinate system. B is the focal length of the left eye camera, in pixels, and B is the binocular baseline length. , () represents the pixel coordinates of the left eye's keypoint. The horizontal pixel coordinates of the key point corresponding to the right eye are ( , () represents the coordinates of the principal point of the left eye camera.

[0063] The above triangulation formula applies to cases where the binocular image has already undergone epipolar correction. If the binocular image is uncorrected or the correction residual is large, a general triangulation form is used: construct a line-of-sight ray originating from the camera's optical center for each keypoint of the left and right eyes, and find the nearest intersection point of the two rays in the least squares sense. In this case, the intrinsic parameter matrices of the left and right eyes and the extrinsic parameter transformation of the right eye relative to the left eye are required. The calibration parameters are determined offline and frozen before deployment and are not updated during operation.

[0064] Based on the above steps S201 to S203, this embodiment of the application uses a binocular camera to acquire left and right eye images, inputs the target key point detection model trained with synthetic data to obtain two-dimensional key points, and performs triangulation calculations in combination with calibration parameters. It can output the three-dimensional key point pose in the left eye coordinate system using a head-mounted binocular camera without relying on external motion capture equipment, multi-view camera arrays, and special markers. This avoids high hardware costs and complex deployment limitations, realizes the combination of realistic appearance and simulated structure, improves simulation transferability, and solves the technical problems of complex hardware deployment and poor simulation transferability in related technologies when performing three-dimensional key point pose estimation.

[0065] Optionally, the 3D keypoint pose estimation method further includes: sparse reconstruction of multi-view images of the target object to obtain sparse 3D points; pixel-level depth estimation of the multi-view images to obtain pixel-level depth; scale offset calibration of the pixel-level depth based on the projected depth of the sparse 3D points to obtain a calibration depth; determining the 3D representation data of the target object based on the calibration depth, and registering and aligning the 3D representation data of the target object with a preset simulation model to obtain a component-level 3D asset; updating the joint state of the component-level 3D asset through a simulation engine, rendering and generating a synthetic stereo image and the corresponding ground truth of keypoints; and training an initial keypoint detection model based on the synthetic stereo image and the ground truth of keypoints to obtain a target keypoint detection model.

[0066] During the offline phase, when training the target keypoint detection model, a humanoid target, or a similar-shaped object, is first placed in the acquisition scene. Multi-view RGB images covering various angles of the target are then acquired through surround shooting or simultaneous shooting by multiple cameras. The acquisition environment is not limited to a specific location; a turntable surround shooting, handheld camera surround shooting, or fixed multi-camera arrays are all applicable.

[0067] Sparse reconstruction is the process of recovering camera pose and sparse 3D structure of a scene by utilizing geometric constraints between multi-view images and employing feature matching and bundle adjustment algorithms. Sparse 3D points are point cloud data with 3D spatial coordinates but sparsely distributed, generated during the sparse reconstruction process, typically corresponding to salient feature points in the scene.

[0068] After acquiring multi-view images, the image set is sparsely reconstructed using the Structure from Motion (SfM) algorithm. This process extracts feature points from each image and performs cross-view matching. Bundle adjustment is then used for joint optimization to obtain the camera extrinsic parameters (pose) for each image, as well as a set of sparse 3D points in the scene that can be observed by multiple images. The number of sparse 3D points is typically related to the scene texture richness and the number of images, and is distributed in areas of the target surface with relatively obvious texture. Although these sparse points cannot cover the entire target surface, their 3D coordinates are geometrically constrained by multiple views, making them reliable in scale and suitable as absolute scale anchors for subsequent depth calibration.

[0069] Pixel-level depth estimation is the process of predicting the distance from each pixel in an image to the camera's optical center using deep learning networks or geometric methods, thereby generating a dense depth map. Pixel-level depth can be a set of depth values ​​covering all pixels in an image, obtained from pixel-level depth estimation.

[0070] In parallel or sequential execution with sparse reconstruction, monocular pixel-level depth estimation is performed on each of the multi-view images. The monocular depth estimation network outputs a depth prediction value for each pixel in the image, resulting in a pixel-level depth map with the same resolution as the input image. However, the output of monocular depth networks typically suffers from global scale uncertainty and bias; depth prediction values ​​across different images may not be on a uniform metric scale, and the absolute values ​​systematically deviate from the true scene depth. This problem prevents monocular depth maps from being directly used as geometric constraints for subsequent 3D Gaussian training.

[0071] Therefore, it is necessary to perform scale offset calibration on the pixel-level depth using the projected depth of sparse 3D points as a reference to obtain the calibrated depth. Scale offset calibration can be a process of adjusting the scale and zero point of the monocular depth map through linear transformation to numerically align it with the reference depth. Projected depth can be the theoretical depth value calculated by projecting sparse 3D points onto the image plane using camera intrinsic parameters. Calibrated depth can be the pixel-level depth value that, after scale offset calibration, maintains consistency with the projected depth of the sparse 3D points.

[0072] During offset calibration, sparse 3D points are projected onto their corresponding image pixels, and the projected depth is calculated. By solving the depth calibration optimization objective equation, the target scale coefficient and offset at the pixel level are determined, and these are converted into a calibration depth consistent with the scale of the sparse 3D points. This step solves the problem of the lack of absolute scale in monocular depth, providing geometric prior constraints for 3D reconstruction.

[0073] The rendering depth of the Gaussian-based 3D representation is constrained by a calibration depth within a reliable region, and a 3D representation of the target is obtained by training with a joint color reconstruction loss. The 3D representation data can be a set of high-fidelity point clouds representing the geometry and appearance of the target object, obtained by 3D Gaussian sputtering reconstruction; in this embodiment, it specifically refers to the 3D Gaussian representation based on Gaussian elements.

[0074] The Gaussian primitive-based 3D representation models the scene as a set of 3D Gaussian primitives, each with learnable parameters such as position (mean), covariance (controlling shape and orientation), opacity, and color (spherical harmonic coefficients). During training, images are rendered from known camera poses, the rendered results are compared with real images to calculate the color reconstruction loss, and the Gaussian primitive parameters are updated and primitives are added or removed through gradient backpropagation.

[0075] In this embodiment, in addition to the color reconstruction loss, the calibration depth of the above output is incorporated into the training as a geometric constraint. The overall form of the 3D Gaussian joint loss function for color and depth is:

[0076]

[0077] in, To mitigate the color reconstruction loss between RGB and real images, a weighted combination of L1 loss and structural similarity loss can be used. The depth weight coefficients change dynamically with the number of training iterations t. This is the rendering depth map of the current Gaussian set from the corresponding viewpoint. The depth map is offset-calibrated and obtained by monocular depth calibration using COLMAP sparse points. W is a reliable region mask, which calculates depth error only in regions with high confidence, such as the neighborhood of COLMAP sparse points, to avoid interference from weak textures or reflective areas in geometric reconstruction.

[0078] In an optional embodiment, the generation logic of the reliable region mask W is as follows: For pixel p, if there is at least one sparse projection point within its preset neighborhood and the reliable weight of that point is... If the preset conditions are met, W(p) is set to 1; otherwise, W(p) is set to 0. The size of the neighborhood range is related to the image resolution and the sparse point projection density at the current viewpoint. The denser the sparse points, the smaller the neighborhood value can be; conversely, the smaller the sparse points, the larger the neighborhood value can be to cover more pixels. If the monocular depth network also outputs uncertainty estimates, pixels with excessively high uncertainty are also set to W(p) = 0. The above strategy ensures that depth constraints only apply to regions supported by geometric anchors, avoiding the unreliable calibration depth from pulling Gaussian cells to incorrect positions in weakly textured regions.

[0079] Depth weighting coefficient A strategy of decreasing the frequency as training progresses is adopted. In one embodiment of this application, the decreasing method is linear decay, and the calculation formula is as follows:

[0080]

[0081] in, As the initial depth weight, This represents the number of preheating iterations.

[0082] In the early stages of training, i.e., t < Initially, the larger depth weights force Gaussian primitives to converge quickly to spatial positions consistent with the calibration depth, establishing a roughly correct geometric structure. As iterations progress, the depth weights gradually decrease until they reach zero, allowing the color reconstruction loss to take over the lead in later optimization. The network can then freely adjust the fine-grained positions and appearance attributes of the primitives to fit texture details. This "stabilize structure first, then refine appearance" scheduling strategy avoids interference from residual local noise in the calibration depth during later training stages, preventing interference with primitive position fine-tuning. and The specific value is determined by the reconstruction quality performance on the validation set, and different configurations can be used for different targets and scenarios. In a different embodiment, the weights can also adopt cosine decay or step decay, as long as the overall trend of high weights in the early stage and low or zero weights in the later stage is met.

[0083] After training and convergence, a high-fidelity 3D representation of the target is obtained, which is a set of Gaussian primitives with optimized spatial location and appearance attributes, along with corresponding camera parameters. Due to the involvement of depth constraints, the geometric accuracy of this 3D representation in weakly textured regions is significantly improved compared to pure color supervision, and the phenomenon of inconsistency between floating Gaussian points and depth scales is suppressed. The aforementioned improvement in geometric quality is a prerequisite for the accuracy of label propagation in subsequent steps. If the surface geometry of the 3D representation itself has significant distortion, systematic label misalignment will occur when propagating simulated joint labels to Gaussian points.

[0084] After acquiring the 3D representation data of the target object, the 3D representation data of the target object is registered and aligned with a preset simulation model to obtain a component-level 3D asset. The component-level 3D asset can be a high-fidelity 3D set where the 3D representation data is divided into components and labeled with their respective joints or components. The preset simulation model can be a pre-constructed computer simulation robot model containing a joint hierarchy and initial pose. Registration and alignment can be the process of calculating and applying rigid body transformation matrices to ensure that the component-level 3D asset and the preset simulation model are precisely aligned in space. In another optional embodiment, robust registration or a learning-based registration algorithm can also be used.

[0085] During registration and alignment, the 3D Gaussian point cloud and the preset simulation model point cloud are downsampled, coarsely aligned, and finely aligned, and the rigid body transformation matrix is ​​calculated to make them coincide. For example, coarse alignment can be performed using Principal Component Analysis (PCA) for principal direction alignment, and fine alignment can be performed using the Iterative Closest Point (ICP) algorithm. Subsequently, the joint labels of the simulation model are propagated to the Gaussian point cloud through nearest neighbor search or graph cut algorithms to form component-level 3D assets divided by joints. This step assigns the semantic labels of the simulation structure to the geometric data of the realistic appearance.

[0086] A simulation engine is a software system used to simulate the laws of motion in the physical world, update joint states, and render images. Joint states can be the angle or position parameters of each joint in the simulation model at the current moment. After acquiring component-level 3D assets, the simulation engine updates the joint states of the component-level 3D assets, and renders and generates synthetic stereo images and corresponding keypoint ground truth values.

[0087] In the component-level 3D asset, each Gaussian element is bound to a rigid component belonging to a specific joint. When the physics simulation engine changes the angle of a joint, all Gaussian points belonging to the downstream components of that joint rotate and translate accordingly according to rigid body transformation. This process is consistent with the kinematic relationship of the corresponding link in the simulation model. By sampling the combined angles of each joint, snapshots of the 3D Gaussian asset of the target in different poses can be generated.

[0088] The rendering process is performed using a virtual stereo camera configuration. The intrinsic parameters (focal length, principal point) and extrinsic parameters (baseline length, relative pose of the two eyes) of the virtual stereo camera are referenced from the calibration parameters of the real head-mounted stereo camera. Small random perturbations can be applied near the calibration parameters to increase the diversity of the training data. For each set of joint states, differentiable rendering is performed on the Gaussian asset of the current pose from both the left and right eye perspectives, outputting a pair of left-eye RGB images and a right-eye RGB image. Simultaneously, since the 3D coordinates of each keypoint in the simulation model can be directly calculated from the joint angles using forward kinematics, projecting these 3D coordinates onto the left and right eye image planes respectively yields the 2D keypoint ground truth. The 3D coordinates themselves are the 3D keypoint ground truth.

[0089] The rendered synthetic dataset contains: paired left and right eye RGB images, the two-dimensional pixel coordinates of each keypoint in each image, the three-dimensional coordinates of the keypoints in the left eye coordinate system, and the corresponding camera parameters. This dataset is directly used to train the target keypoint detection model in subsequent steps.

[0090] After rendering and generating synthetic stereo images and corresponding ground truth keypoints, the initial keypoint detection model is trained based on the synthetic stereo images and ground truth keypoints to obtain the target keypoint detection model. For example, the synthetic stereo images are used as input, and the ground truth keypoints are used as supervision signals to perform backpropagation training on the initial keypoint detection model, optimizing the network parameters until the model converges, ultimately obtaining the target keypoint detection model. This step completes the construction of the model's generalization ability from simulation to real-world scenes.

[0091] Based on the above optional embodiments, this application embodiment utilizes sparse reconstruction to calibrate monocular depth to constrain 3D Gaussian reconstruction, propagates simulated joint labels to Gaussian assets with realistic appearance, and uses a physical simulation engine to render synthetic binocular images and ground truth key points to train the detection model. This enables the generation of high-fidelity synthetic data with accurate 3D labels without relying on motion capture equipment and markers, thereby training a key point detection network that can adapt to real-world scenes. This effectively reduces the gap between simulation and reality, realizes a closed-loop process from offline asset creation to online deployment, and provides a high-quality data foundation for subsequent 3D pose estimation that relies solely on head binoculars.

[0092] Optionally, using the projected depth of sparse 3D points as a reference, scale offset calibration is performed on the pixel-level depth to obtain the calibration depth. This includes: constructing a depth calibration optimization objective based on the projected depth of sparse 3D points, the pixel-level depth, and geometric constraint weights, where the depth calibration optimization objective represents the minimum value of the weighted residual sum of squares; solving the depth calibration optimization objective to obtain the target scale coefficient and the target offset; and using the target scale coefficient and the target offset, performing a linear transformation on the pixel-level depth of all pixels in the image to obtain the calibration depth.

[0093] The depth calibration optimization objective can be a mathematical optimization function used to solve for the scale coefficient and offset. This function is expressed as the minimum of the weighted sum of squared residuals, aiming to minimize the error between the calibrated depth and the reference depth. Geometric constraint weights can be weighting coefficients used to measure the reliability of the projected depth of sparse 3D points. These coefficients are determined based on the reprojection error or local point density of sparse points and are used to weight data points with different levels of reliability during the optimization process.

[0094] The target scale coefficient can be a linear transformation parameter obtained through optimization, used to adjust the pixel-level depth scale. The target offset can be a linear transformation parameter obtained through optimization, used to adjust the pixel-level depth zero point.

[0095] During the offline asset creation stage, for each multi-view image, the 3D points obtained from sparse reconstruction are projected onto the image plane according to the camera pose corresponding to that image. The depth value of each sparse point in the camera coordinate system is calculated, which is called the sparse projection depth. The sparse projection depth is then paired with the pixel-level depth predicted by the monocular depth network at the same pixel position to establish a calibration relationship.

[0096] The calibration process employs weighted least squares fitting, calculating a set of target scale coefficients *a* and target offsets *b* for each image. This ensures that the calibrated depth closely matches the sparse projected depth in regions supported by sparse points. Specifically, the formula for calculating the depth calibration optimization target is:

[0097]

[0098] in, This represents the monocular depth prediction value at pixel p, i.e., the pixel-level depth. The projection depth is the depth value projected onto the corresponding sparse 3D point. The reliable weight of a sparse point can be determined by its reprojection error or its local point density in three-dimensional space; the smaller the reprojection error or the higher the local point density, the greater the weight. This is the set of sparse points that participate in the calibration.

[0099] After obtaining the target scale coefficient 'a' and the target offset 'b', a linear transformation is applied to all pixels of the image. The calculation formula is as follows: This results in a globally consistent calibration depth map. This is the calibrated depth.

[0100] The essence of calibration is to map the output of a monocular depth network from an unknown scale space to a metric space determined by sparse reconstruction. Since sparse points are only distributed across a subset of pixel locations, the depth of all pixels after calibration, while highly accurate in areas with anchor points, may still exhibit local biases in regions far from anchor points, such as large areas of weakly textured planes. Therefore, the calibrated depth is not directly used as a hard label, but rather participates in training in subsequent steps as a weighted soft constraint.

[0101] Based on the above optional embodiments, this application embodiment constructs a weighted residual sum of squares minimization optimization objective based on geometric constraint weights, using the projection depth of sparse 3D points as a reference, solves for the target scale coefficient and target offset, and performs linear transformation on the pixel-level depth of the entire image to obtain the calibration depth. This can effectively align the monocular depth estimation results with the geometric reference of sparse reconstruction, solve the problems of monocular depth lacking absolute scale and having offset, provide accurate geometric priors for subsequent 3D Gaussian reconstruction, thereby constraining the geometric structure of 3D representation data, improving the geometric accuracy of reconstructed assets, and avoiding geometric drift caused by weak texture regions.

[0102] Optionally, registering and aligning the 3D representation data of the target object with a preset simulation model to obtain component-level 3D assets includes: downsampling the 3D representation data to obtain target point cloud data, and downsampling the point cloud data of the preset simulation model to obtain simulation point cloud data; aligning the target point cloud data and the simulation point cloud data to obtain point cloud alignment results; mapping the joint component labels in the preset simulation model to the 3D representation data based on the point cloud alignment results to obtain component label allocation results; and generating component-level 3D assets based on the component label allocation results.

[0103] The Gaussian point cloud of the output 3D representation data carries the target's true appearance information but lacks semantic annotations at the joint level. The preset simulation model, on the other hand, has clear joint definitions and component divisions, with each point in its point cloud pre-labeled with its corresponding joint or component number. The goal of this step is to register and align the target object's 3D representation data with the preset simulation model, "transferring" the joint semantics of the simulation model to the Gaussian point cloud. This allows each Gaussian primitive to obtain a unique component affiliation label, thereby forming a component-level 3D asset that can be disassembled and driven by joints.

[0104] The target point cloud data can be a set of point clouds with low density and uniform distribution obtained by downsampling the 3D representation data. During the registration process, this data serves as the geometric representative of the real reconstructed asset, and its point cloud density is controlled to avoid the registration algorithm getting stuck in local optima or consuming excessive computational resources due to excessively high Gaussian point cloud density.

[0105] Simulated point cloud data can be a set of point clouds with low density and uniform distribution obtained by downsampling the point cloud data of a preset simulation model. During the registration process, this data serves as a representative of the geometric structure of the virtual model, and its point cloud density is adjusted to a level similar to that of the target point cloud data. This ensures that the two point cloud data from different sources are statistically comparable in subsequent coarse and fine alignment steps, thereby eliminating the negative impact of density differences on registration accuracy.

[0106] Point cloud alignment results can be the transformation relationship or alignment state calculated by the registration algorithm that makes the target point cloud data and the simulated point cloud data coincide precisely in spatial position.

[0107] Joint component labels can be used as classification identifiers in a preset simulation model to identify various joints or body parts (such as the left arm, torso, etc.).

[0108] The component label assignment result can be the label distribution result formed by mapping the joint component labels in the preset simulation model to various points of the three-dimensional representation data.

[0109] For example, registration and alignment are divided into three stages: preprocessing, coarse alignment, and fine registration.

[0110] In the preprocessing stage, uniform voxel downsampling is performed on both sets of point clouds. The number of points in the Gaussian point cloud, i.e., the target point cloud, is often much larger than that in the simulation point cloud. The former can reach hundreds of thousands or even millions, while the latter is usually in the thousands to tens of thousands. Direct registration will introduce bias due to density differences. The uniform voxel scale is selected based on the premise that the number of points in the two sets of point clouds is on the same order of magnitude after downsampling, while filtering out outliers in the space that are significantly deviated from the subject.

[0111] In the coarse alignment stage, principal component analysis (PCA) is used to extract the principal directions of the two sets of point clouds. For a humanoid target, the first principal direction corresponds to the long axis of the torso (vertical direction), and the second principal direction corresponds to the shoulder width (horizontal direction). Aligning the principal directions of the two sets of point clouds yields the initial rigid body transformation matrix. If the geometric difference between the target's appearance and the simulation model is significant, making the principal directions unreliable, an alternative embodiment can employ a coarse registration method based on local geometric feature descriptors, or manually specify a small number of corresponding points to provide the initial transformation. Alternatively, coarse registration can be performed using Fast Point Feature Histograms (FPFH) combined with Random Sample Consensus (RANSAC).

[0112] In the fine registration stage, the coarse alignment result is used as the initial value, and the Iterative Nearest Point (ICP) algorithm is run from point to surface. In each iteration, this algorithm finds the nearest neighbor surface in the target point cloud for each point in the source point cloud, minimizing the sum of the point-to-surface distances to update the rigid body transformation parameters. To suppress the influence of a small number of incorrectly corresponding points on the results during registration, a robust kernel function can be introduced to truncate large residual terms. The iteration terminates when the transformation increment between two adjacent iterations is less than a preset accuracy threshold or when the maximum number of iterations is reached. After registration, the deviation between the shoulder width and torso length and the nominal values ​​of the simulation model is checked. If the deviation exceeds the allowable range, registration is considered a failure, subsequent label propagation is rejected, and alternative solutions, such as mesh projection or manual annotation, are switched.

[0113] After successful registration, label propagation begins. For each Gaussian point in the Gaussian point cloud, the nearest neighbor search is used in the transformed and aligned simulated point cloud to find the point with the closest spatial distance. The joint / component label carried by this simulated point is then assigned to the target Gaussian point. In another optional embodiment, graph cut algorithm or mesh projection can also be used for label propagation. The aforementioned nearest neighbor propagation can usually provide clear label assignments in the mid-limb region, but at joint junctions, such as the transition area between the upper arm and forearm near the elbow joint, label ambiguity may occur.

[0114] For label conflicts in the boundary region, the processing logic is as follows: For each Gaussian point, record the labels of the two nearest different parts and their corresponding distance values ​​d1 and d2. If d2-d1 is less than the preset boundary bandwidth, which is on the same order of magnitude as the voxel scale used for registration, then the point is marked as an ambiguous point. A k-nearest neighbor graph structure is built for all ambiguous points, and graph model energy minimization (graph cut) is performed on this graph. The unary term (data term) is the distance from the point to each candidate part, and the binary term (smoothing term) penalizes the inconsistency of labels between spatially adjacent points. The result of energy minimization assigns a unique label to each ambiguous point, making the part boundaries spatially smooth and continuous. For extrema such as hands and heads, a higher preservation prior is set in the graph cut to avoid being swallowed by adjacent limbs.

[0115] Following the above process, each primitive in the Gaussian point cloud carries a unique part label. This set of labeled Gaussian primitives constitutes the part-level 3D asset, whose appearance is derived from the reconstruction results of real multi-view images, and whose joint semantics are derived from the known structural definitions of the simulation model.

[0116] Based on the above optional embodiments, this application embodiment performs downsampling processing on the point cloud data of the three-dimensional representation data and the preset simulation model respectively, obtains the point cloud alignment result by using alignment processing, maps the joint component labels in the preset simulation model to the three-dimensional representation data to obtain the component label allocation result, and generates component-level three-dimensional assets according to the component label allocation result. This can effectively transfer the joint semantic information of the simulation model to the high-fidelity geometric data of the real appearance, solve the problem of lack of joint semantics in simple geometric reconstruction, and enable the subsequent physics engine to drive and render based on the clear joint components, thereby generating synthetic data with accurate three-dimensional labels.

[0117] Optionally, training the initial keypoint detection model based on the synthetic binocular image and the ground truth of keypoints to obtain the target keypoint detection model includes: using the initial keypoint detection model to perform prediction processing on the synthetic binocular image to obtain two-dimensional predicted keypoints, wherein the two-dimensional predicted keypoints include: two-dimensional predicted keypoints for the left eye and two-dimensional predicted keypoints for the right eye; determining the target training loss based on the two-dimensional predicted keypoints and the ground truth of keypoints; and updating the model parameters of the initial keypoint detection model using the target training loss to obtain the target keypoint detection model.

[0118] Two-dimensional predicted keypoints can be the set of keypoint coordinates on the image plane output by the initial keypoint detection model after performing forward inference on the synthetic stereo image. Left-eye two-dimensional predicted keypoints can be the image plane keypoint coordinates predicted by the model for the left-eye image in the synthetic stereo image. Right-eye two-dimensional predicted keypoints can be the image plane keypoint coordinates predicted by the model for the right-eye image in the synthetic stereo image.

[0119] The target training loss can be a scalar value calculated based on the difference between the two-dimensional predicted keypoints and the ground truth keypoints. This value quantifies the error between the model's prediction and the true labels and is used to guide model optimization.

[0120] Training is conducted offline. The keypoint detection model receives a single RGB image as input and outputs the two-dimensional pixel coordinates and corresponding confidence scores of each predefined keypoint in the image. The backbone network of the model can be a Convolutional Neural Network (CNN) or a Visual Transformer. The backbone is responsible for extracting multi-scale feature representations from the input image, and the keypoint prediction head regresses on the feature map output by the backbone or decodes it through a heatmap to obtain the sub-pixel positions of each keypoint. During training, the synthesized left and right images are independently fed into the same network (sharing weights) to obtain the corresponding predicted keypoint coordinates.

[0121] The target training loss is determined based on the ground truth values ​​of 2D predicted keypoints. For example, the error between the left-eye 2D predicted keypoint and its corresponding ground truth value, and the error between the right-eye 2D predicted keypoint and its corresponding ground truth value, are calculated. Simultaneously, combining reprojection consistency and left-right eye consistency constraints, a total loss function is constructed, comprising 2D supervision loss, reprojection loss, and consistency loss. This step quantifies the deviation between the current model's prediction results and the true labels and geometric constraints, providing gradient direction for target optimization.

[0122] After obtaining the target training loss, the model parameters of the initial keypoint detection model are updated using the target training loss to obtain the target keypoint detection model. For example, a backpropagation algorithm is performed on the target training loss to calculate the gradient of the loss with respect to the model parameters. Using optimization algorithms, such as stochastic gradient descent or Adaptive Moment Estimation (Adam), the weights and biases of the initial keypoint detection model are updated based on the gradient. The above forward prediction, loss calculation, and parameter update process is repeated until the model converges, finally obtaining the target keypoint detection model. This step enables the model to learn the mapping relationship from real or synthetic images to accurate keypoints.

[0123] After training is complete, the network parameters are frozen, and the model is deployed to the onboard processor of the humanoid robot.

[0124] Based on the above optional embodiments, this application embodiment obtains left-eye 2D predicted key points and right-eye 2D predicted key points by using an initial key point detection model to predict synthetic binocular images. Based on the 2D predicted key points and the ground truth key points, the target training loss is determined, and the model parameters of the initial key point detection model are updated using the target training loss to obtain the target key point detection model. This can effectively utilize synthetic data with precise geometric constraints to supervise the training of the detection model, enabling the model to learn key point detection capabilities with consistency between the left and right eyes and reprojection consistency. This significantly reduces the gap between simulation and reality, and improves the accuracy and robustness of the model's 3D pose estimation in real scenes.

[0125] Optionally, determining the target training loss based on the two-dimensional predicted keypoints and the ground truth of the keypoints includes: determining the two-dimensional supervision loss and the reprojection consistency loss based on the two-dimensional predicted keypoints and the ground truth of the keypoints; determining the left and right eye consistency loss based on the geometric constraint relationship between the left and right eye two-dimensional predicted keypoints; and performing a weighted operation on the two-dimensional supervision loss, the reprojection consistency loss, and the left and right eye consistency loss to obtain the target training loss.

[0126] Two-dimensional supervised loss can be a numerical value that quantifies the difference between the predicted keypoints and the ground truth keypoints, and is used to constrain the prediction accuracy of the model on the image plane.

[0127] Reprojection consistency loss can be a numerical value that quantifies the difference between the reprojection result and the original keypoint truth value, and is used to constrain the geometric consistency between the 3D predicted keypoints and the 2D observation data.

[0128] The left-right eye consistency loss can be a loss value that quantifies the geometric constraint relationship between the key points of the left-eye 2D prediction and the key points of the right-eye 2D prediction, and is used to enhance the internal consistency of the binocular prediction results.

[0129] The formula for calculating the target training loss is:

[0130]

[0131] in, For two-dimensional supervision loss, smoothed L1 distance or mean square error can be used. For reprojection consistency loss, For loss of consistency between the left and right eyes, and These are the weighting coefficients for projection consistency loss and left / right eye consistency loss, used to balance the contributions of different loss terms, and configured based on the validation set performance.

[0132] When calculating the target training loss, the two-dimensional supervised loss is determined based on the two-dimensional predicted keypoints and their ground truth values. For example, the pixel-level error between the left-eye two-dimensional predicted keypoints and their ground truth values ​​is calculated, as well as the pixel-level error between the right-eye two-dimensional predicted keypoints and their ground truth values. These differences are typically calculated using metrics such as mean squared error or smoothed L1 norm, thus obtaining the two-dimensional supervised loss that quantifies the prediction bias. This step ensures that the model can accurately locate keypoints on the image plane.

[0133] The reprojection consistency loss is determined based on 2D predicted keypoints and their ground truth values. For example, the system uses the intrinsic and extrinsic parameters of the binocular camera to perform triangulation calculations on the 2D predicted keypoints of the left and right eyes, solving for the 3D predicted keypoints of the target in 3D space. Subsequently, these 3D predicted keypoints are mapped back to the image planes of the left and right eyes respectively using the camera projection model, generating corresponding reprojection results. Finally, the system calculates the pixel-level deviation between the reprojection results and the original ground truth keypoints, and constructs a reprojection consistency loss term by aggregating these deviations. This forces the 3D pose optimized by the network during training to remain consistent with the actual observation when projected back into the 2D image, ensuring the rationality of the geometric structure.

[0134] Based on the geometric constraints between the left and right 2D predicted keypoints, the left-right eye consistency loss is determined. For example, the correspondence between the left and right 2D predicted keypoints on the epipolar geometry is analyzed. This includes checking whether the right eye keypoint falls on the epipolar line defined by the left eye keypoint and camera extrinsic parameters, or calculating the mutual information or distance difference between the two in stereo matching. Based on the measurement results of the above geometric constraints, the left-right eye consistency loss is constructed. This step forces the model to maintain prediction consistency under binocular input, reducing left-right eye prediction conflicts caused by monocular noise.

[0135] Generally, it is desirable to minimize the deviation of corresponding key points in the epipolar direction between the left and right eyes. For a calibrated binocular system, ideally, the ordinates of corresponding points should be consistent. Therefore, the left-right eye consistency loss is utilized. Penalize the difference in the vertical axis.

[0136] The target training loss is obtained by weighting the 2D supervision loss, reprojection consistency loss, and left / right eye consistency loss. For example, preset weight coefficients are assigned to each of the 2D supervision loss, reprojection consistency loss, and left / right eye consistency loss. The three loss values ​​are then multiplied by their corresponding weights and summed or weighted summed to finally calculate the target training loss. This step integrates constraints from three dimensions: 2D accuracy, 3D geometric consistency, and binocular consistency, forming the total loss function used for backpropagation optimization.

[0137] Based on the above optional embodiments, this application embodiment determines the two-dimensional supervision loss and reprojection consistency loss based on the two-dimensional predicted key points and the ground truth of the key points, determines the left and right eye consistency loss based on the geometric constraint relationship between the left and right eye two-dimensional predicted key points, and obtains the target training loss by weighting the two-dimensional supervision loss, reprojection consistency loss and left and right eye consistency loss. It can construct a multi-task loss function that integrates two-dimensional positioning accuracy, three-dimensional geometric consistency and binocular observation consistency, thereby guiding the initial key point detection model to learn feature representations with stronger geometric rationality and robustness, and significantly improving the key point detection accuracy and three-dimensional pose estimation stability of the model in synthetic data and even real scenes.

[0138] Optionally, determining the reprojection consistency loss based on the two-dimensional predicted key points and the true values ​​of the key points includes: performing triangulation calculation on the two-dimensional predicted key points to obtain the three-dimensional predicted key points; reprojecting the three-dimensional predicted key points to obtain the reprojection result; and determining the reprojection consistency loss based on the reprojection result and the true values ​​of the key points.

[0139] Three-dimensional prediction key points can be a set of three-dimensional spatial coordinates derived from two-dimensional prediction key points through triangulation operations.

[0140] The reprojection result can be a set of two-dimensional coordinates obtained by projecting the three-dimensional predicted key points onto the left and right image planes respectively.

[0141] When calculating the reprojection consistency loss, the 2D predicted keypoints are triangulated to obtain 3D predicted keypoints. For example, the left and right 2D predicted keypoints are extracted, and using pre-calibrated binocular camera intrinsic parameters (focal length, principal point) and extrinsic parameters (baseline, rotation and translation matrix), the intersection coordinates of these two 2D points in 3D space are calculated using geometric triangulation principles, thereby generating the 3D predicted keypoints. This step establishes a mapping relationship between 2D image features and 3D spatial locations.

[0142] The 3D predicted keypoints are reprojected to obtain reprojection results. The reprojection consistency loss is then determined based on these results and the ground truth keypoint values. For example, the calculated 3D predicted keypoints are projected back to the left and right image planes using the left and right camera projection matrices, respectively, generating left-eye and right-eye reprojection results. Subsequently, the errors between the left-eye reprojection results and the left-eye keypoint ground truth values, as well as the errors between the right-eye reprojection results and the right-eye keypoint ground truth values, are calculated, and these errors are combined to form the reprojection consistency loss. This step utilizes geometric constraints to verify the effectiveness of the 3D predicted keypoints, preventing serious deviations in depth estimation.

[0143] Specifically, the reprojection consistency loss is calculated by solving the three-dimensional coordinates of the two-dimensional keypoints predicted by the left and right eyes using a triangulation formula. Then Reproject the images back to the left and right eye image planes respectively, and calculate the mean Euclidean distance between the reprojected pixel coordinates and the original predictions. The calculation formula is as follows:

[0144]

[0145] in, These are the 3D key points obtained through detection and triangulation. and These are the projection operations for the left and right eyes, respectively. and For the corresponding two-dimensional observation or true value, N is the number of valid keypoints.

[0146] The introduction of a reprojection consistency term ensures that the 2D keypoints predicted by the network remain geometrically consistent in 3D space. If there are spatial discrepancies between the left and right view predictions—for example, a keypoint is higher in the left view and lower in the right view—the reprojected position after triangulation will have a significant error compared to the original prediction. The loss function forces the network to correct this inconsistency through the gradient signal. This mechanism has a stabilizing effect on the transfer from synthetic data to real-world scenes. Even if there are differences in texture distribution between the synthetic and real images, the geometric consistency constraint remains effective because the binocular geometric relationship does not change with the appearance distribution.

[0147] Based on the above optional embodiments, this application embodiment obtains three-dimensional predicted key points by triangulating the two-dimensional predicted key points, reprojects the three-dimensional predicted key points to obtain reprojection results, and determines the reprojection consistency loss based on the reprojection results and the true values ​​of the key points. This can utilize three-dimensional geometric constraints to verify the rationality of the two-dimensional detection results, enhance the geometric consistency of the key point detection model during the training process, and thus improve the accuracy and robustness of the model for three-dimensional pose estimation.

[0148] Optionally, the 3D keypoint pose estimation method further includes randomizing at least one of the following during the rendering of the synthetic binocular image: illumination intensity, light source direction, background scene, and image noise.

[0149] Light intensity can be defined as the brightness level produced when a light source illuminates the surface of an object in a scene. During the rendering process, it is expressed as the overall brightness and darkness of the image pixel values.

[0150] The direction of the light source can be the angle of incidence of light when it is projected onto the surface of an object, which determines how the highlights, shadows, and geometric features of the object's surface are visually presented.

[0151] The background scene can be virtual environment content outside the main 3D Gaussian asset, which can be procedurally generated geometry, texture maps, high dynamic range images captured from real scenes, or video frames.

[0152] Image noise can be random pixel value fluctuations in an image, simulating unstructured interference introduced by thermal noise, quantum noise, or signal processing in a real camera sensor, manifesting as graininess or color distortion in the image.

[0153] Randomization can be a process of independently and unpredictably changing the light intensity, light source direction, background scene, or image noise parameters each time a stereo image is rendered and synthesized, using a preset probability distribution or a random number generator.

[0154] To narrow the domain gap between synthetic data and real-machine observations, a domain randomization strategy is overlaid during rendering: the background is randomly sampled from a 3D scene library, procedural random textures, or pre-acquired environment maps; the lighting direction, intensity, and color temperature are randomly perturbed within a reasonable range; and a slight Gaussian blur and random noise are applied to the image after rendering to simulate sensor characteristics. These enhancements make the visual distribution of the training set closer to the real conditions that may be encountered during deployment.

[0155] For example, when the physics simulation engine updates the joint states of the part-level 3D Gaussian asset and prepares to render the left and right eye images, the environment settings module in the rendering pipeline calls a random number generator. For illumination intensity, a brightness coefficient is randomly selected and applied to a global or local light source. For light source direction, a set of spherical coordinate angles is randomly generated to change the ray incident vector. For the background scene, a scene model or texture is randomly selected from a pre-set background library for replacement. For image noise, a noise map is generated based on a selected noise model, such as Gaussian noise or Poisson noise, and superimposed on the rendering result. This process is repeated for each frame or batch of synthesized data to ensure that the generated synthesized stereo images have visual feature diversity.

[0156] Based on the above optional embodiments, this application embodiment, by randomizing at least one of the following during the rendering and generation of synthetic binocular images: illumination intensity, light source direction, background scene, and image noise, can effectively increase the visual diversity of synthetic training data, simulate the complex and varied lighting environment and imaging conditions in the real world, thereby enhancing the robustness of the key point detection model to lighting changes, background interference, and imaging noise, reducing the domain difference between simulated data and real images, and improving the model's generalization ability and detection stability when deployed with a real robot head binocular camera.

[0157] Optionally, the 3D keypoint pose estimation method further includes: obtaining the reprojection error and detection confidence corresponding to the 3D keypoint coordinates; outputting the 3D keypoint coordinates in response to the reprojection error and detection confidence meeting the preset quality conditions; and downgrading the 3D keypoint coordinates in response to the reprojection error and detection confidence not meeting the preset quality conditions, and outputting a structured result with downgraded labels.

[0158] Reprojection error can be defined as the difference in geometric distance between the coordinates of the projected points obtained after projecting the coordinates of 3D keypoints back to the left and right image planes and the coordinates of the 2D keypoints detected in the original images. It is used to measure the geometric consistency of 3D reconstruction.

[0159] Detection confidence score is a score output by the keypoint detection model that represents the probability or credibility of detecting a specific keypoint in the current image, and is used to quantify the reliability of two-dimensional detection results.

[0160] The preset quality conditions can be a set of judgment criteria composed of a pre-defined reprojection error threshold and a detection confidence threshold, used to evaluate whether the quality of the 3D keypoint coordinates at the current moment meets the release requirements.

[0161] Degradation processing is a conservative data output strategy that can be adopted when the quality of the 3D keypoint coordinates does not meet the preset quality conditions. It usually includes retaining the valid 3D keypoint coordinates of the previous frame, marking the data frame as invalid, or reducing the frequency of data updates to prevent incorrect poses from interfering with downstream tasks.

[0162] Structured results with degradation markers can be data packets containing 3D keypoint coordinates, timestamps, coordinate system information, and explicit indications of whether the data status is "degraded" or "untrusted," which can be read and processed accordingly by downstream modules.

[0163] During the deployment phase, after completing binocular triangulation and calculating the 3D keypoint coordinates in the left-eye coordinate system, the system immediately executes a quality verification process. On one hand, the system uses camera intrinsic and extrinsic parameters to reproject the calculated 3D keypoint coordinates onto the image planes of the left and right eyes, calculating the pixel distance between the projected points and the originally detected 2D keypoints in the left and right eyes, thus obtaining the reprojection error. On the other hand, the system extracts the confidence score corresponding to the currently detected 2D keypoints from the forward propagation results of the keypoint detection model. Simultaneously, it acquires the binocular synchronization acquisition flag. These indicators together constitute the basis for evaluating the quality of the current frame pose data.

[0164] For each key point, check whether the three indicators are met simultaneously: the left and right eye reprojection errors do not exceed the preset threshold. The confidence scores for both left and right eye detection are not lower than the preset threshold. The binocular synchronous acquisition is effective, meaning that the difference in timestamps between the left and right eye frames is within the allowable range. and The deployment is frozen after the validation set is calibrated according to availability and security error. When all three conditions are met, the 3D coordinates of the key point are marked as valid and published normally. At this time, the system generates structured data containing a normal status identifier, timestamp, left-eye coordinate system, 3D key point coordinates, and confidence score, and publishes it to downstream modules.

[0165] If any condition is not met, the valid 3D coordinates of the previous frame are retained and a downgrade flag is added. If the keypoint cannot be recovered after exceeding a preset number of frames, the keypoint is marked as invalid and not output to the downstream. Subsequently, the system generates a structured result containing a "downgraded" or "untrusted" flag, a timestamp, a coordinate system, the current 3D keypoint coordinates or the previous frame data or null values, and a confidence level, and publishes it to the downstream modules so that the downstream modules can take appropriate fault tolerance or security strategies based on the flags.

[0166] Based on the above optional embodiments, this application embodiment obtains the reprojection error and detection confidence corresponding to the 3D key point coordinates. In response to the reprojection error and detection confidence meeting the preset quality conditions, it outputs the 3D key point coordinates. In response to the reprojection error and detection confidence not meeting the preset quality conditions, it performs downgrading processing on the 3D key point coordinates and outputs a structured result with downgrading markers. This enables real-time monitoring of the geometric consistency and detection reliability of 3D pose estimation in real-machine deployment scenarios without relying on external motion capture equipment. It effectively filters abnormal key points caused by weak textures, reflections, or occlusions, prevents erroneous pose data from interfering with downstream control logic, and improves the robustness and safety of the humanoid robot head binocular perception system.

[0167] In one optional embodiment, this application addresses the Sim2Real domain transfer problem from simulation to reality. On one hand, it utilizes Gaussian assets reconstructed from real multi-viewpoints to provide realistic textures and reflective properties, compensating for the shortcomings of pure mesh simulation in terms of appearance. On the other hand, it enhances the diversity and robustness of synthetic data by using random lighting, exposure, background replacement, and adding mild blur and noise during rendering. In terms of deployment strategy, after training the network on synthetic data, it can be fine-tuned using a small amount of unlabeled real-device binocular data based on the consistency of left and right eye reprojection, without relying on expensive real-device motion capture data. The transfer effect is verified by metrics such as real-device binocular reprojection error, left and right eye consistency, and keypoint temporal jitter, and is coupled with a quality gating mechanism: if the metrics fail to meet the standards, a degradation strategy is triggered, such as maintaining valid points from the previous frame or marking them as invalid, ensuring that untrusted 3D poses are not published, thereby achieving effective transfer from simulation to real-device while ensuring safety.

[0168] In a different embodiment, if the domain distribution of the synthetic training data still differs significantly from real-machine observations, an additional round of unlabeled real-machine adaptation can be added before deployment: On a small number of unlabeled real-machine binocular images, the consistency of left and right eye reprojection and temporal smoothness are used as self-supervised signals to fine-tune the network parameters without additional motion capture annotations. The parameters are then frozen again for formal deployment. This step is optional and does not constitute a necessary prerequisite for the operation of the neural network-based 3D keypoint pose estimation method in this application.

[0169] In an optional embodiment, the boundary between the offline and online phases occurs after the keypoint detection model parameters are frozen. The offline phase can be performed on a workstation or server cluster, with no real-time requirements on computing resources and time. The online phase involves only forward inference from two images and one triangulation operation, resulting in low computational complexity and enabling real-time frame rates on the humanoid robot's onboard processor. The input sources for the two phases differ: the offline phase requires multi-view RGB images of the target, acquired in a turntable or multi-camera scene, while the online phase only requires synchronized left and right eye images from the head's binocular camera and pre-calibrated camera parameters. The online phase does not perform sparse reconstruction, update Gaussian assets, or initiate physical engine rendering.

[0170] In a different embodiment, the Gaussian-based 3D representation described above can be replaced with Neural Radiance Fields (NeRF). In this case, the training objects in the 3D Gaussian reconstruction stage with depth geometry priors change from Gaussian parameters to network weights of the implicit NeRF, and the calibration depth is constrained by the desired depth obtained through volume rendering. Label propagation in the component-level Gaussian asset and synthetic data stages requires first sampling the NeRF into an explicit point cloud or voxel mesh at a preset resolution, then registering it with the simulated point cloud and propagating the labels. The volume rendering interface of the radiance field is directly called when rendering the synthetic stereo image. Subsequent training and deployment steps remain unchanged. This alternative embodiment may have storage advantages in scenes with rich surface details but extremely high point counts. In another optional embodiment, the Gaussian-based 3D representation described above can also employ other equivalent techniques such as polygonal meshes.

[0171] In one different embodiment, the monocular depth estimation method and its calibration method can be replaced according to actual needs. In terms of algorithm architecture, the specific structure of the target keypoint detection network can be replaced according to performance requirements, and the output can be further fitted into a skeleton model or joint angle parameters.

[0172] Figure 3 This is a schematic diagram of an optional robot according to an embodiment of this application, such as... Figure 3 As shown. The robot 30 includes: a torso 31; two upper limbs 32 and two lower limbs 33 respectively connected to the torso; a binocular camera 34 disposed on the head of the torso; at least one processor 35; and at least one storage medium 36, the at least one storage medium storing instructions, which, when executed by the at least one processor, cause the robot to:

[0173] Step S301: Acquire left and right eye images using a binocular camera;

[0174] Step S302: Input the left and right eye images into the target keypoint detection model to obtain the left eye 2D keypoints and the right eye 2D keypoints. The target keypoint detection model is trained using multiple sets of training data and target training loss. Each set of training data includes: the synthesized binocular image and the corresponding ground truth keypoints. The target training loss is obtained by weighting the 2D supervision loss, reprojection consistency loss and left and right eye consistency loss.

[0175] Step S303: Triangulation calculation is performed based on the left-eye 2D key points, the right-eye 2D key points, and the calibration parameters of the binocular camera to obtain the 3D key point coordinates of the target object in the left-eye coordinate system. The calibration parameters of the binocular camera include at least one of the following: focal length, baseline length, and principal point coordinates.

[0176] The torso can be the main body of the robot, with the upper and lower limbs respectively located on the torso to support the robot's motion structure.

[0177] During the robot's deployment and operation phase, the binocular cameras located on the robot's head simultaneously trigger shutters or continuously capture images to obtain the left and right eye images of the scene at the current moment. These two images serve as input data for subsequent keypoint detection, reflecting the robot's visual perception information in the real environment.

[0178] The left and right images are input into the target keypoint detection model respectively to obtain the left and right 2D keypoints. For example, the acquired left image is input into a pre-trained target keypoint detection model. The model extracts image features and regresses the keypoint positions on the image plane through forward propagation, outputting the left 2D keypoints. Simultaneously, the right image is input into the same target keypoint detection model, outputting the right 2D keypoints. This process only involves model inference calculations and does not update model parameters, achieving a fast mapping from the real image to 2D keypoint coordinates.

[0179] Based on the left and right 2D keypoints and the calibration parameters of the binocular camera, triangulation calculations are performed to obtain the 3D keypoint coordinates of the target object in the left-eye coordinate system. For example, the obtained left and right 2D keypoints are extracted and combined with pre-calibrated binocular camera intrinsic and extrinsic parameters (e.g., intrinsic parameters such as focal length and principal point, and extrinsic parameters such as baseline length and rotation / translation matrix). The intersection of the two lines of sight is calculated using geometric triangulation principles. Through this calculation, the 2D image coordinates are converted into 3D spatial coordinates, ultimately outputting the 3D keypoint coordinates of the target object in the left-eye coordinate system, providing accurate pose information for downstream tasks.

[0180] Based on the above steps S301 to S303, this embodiment of the application uses a binocular camera to acquire left and right eye images, inputs the target key point detection model trained with synthetic data to obtain two-dimensional key points, and performs triangulation calculations in combination with calibration parameters. It can output the three-dimensional key point pose in the left eye coordinate system using a head-mounted binocular camera without relying on external motion capture equipment, multi-view camera arrays, and special markers. This avoids high hardware costs and complex deployment limitations, realizes the combination of realistic appearance and simulated structure, improves simulation transferability, and solves the technical problems of complex hardware deployment and poor simulation transferability in related technologies when performing three-dimensional key point pose estimation.

[0181] Figure 4 This is a schematic diagram of an optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application, as shown below. Figure 4 As shown, this method comprises three stages: 3D Gaussian reconstruction with depth geometric priors, component-level Gaussian assets and synthetic data, and binocular keypoint training and deployment. In the 3D Gaussian reconstruction stage with depth geometric priors, COLMAP sparse reconstruction is performed using multi-view RGB images of the real scene to obtain camera pose and sparse 3D points. Simultaneously, monocular depth is estimated, and scale-offset calibration is performed on the monocular depth using sparse projected depth to obtain a calibrated depth. Subsequently, using the calibrated depth as a geometric prior, a 3D Gaussian sputtering model is trained in conjunction with color reconstruction loss, outputting a high-fidelity 3D Gaussian point cloud and camera parameters.

[0182] In the component-level Gaussian asset and synthetic data stage, the output Gaussian point cloud is coarsely aligned and finely aligned with the simulation model point cloud with joint labels. The simulation joint labels are then propagated to the Gaussian points to construct a component-level 3D Gaussian asset. Next, the joint state of the asset is updated through a physical simulation engine, and synthetic binocular images and corresponding ground truth key values ​​are generated by rendering from the set left and right viewpoints, forming a high-quality synthetic dataset for training.

[0183] In the binocular keypoint training and deployment phase, in training mode, the keypoint detection model is trained using synthetic binocular images and ground truth keypoints. The loss function includes 2D supervised loss, reprojection consistency loss, and left / right eye consistency loss. In deployment mode, the left and right eye RGB images captured by the binocular camera on the real device's head are input into the trained model to detect the left and right eye 2D keypoints respectively. Triangulation calculation is then performed using the binocular camera calibration parameters to calculate the 3D keypoint coordinates of the target object in the left eye coordinate system. Finally, the final result is output after verification by the quality gating module.

[0184] Figure 5 This is a schematic diagram of another optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application, as shown below. Figure 5As shown, the data flow in the system follows a closed-loop structure of "input → processing → output → feedback". The forward data flow starts from binocular observation and calibration parameters, and is processed by the 3D keypoint pose estimation method of this application to obtain intermediate states and quality assessment results, namely 2D keypoints and 3D keypoints. After gating, it forms a structured release for use by downstream decision-making and control modules. The reverse data flow returns from the downstream usage stage to the estimation processing stage, forming a closed loop to optimize the processing.

[0185] Figure 6 This is a schematic diagram of another optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application, as shown below. Figure 6 As shown. After acquiring the observed and intermediate values, this embodiment first performs monocular depth scale-offset calibration using an offset calibration formula to ensure the accuracy of the monocular depth information. Next, it performs 3D Gaussian color and depth joint loss processing to optimize the color and depth information represented by the 3D Gaussian. Then, it performs corrected binocular triangulation, utilizing binocular vision principles for more accurate 3D reconstruction. Subsequently, it evaluates and optimizes the keypoint training process using the total training loss of the binocular keypoints. After completing the above processing, the process enters the threshold judgment stage, determining subsequent operations based on the judgment result. If the judgment is "passed," the system outputs "normal output." If the judgment is "failed," a "degradation or review" strategy is executed to ensure the system's robustness under abnormal conditions. The above process embodies a complete closed loop from data input to processing, optimization, judgment, and response, aiming to ensure the accuracy and reliability of the system output.

[0186] Figure 7 This is a schematic diagram of another optional neural network-based 3D keypoint pose estimation method according to an embodiment of this application, as shown below. Figure 7 As shown, this method first acquires RGB images for both the left and right eyes, then inputs them into a target keypoint detection model. Two-dimensional regression is performed using the keypoints from both eyes, outputting the two-dimensional keypoints for the left and right eyes respectively. The training process utilizes a target loss function, and the parameters are frozen after training and not updated during deployment. Then, triangulation is performed based on the left and right eye two-dimensional keypoints and the calibration parameters of the binocular camera to obtain the three-dimensional keypoint coordinates of the target object in the left eye coordinate system. Finally, gating is performed based on the reprojection error and detection confidence corresponding to the three-dimensional keypoint coordinates. If the preset quality conditions are met, normal deployment is performed; otherwise, downgrading is applied.

[0187] Figure 8 This is a structural block diagram of an optional neural network-based 3D keypoint pose estimation device according to an embodiment of this application, such as... Figure 8As shown. The device includes: an acquisition module 801, used to acquire left and right eye images through a binocular camera; a detection module 802, used to input the left and right eye images into a target keypoint detection model to obtain left-eye two-dimensional keypoints and right-eye two-dimensional keypoints, wherein the target keypoint detection model is trained using multiple sets of training data and target training loss to train an initial keypoint detection model, each set of training data includes: a synthesized binocular image and the corresponding ground truth value of keypoints, and the target training loss is obtained by weighted calculation based on two-dimensional supervision loss, reprojection consistency loss and left-right eye consistency loss; and a calculation module 803, used to perform triangulation calculation based on the left-eye two-dimensional keypoints, right-eye two-dimensional keypoints and the calibration parameters of the binocular camera to obtain the three-dimensional keypoint coordinates of the target object in the left-eye coordinate system, wherein the calibration parameters of the binocular camera include at least one of the following: focal length, baseline length and principal point coordinates.

[0188] Optionally, the 3D keypoint pose estimation device further includes: a reconstruction module for sparse reconstruction of multi-view images of the target object to obtain sparse 3D points; an estimation module for pixel-level depth estimation of multi-view images to obtain pixel-level depth; a calibration module for scale offset calibration of the pixel-level depth based on the projected depth of the sparse 3D points to obtain a calibration depth; an alignment module for determining the 3D representation data of the target object based on the calibration depth, and registering and aligning the 3D representation data of the target object with a preset simulation model to obtain a component-level 3D asset; a rendering module for updating the joint state of the component-level 3D asset through a simulation engine, and rendering and generating a synthetic stereo image and corresponding keypoint ground truth; and a training module for training an initial keypoint detection model based on the synthetic stereo image and keypoint ground truth to obtain a target keypoint detection model.

[0189] Optionally, the calibration module is also used to: construct a depth calibration optimization target based on the projected depth, pixel-level depth, and geometric constraint weights of sparse 3D points, wherein the depth calibration optimization target is used to represent the minimum value of the weighted sum of squared residuals; solve the depth calibration optimization target to obtain the target scale coefficient and target offset; and use the target scale coefficient and target offset to perform a linear transformation on the pixel-level depth of all pixels in the image to obtain the calibration depth.

[0190] Optionally, the alignment module is also used to: perform downsampling processing on the 3D representation data to obtain target point cloud data, and perform downsampling processing on the point cloud data of the preset simulation model to obtain simulation point cloud data; perform alignment processing on the target point cloud data and the simulation point cloud data to obtain point cloud alignment results; based on the point cloud alignment results, map the joint component labels in the preset simulation model to the 3D representation data to obtain component label allocation results; and generate component-level 3D assets based on the component label allocation results.

[0191] Optionally, the training module is further configured to: perform prediction processing on the synthetic binocular image using the initial keypoint detection model to obtain two-dimensional predicted keypoints, wherein the two-dimensional predicted keypoints include: two-dimensional predicted keypoints for the left eye and two-dimensional predicted keypoints for the right eye; determine the target training loss based on the two-dimensional predicted keypoints and the ground truth values ​​of the keypoints; and update the model parameters of the initial keypoint detection model using the target training loss to obtain the target keypoint detection model.

[0192] Optionally, the training module is also used to: determine the two-dimensional supervision loss and reprojection consistency loss based on the two-dimensional predicted keypoints and the ground truth of the keypoints; determine the left and right eye consistency loss based on the geometric constraint relationship between the left and right eye two-dimensional predicted keypoints; and perform a weighted operation on the two-dimensional supervision loss, reprojection consistency loss and left and right eye consistency loss to obtain the target training loss.

[0193] Optionally, the training module is also used to: perform triangulation calculation on the two-dimensional predicted key points to obtain the three-dimensional predicted key points; perform reprojection on the three-dimensional predicted key points to obtain the reprojection result; and determine the reprojection consistency loss based on the reprojection result and the ground truth value of the key points.

[0194] Optionally, the 3D keypoint pose estimation device further includes a processing module for randomizing at least one of illumination intensity, light source direction, background scene, and image noise during the rendering of a synthetic binocular image.

[0195] Optionally, the 3D keypoint pose estimation device further includes: an acquisition module for acquiring the reprojection error and detection confidence corresponding to the 3D keypoint coordinates; a first output module for outputting the 3D keypoint coordinates in response to the reprojection error and detection confidence meeting a preset quality condition; and a second output module for downgrading the 3D keypoint coordinates in response to the reprojection error and detection confidence not meeting the preset quality condition, and outputting a structured result with downgrade markers.

[0196] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0197] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0198] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0199] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0200] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0201] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0203] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0205] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0206] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for 3D keypoint pose estimation based on neural networks, characterized in that, include: Acquire left and right eye images using a binocular camera; The left and right eye images are respectively input into the target keypoint detection model to obtain left eye 2D keypoints and right eye 2D keypoints. The target keypoint detection model is obtained by training an initial keypoint detection model using multiple sets of training data and a target training loss. Each set of training data includes: a synthesized binocular image and the corresponding ground truth value of keypoints. The target training loss is obtained by weighting two-dimensional supervision loss, reprojection consistency loss and left and right eye consistency loss. Based on the left-eye 2D key points, the right-eye 2D key points, and the calibration parameters of the binocular camera, triangulation calculations are performed to obtain the 3D key point coordinates of the target object in the left-eye coordinate system. The calibration parameters of the binocular camera include at least one of the following: focal length, baseline length, and principal point coordinates.

2. The method according to claim 1, characterized in that, The method further includes: Sparse reconstruction is performed on the multi-view images of the target object to obtain sparse three-dimensional points; Pixel-level depth estimation is performed on the multi-view images to obtain pixel-level depth; Using the projected depth of the sparse three-dimensional points as a reference, the pixel-level depth is calibrated by scale offset to obtain the calibrated depth; Based on the calibration depth, the three-dimensional representation data of the target object is determined, and the three-dimensional representation data of the target object is registered and aligned with a preset simulation model to obtain a component-level three-dimensional asset; The joint states of the component-level 3D assets are updated through a simulation engine, and the synthetic binocular image and the corresponding ground truth values ​​of the key points are rendered and generated. The initial keypoint detection model is trained based on the synthesized stereo image and the ground truth of the keypoints to obtain the target keypoint detection model.

3. The method according to claim 2, characterized in that, Using the projected depth of the sparse 3D points as a reference, the pixel-level depth is scale-off calibrated to obtain the calibrated depth, which includes: A depth calibration optimization objective is constructed based on the projected depth of the sparse 3D points, the pixel-level depth, and the geometric constraint weights, wherein the depth calibration optimization objective is used to represent the minimum value of the weighted residual sum of squares; The depth calibration optimization target is solved to obtain the target scale coefficient and target offset; Using the target scale coefficient and target offset, the pixel-level depth of all pixels in the image is linearly transformed to obtain the calibration depth.

4. The method according to claim 2, characterized in that, By registering and aligning the three-dimensional representation data of the target object with the preset simulation model, the component-level three-dimensional asset is obtained, including: The three-dimensional representation data is downsampled to obtain target point cloud data, and the point cloud data of the preset simulation model is downsampled to obtain simulation point cloud data. The target point cloud data and the simulated point cloud data are aligned to obtain the point cloud alignment result. Based on the point cloud alignment results, the joint component labels in the preset simulation model are mapped to the three-dimensional representation data to obtain the component label assignment results; The component-level 3D asset is generated based on the component label assignment results.

5. The method according to claim 2, characterized in that, The initial keypoint detection model is trained based on the synthesized stereo image and the ground truth values ​​of the keypoints to obtain the target keypoint detection model, which includes: The initial key point detection model is used to perform prediction processing on the synthetic binocular image to obtain two-dimensional predicted key points, wherein the two-dimensional predicted key points include: left-eye two-dimensional predicted key points and right-eye two-dimensional predicted key points; The target training loss is determined based on the two-dimensional predicted key points and the ground truth values ​​of the key points; The model parameters of the initial keypoint detection model are updated using the target training loss to obtain the target keypoint detection model.

6. The method according to claim 5, characterized in that, Determining the target training loss based on the two-dimensional predicted keypoints and the ground truth values ​​of the keypoints includes: The two-dimensional supervision loss and the reprojection consistency loss are determined based on the two-dimensional predicted key points and the true values ​​of the key points. Based on the geometric constraint relationship between the left-eye 2D prediction key points and the right-eye 2D prediction key points, the left-eye consistency loss is determined; The target training loss is obtained by weighting the two-dimensional supervision loss, the reprojection consistency loss, and the left-right eye consistency loss.

7. The method according to claim 6, characterized in that, Determining the reprojection consistency loss based on the two-dimensional predicted key points and the true values ​​of the key points includes: Triangulation calculation is performed on the two-dimensional predicted key points to obtain the three-dimensional predicted key points; The three-dimensional predicted key points are reprojected to obtain the reprojection result; The reprojection consistency loss is determined based on the reprojection results and the true values ​​of the key points.

8. The method according to claim 1, characterized in that, The method further includes: During the rendering process of generating a synthetic binocular image, at least one of the following is randomized: illumination intensity, light source direction, background scene, and image noise.

9. The method according to claim 1, characterized in that, The method further includes: Obtain the reprojection error and detection confidence level corresponding to the coordinates of the three-dimensional key points; In response to the reprojection error and the detection confidence meeting the preset quality conditions, the coordinates of the three-dimensional key points are output; In response to the reprojection error and the detection confidence level failing to meet the preset quality conditions, the coordinates of the three-dimensional key points are downgraded, and a structured result with downgrade markers is output.

10. A robot, characterized in that, include: trunk; Two upper limbs and two lower limbs, respectively connected to the torso; A binocular camera is mounted on the head of the torso; At least one processor; as well as At least one storage medium storing instructions that, when executed by the at least one processor, cause the robot to: The binocular camera acquires left and right eye images; The left and right eye images are respectively input into the target keypoint detection model to obtain left eye 2D keypoints and right eye 2D keypoints. The target keypoint detection model is obtained by training an initial keypoint detection model using multiple sets of training data and a target training loss. Each set of training data includes: a synthesized binocular image and the corresponding ground truth value of keypoints. The target training loss is obtained by weighting two-dimensional supervision loss, reprojection consistency loss and left and right eye consistency loss. Based on the left-eye 2D key points, the right-eye 2D key points, and the calibration parameters of the binocular camera, triangulation calculations are performed to obtain the 3D key point coordinates of the target object in the left-eye coordinate system. The calibration parameters of the binocular camera include at least one of the following: focal length, baseline length, and principal point coordinates.

Images