A somatic intelligent lightweight 6D pose estimation tracking method, device and medium

Abstract

The application discloses a body intelligent lightweight 6D pose estimation tracking method and device and a medium, and relates to the field of body intelligence, and comprises the following steps: acquiring a current frame RGB-D image; performing shared feature coding on the current frame RGB-D image to obtain current frame fusion features; performing initial pose regression on the current frame fusion features to obtain an initial 6D pose; calculating a pose residual through a differential iterative closest point network layer according to the initial 6D pose and the current frame fusion features, and iteratively updating the initial 6D pose based on the pose residual to obtain an accurate 6D pose; the differential iterative closest point network layer is a differential neural network layer, and the initial 6D pose and the current frame fusion features are input to perform differential feature correlation on the current frame fusion features based on the initial 6D pose, and the pose residual is output by regression. The application can realize real-time and high-precision six-degree-of-freedom object pose estimation on an edge device.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a lightweight, embodied intelligent 6D pose estimation and tracking method, device, and medium. Background Technology

[0002] In applications such as embodied intelligence and robotic manipulation, machine vision systems need to accurately acquire the six-degree-of-freedom pose of target objects to support delicate operations such as grasping and assembly. Furthermore, six-degree-of-freedom pose estimation involves the joint determination of position and orientation, and often needs to be deployed on edge devices with limited computing resources, placing high demands on the real-time performance and accuracy of the algorithms.

[0003] Currently, in existing technologies, one type of method relies on iterative registration using the physical model of the object, gradually approximating the true pose by minimizing geometric errors. This type of method achieves high accuracy under ideal conditions, but the computational cost of the iterative process is high, making it difficult to meet real-time processing requirements. Furthermore, another type of method directly regresses the pose based on a data-driven approach, which is faster, but its accuracy significantly decreases under complex conditions such as occlusion and noise. Due to the shortcomings of these existing technologies, current methods often require a compromise between accuracy and real-time performance in practical deployments, making it difficult to simultaneously achieve real-time and high-precision six-DOF object pose estimation and tracking on edge devices. Summary of the Invention

[0004] This invention provides a lightweight, intelligent 6D pose estimation and tracking method, device, and medium, aiming to solve the technical problem of how to provide an effective solution for real-time and high-precision six-degree-of-freedom object pose estimation on edge devices.

[0005] In a first aspect, the present invention provides an embodied intelligent lightweight 6D pose estimation and tracking method, comprising: Get the RGB-D image of the current frame; Shared feature encoding is performed on the current frame RGB-D image to obtain the current frame fusion feature; Initialize the pose regression of the current frame fusion features to obtain the initial 6D pose; Based on the initial 6D pose and the current frame fusion features, the pose residual is calculated through a differentiable iterative nearest-point network layer, and the initial 6D pose is iteratively updated based on the pose residual to obtain the accurate 6D pose. The differentiable iterative nearest point network layer is a differentiable neural network layer. The differentiable iterative nearest point network layer takes the initial 6D pose and the current frame fusion features as input, performs differentiable feature association on the current frame fusion features based on the initial 6D pose, and regresses and outputs the pose residual based on the differentiable feature association results.

[0006] Optionally, the step of calculating the pose residual through a differentiable iterative nearest-neighbor network layer based on the initial 6D pose and the fused features of the current frame, and iteratively updating the initial 6D pose based on the pose residual to obtain the accurate 6D pose, includes: Determine if the current frame is in tracking mode; If it is in tracking mode, the pose of the previous frame of the current frame RGB-D image is obtained. The pose of the previous frame is the precise 6D pose of the previous frame RGB-D image of the current frame RGB-D image. The pose of the previous frame is converted into a prior transformation matrix. The prior transformation matrix is ​​applied to the current frame fusion feature to obtain the prior alignment feature. The prior alignment feature and the initial 6D pose are input into the differentiable iterative nearest point network layer to calculate the first pose residual. If it is not in tracking mode, the current frame fusion features and the initial 6D pose are input into the differentiable iterative nearest point network layer to calculate the second pose residual; The initial 6D pose is iteratively updated based on the first pose residual or the second pose residual to obtain the accurate 6D pose.

[0007] Optionally, the step of encoding shared features in the current frame RGB-D image to obtain the current frame fused features includes: Bilateral filtering is performed on the depth image in the current frame RGB-D image to obtain a denoised depth image; The denoised depth image is filled with depth holes to obtain a preprocessed depth image; The preprocessed depth image is converted into point cloud data. Perform a two-dimensional convolution on the RGB image in the current frame's RGB-D image to obtain RGB features; Perform 3D sparse convolution on the point cloud data to obtain geometric features; The geometric features are upsampled so that the upsampled geometric features have the same spatial resolution as the RGB features, thus obtaining aligned geometric features; Attention-weighted fusion is performed on the RGB features and the alignment geometric features to obtain point-by-point fusion weights, and the RGB features and the alignment geometric features are weighted and summed according to the point-by-point fusion weights to obtain RGB-D fusion features; The RGB-D fusion features are downsampled at multiple scales to obtain downsampled features at multiple scales; The downsampled features from the multiple scales are concatenated, and the concatenated features are then subjected to convolutional dimensionality reduction to obtain the fused features of the current frame.

[0008] Optionally, the step of encoding shared features in the current frame RGB-D image to obtain the current frame fused features includes: Bilateral filtering is performed on the depth image in the current frame RGB-D image to obtain a denoised depth image; The denoised depth image is filled with depth holes to obtain a preprocessed depth image; The preprocessed depth image is converted into point cloud data. The point cloud data is sampled from the farthest point to obtain a set of sampled points; A local neighborhood graph is constructed based on the set of sampling points, and graph convolution is performed on the local neighborhood graph to obtain local geometric features; The local geometric features are extracted point by point using a multilayer perceptron, and the extracted point by point features are max pooled to obtain the global geometric features. Perform a two-dimensional convolution on the RGB image in the current frame's RGB-D image to obtain RGB features; The global geometric features are broadcast to the same spatial size as the RGB features and then concatenated with the RGB features to obtain the concatenated features; Channel attention weighting is applied to the spliced ​​features to obtain RGB-D fusion features; The RGB-D fusion features are downsampled at multiple scales to obtain downsampled features at multiple scales; The downsampled features from the multiple scales are concatenated, and the concatenated features are then subjected to convolutional dimensionality reduction to obtain the fused features of the current frame.

[0009] Optionally, the initial pose regression of the fused features of the current frame to obtain the initial 6D pose includes: Semantic segmentation is performed on the fused features of the current frame to obtain a foreground point set; For each point in the foreground point set, a direction vector prediction is performed to obtain the direction vector of each point in the foreground point set pointing to the key point of the object. The direction vector set is composed of the direction vectors of each point in the foreground point set pointing to the key point of the object. Clustering and voting are performed on the set of direction vectors to obtain the locations of multiple key points; Random sampling and consistency filtering are performed on the multiple key point locations to remove outlier key points and obtain the effective key point locations. Based on the effective key point locations and the pre-stored object model, the initial 6D pose is obtained through least squares fitting.

[0010] Optionally, the initial pose regression of the fused features of the current frame to obtain the initial 6D pose includes: Global average pooling is performed on the fused features of the current frame to obtain global features; The global features are input into the translation regression layer to obtain the translation vector; The global features are input into the rotational regression layer to obtain rotational quaternions; The rotated quaternion is normalized to obtain a normalized rotated quaternion; The initial 6D pose is obtained by combining the translation vector and the normalized rotation quaternion.

[0011] Optionally, the method further includes: Obtain a training dataset, which contains multiple sample RGB-D images and ground truth poses corresponding to each sample RGB-D image; Shared feature encoding is performed on each sample RGB-D image to obtain sample fusion features; The sample fusion features are initialized with pose regression to obtain the initial 6D pose of the sample; Based on the initial 6D pose of the sample and the fusion features of the sample, the sample pose residual is calculated through the differentiable iterative nearest point network layer, and the initial 6D pose of the sample is iteratively updated based on the sample pose residual to obtain the accurate 6D pose of the sample. The pose loss is calculated based on the initial 6D pose of the sample and the ground truth pose, and the point-to-point distance loss is calculated based on the accurate 6D pose of the sample and the ground truth pose. A course learning strategy is adopted. First, the network parameters used to perform the initial pose regression are trained only based on the pose loss. Then, the network parameters used to perform the initial pose regression and the differentiable iterative nearest point network layer are trained together based on the pose loss and the point-to-point distance loss to obtain the integrated network parameters after training. The trained integrated network parameters are converted into edge inference format and quantized using INT8 to obtain a real-time inference model for edge devices.

[0012] Optionally, the step of converting the trained integrated network parameters into an edge inference format and performing INT8 quantization to obtain a real-time inference model for edge devices includes: Obtain a calibration dataset, which contains multiple RGB-D images; Each RGB-D image in the calibration dataset is input into an inference network loaded with the trained integrated network parameters, and the activation value range of each network layer in the inference network is calculated. Based on the activation value range of each network layer in the inference network, determine the quantization scale corresponding to each network layer in the inference network; Based on the quantization scale corresponding to each network layer in the inference network, the parameters of the trained integrated network are quantized layer by layer using INT8 to obtain the quantized network parameters. The quantized network parameters are converted into an edge inference format, and the converted quantized network parameters and the quantization scale corresponding to each network layer in the inference network are encapsulated to obtain the real-time inference model of the edge device.

[0013] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0015] This invention provides an embodied intelligent lightweight 6D pose estimation and tracking method, device, and medium. It encodes shared features of the current frame's RGB-D image to obtain the current frame's fused features. These fused features serve as input to both the initial pose regression and the differentiable iterative nearest-point network layer, avoiding redundant computations caused by extracting features from the same image separately, thus reducing overall computational overhead. Furthermore, initial pose regression using the current frame's fused features quickly yields the initial 6D pose, providing a near-true starting value for subsequent optimization. This eliminates the need for iterative optimization to start from a large error state, reducing the number of iterations required for convergence and improving processing speed. Further, the differentiable iterative nearest-point network layer takes the initial 6D pose and the current frame's fused features as input. It establishes a correspondence through differentiable feature association and regresses the pose residual, transforming the traditional non-differentiable iterative nearest-point optimization process into a differentiable neural network operation. This retains the ability of iterative refinement to improve pose accuracy while allowing the process to be optimized through training, improving the accuracy of pose correction. Finally, the initial 6D pose is iteratively updated based on the pose residual, gradually reducing the pose error to obtain an accurate 6D pose. Therefore, this method improves pose estimation accuracy while reducing computational overhead, enabling real-time and high-precision six-DOF object pose estimation on edge devices. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1A flowchart illustrating an embodied intelligent lightweight 6D pose estimation and tracking method provided in an embodiment of the present invention; Figure 2 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Please see Figure 1This invention provides a lightweight 6D pose estimation and tracking method for embodied intelligence. Embodied intelligence refers to a technological paradigm in which an intelligent agent utilizes its own sensors and actuators to complete perception, decision-making, and operation during interaction with the physical environment. When an embodied intelligent robot performs operations such as grasping, assembly, or navigation, the machine vision system needs to estimate the position and orientation of the target object in three-dimensional space based on data collected by the vision sensors. This position and orientation is called the 6D pose. The 6D pose consists of 3 translational degrees of freedom and 3 rotational degrees of freedom, where the translational degrees of freedom describe the spatial position of the object's center in the camera coordinate system, and the rotational degrees of freedom describe the object's orientation relative to the camera coordinate system. The accuracy of 6D pose estimation directly affects the success rate of subsequent robot operations. For example, when a robotic arm grasps an object, it needs to plan the approach direction and grasping position of the end effector based on the object's pose. Furthermore, edge devices refer to small computing devices deployed on the robot body or in industrial settings, whose computing resources, storage space, and power consumption budgets are lower than those of server-level devices. The machine vision system running on the edge device needs to process image data and output pose results in real time under limited computing power. Current high-precision 6D pose estimation methods typically rely on iterative nearest-point (RIB) optimization algorithms to repeatedly correct the pose. These methods are computationally intensive and time-consuming, making them unsuitable for the real-time requirements of edge devices. While direct regression methods are faster, their accuracy is insufficient in complex scenarios such as occlusion, noise interference, and lighting variations. Therefore, the method provided in this embodiment integrates initialization regression and iterative optimization into a single network using shared feature encoding and a differentiable iterative nearest-point network layer. This enables real-time and high-precision 6D pose estimation and tracking on edge devices. Specifically, the method includes the following steps: S1, acquire the RGB-D image of the current frame.

[0024] In specific implementation, the machine vision system first acquires the current frame RGB-D image. An RGB-D image refers to image data that simultaneously contains color and depth information. The RGB image is acquired by a color camera and contains color and texture information of the object's surface; the depth image is acquired by a depth sensor and contains distance information from each pixel to the camera's imaging plane. Further, the depth sensor can be a structured light camera, a time-of-flight camera, or a stereo camera; this invention is not specifically limited to these. A structured light camera calculates depth by projecting an coded light spot and analyzing its deformation; a time-of-flight camera calculates depth by measuring the round-trip time of the infrared light signal; and a stereo camera calculates depth by the parallax between the left and right eye images. The color image and depth image in the RGB-D image are registered and aligned so that each pixel in the color image corresponds to a depth value in the depth image. Further, the current frame RGB-D image refers to the RGB-D image acquired by the vision sensor at the current processing moment, and is the original input data for all subsequent processing steps.

[0025] S2, perform shared feature encoding on the current frame RGB-D image to obtain the current frame fusion feature.

[0026] In specific implementation, after acquiring the current frame RGB-D image, shared feature encoding is performed on the current frame RGB-D image to obtain the current frame fused features. Shared feature encoding refers to using the same encoder network to extract features and fuse cross-modal information from color and depth information in the RGB-D image, generating a unified feature representation. Furthermore, this feature representation is used simultaneously by the subsequent initial pose regression branch and the differentiable iterative nearest-point network layer, hence the term shared features. Further, the encoder can adopt a lightweight network structure based on point cloud networks or sparse convolution. A point cloud network is a neural network that directly processes 3D point cloud data, extracting and aggregating features from each point in the point cloud to obtain global or local features. Point cloud data is a set of 3D spatial points converted from a depth image, where each point has 3D coordinates and corresponding features. Further, sparse convolution is a convolution method that performs convolution operations only at non-zero positions, suitable for sparse 3D data formed after point cloudification of depth images. Furthermore, since most of the space in point cloud data does not contain points, ordinary 3D convolution requires a huge amount of computation, while sparse convolution only convolves the voxel positions where points exist, greatly reducing computational overhead.

[0027] Furthermore, the specific processing steps of shared feature encoding are as follows: RGB images are input into a 2D convolutional network to extract color features; the depth image is converted into a point cloud and then input into a 3D feature extraction network to extract geometric features; finally, the color features and geometric features are fused to obtain the current frame fused features. The current frame fused features can be a high-dimensional feature map or a set of point features. The feature vector at each spatial location in the feature map or the feature vector at each point in the point feature set contains the color and geometric information corresponding to that location or point. Furthermore, the spatial resolution and number of channels of the current frame fused features can be adjusted according to actual accuracy and speed requirements. The significance of shared feature encoding lies in the fact that the initial pose regression branch requires relatively global and semantic features to predict a coarse pose, while the differentiable iterative nearest neighbor network layer requires relatively local and geometric features for point cloud association and residual calculation. By sharing the same set of features, the overall network parameter count and computational redundancy can be reduced, and the feature can be adapted to the needs of both branches through training.

[0028] In some preferred embodiments, the step of encoding shared features in the current frame RGB-D image to obtain the current frame fusion features includes: performing bilateral filtering on the depth image in the current frame RGB-D image to obtain a denoised depth image; performing depth hole filling on the denoised depth image to obtain a preprocessed depth image; performing point cloudification on the preprocessed depth image to obtain point cloud data; performing two-dimensional convolution on the RGB image in the current frame RGB-D image to obtain RGB features; performing three-dimensional sparse convolution on the point cloud data to obtain geometric features; and performing... Upsampling is performed to make the upsampled geometric features consistent with the spatial resolution of the RGB features, resulting in aligned geometric features. Attention-weighted fusion is applied to the RGB features and the aligned geometric features to obtain point-by-point fusion weights. The RGB features and the aligned geometric features are then weighted and summed according to the point-by-point fusion weights to obtain RGB-D fusion features. Multi-scale downsampling is performed on the RGB-D fusion features to obtain downsampled features at multiple scales. The downsampled features at multiple scales are concatenated, and the concatenated features are subjected to convolutional dimensionality reduction to obtain the current frame fusion features.

[0029] In practical implementation, depth images often contain noise and holes in 6D pose estimation tasks. Directly converting the original depth image into point clouds can affect the quality of subsequent feature extraction. Furthermore, color features and geometric features have significant differences in spatial resolution and modality, making simple fusion insufficient to fully utilize multimodal information. Therefore, this implementation improves the robustness and expressive power of shared feature encoding through depth image preprocessing, 3D sparse convolution, spatial alignment, attention-weighted fusion, and multi-scale feature fusion. The specific implementation is as follows: Bilateral filtering is applied to the depth image in the current frame's RGB-D image to obtain a denoised depth image. Bilateral filtering is an image filtering method that preserves edges while denoising. Its filter kernel is jointly determined by a Gaussian function in the spatial domain and a Gaussian function in the pixel value domain. Furthermore, in the depth image, the Gaussian function in the spatial domain increases the weight of pixels with closer neighborhoods, while the Gaussian function in the pixel value domain increases the weight of pixels with smaller depth value differences. Thus, bilateral filtering smooths depth noise while preserving depth transitions at object edges, preventing edge blurring. Furthermore, noise in the depth image may originate from measurement errors in the depth sensor itself or environmental interference; for example, time-of-flight cameras can generate significant noise on low-reflectivity surfaces. Bilateral filtering can effectively suppress this noise.

[0030] Further, the denoised depth image is filled with depth holes to obtain a preprocessed depth image. Holes may appear in the depth image due to occlusion, reflection, or sensor blind spots, resulting in a lack of effective depth values ​​at these holes. Depth hole filling can employ nearest-neighbor filling or interpolation methods. Specifically, nearest-neighbor filling sets the depth value of the hole pixel to the depth value of the nearest known depth pixel to the hole. Interpolation methods use linear or non-linear interpolation based on the known depth values ​​surrounding the hole; this is not specifically limited in this invention. The filled depth image will not produce holed point clouds due to missing depth values ​​during subsequent point cloudification.

[0031] Further, the preprocessed depth image is converted into point cloud data. Point cloud conversion refers to converting two-dimensional depth pixels into points in three-dimensional space based on the pixel coordinates and depth values ​​of the depth image, combined with camera intrinsic parameters. The camera intrinsic parameters include focal length and principal point coordinates, describing the mapping relationship between pixel coordinates and the camera coordinate system. For each pixel with a valid depth value in the depth image, its three-dimensional coordinates in the camera coordinate system are: x equals the difference between the pixel's x-coordinate and the principal point's x-coordinate multiplied by the depth value and then divided by the focal length; y equals the difference between the pixel's y-coordinate and the principal point's y-coordinate multiplied by the depth value and then divided by the focal length; z equals the depth value. After conversion, point cloud data is obtained, where each point has three-dimensional coordinates, reflecting the geometry of the object's surface.

[0032] Furthermore, a two-dimensional convolution is performed on the RGB image in the current frame's RGB-D image to obtain RGB features. Two-dimensional convolution is an operation that extracts local features by sliding a convolution kernel across the image. The convolution kernel is a learnable weight matrix that slides across the input image with a certain stride, calculating the dot product between the kernel and a local region of the input image at each position, and outputting a feature value. Further, through multi-layer two-dimensional convolution and pooling operations, edge, texture, and semantic information in the image can be extracted. Pooling is an operation that downsamples local region features; max pooling and average pooling are commonly used. RGB features are two-dimensional feature maps with high spatial resolution. Two-dimensional convolution can use a pre-trained lightweight network as the backbone, such as MobileNet or ShuffleNet, to reduce computational cost. MobileNet and ShuffleNet are both lightweight convolutional neural network architectures designed for mobile or edge devices.

[0033] Furthermore, a 3D sparse convolution is performed on the point cloud data to obtain geometric features. Point cloud data is sparse 3D data; if it is voxelized and ordinary 3D convolution is used, the computational cost is enormous because most voxels are empty. Voxelization refers to dividing the 3D space into cubic units of the same size and assigning points in the point cloud to corresponding units. Furthermore, the 3D sparse convolution only performs convolution operations on non-empty voxels, recording the positions of non-empty voxels using a hash table or sparse index, significantly reducing the computational cost. Sparse convolution can be a submanifold sparse convolution, which keeps the output activation position unchanged only at the input activation position to avoid activation diffusion; or it can be a regular sparse convolution, allowing the output activation position to expand. Geometric features reflect the 3D geometric structure of the object's surface. The input to the 3D sparse convolution is the sparse voxel features after voxelizing the point cloud, where the feature of each voxel is the mean or maximum value of the features of all points within the voxel.

[0034] Furthermore, the geometric features are upsampled to achieve the same spatial resolution as the RGB features, resulting in aligned geometric features. Since point cloud data and RGB images have different spatial resolutions, the resolution of the geometric features may be lower than that of the RGB features. Upsampling operations can employ methods such as nearest neighbor interpolation, bilinear interpolation, or transposed convolution to increase the resolution of the geometric features to match that of the RGB features, facilitating subsequent fusion. Transposed convolution is a learnable upsampling method that expands the feature map size by inserting zero values ​​between input features and performing convolution. The aligned geometric features correspond one-to-one with the RGB features in spatial dimensions.

[0035] Furthermore, attention-weighted fusion is performed on the RGB features and the aligned geometric features to obtain point-by-point fusion weights. Then, the RGB features and the aligned geometric features are weighted and summed according to these point-by-point fusion weights to obtain the RGB-D fusion features. Attention-weighted fusion is an adaptive feature fusion method. First, the RGB features and the aligned geometric features are concatenated along the channel dimension and input into a convolutional network, outputting point-by-point fusion weights with the same resolution as the feature map space. The point-by-point fusion weights are single-channel weight maps with the same size as the feature map space, and the weight values ​​range from 0 to 1. Further, the RGB features and the aligned geometric features are weighted and summed according to the point-by-point fusion weights. The fusion result is the point-by-point fusion weight multiplied by the RGB feature plus 1 minus the point-by-point fusion weight multiplied by the aligned geometric feature. Thus, in regions rich in color information, the point-by-point fusion weights are close to 1, and the fusion result relies more on the RGB features; in regions where the geometric structure is important, the point-by-point fusion weights are close to 0, and the fusion result relies more on the geometric features. Attention-weighted fusion enables the network to adaptively select modalities with richer information, improving the discriminative power of the fused features.

[0036] Furthermore, the RGB-D fused features are downsampled at multiple scales to obtain downsampled features at multiple scales. Multi-scale downsampling refers to downsampling the feature map at different magnifications, such as 2x, 4x, and 8x downsampling, resulting in multiple feature maps with different resolutions. Downsampling can be implemented using max pooling or strided convolution. Stride convolution refers to a convolution kernel sliding with a stride greater than 1 across the input features, thereby reducing the size of the output feature map. Feature maps at different scales capture object information at different scales; small-scale features contain global contextual information, while large-scale features contain local detail information.

[0037] Further, the downsampled features from multiple scales are concatenated, and the concatenated features are subjected to convolutional dimensionality reduction to obtain the current frame fused features. The concatenation operation connects multiple features of different resolutions along the channel dimension; before concatenation, the features at each scale need to be upsampled to a uniform resolution. Convolutional dimensionality reduction compresses the concatenated high-dimensional features into a feature representation of appropriate dimensions using 1x1 convolutions. A 1x1 convolution refers to a convolution operation with a kernel size of 1x1, used to change the number of feature channels without changing the spatial resolution. The current frame fused features, as the output of shared feature encoding, are used jointly by the initialization pose regression branch and the differentiable iterative nearest-point network layer.

[0038] This implementation improves the quality of point cloud data and reduces the impact of noise and missing data on geometric feature extraction through bilateral filtering and hole filling of depth images. Furthermore, 3D sparse convolution reduces computational overhead while preserving geometric structure information. Further, spatial alignment and attention-weighted fusion address the resolution inconsistencies and modal differences between color and geometric features, adaptively fusing color and geometric information to improve the discriminative power of the fused features. Furthermore, multi-scale downsampling and stitching further enrich the multi-scale expressive power of features, enabling shared feature encoding to adapt to objects of different sizes and distances. These processes collectively improve the input feature quality of subsequent initial pose regression and differentiable iterative nearest-point network layers, thereby enhancing the accuracy and robustness of 6D pose estimation.

[0039] Alternatively, in some preferred embodiments, the step of encoding shared features in the current frame RGB-D image to obtain the current frame fused features includes: performing bilateral filtering on the depth image in the current frame RGB-D image to obtain a denoised depth image; performing depth hole filling on the denoised depth image to obtain a preprocessed depth image; converting the preprocessed depth image into point cloud data; sampling the farthest point of the point cloud data to obtain a sampling point set; constructing a local neighborhood graph based on the sampling point set, and performing graph convolution on the local neighborhood graph to obtain local geometric features; and performing point-by-point feature processing on the local geometric features using a multilayer perceptron. Extract and max-pool the extracted point-by-point features to obtain global geometric features; perform 2D convolution on the RGB image in the current frame's RGB-D image to obtain RGB features; broadcast the global geometric features to the same spatial size as the RGB features and concatenate them with the RGB features to obtain concatenated features; apply channel attention weighting to the concatenated features to obtain RGB-D fusion features; perform multi-scale downsampling on the RGB-D fusion features to obtain downsampled features at multiple scales; concatenate the downsampled features at multiple scales and perform convolutional dimensionality reduction on the concatenated features to obtain the current frame's fusion features.

[0040] In practice, point cloud data is disordered and sparse, making it difficult to effectively capture local geometric structures by directly extracting features from the original point cloud. Therefore, this implementation constructs a local neighborhood graph through farthest point sampling and extracts local geometric features through graph convolution. Global geometric features are then obtained using a multilayer perceptron and max pooling. Finally, channel attention fusion and multi-scale fusion are performed between the global geometric features and RGB features to achieve high-quality RGB-D feature encoding. The specific implementation method is as follows: Bilateral filtering is applied to the depth image in the current frame's RGB-D image to obtain a denoised depth image. Bilateral filtering is an edge-preserving image filtering method; its filter kernel is jointly determined by a Gaussian function in the spatial domain and a Gaussian function in the pixel value domain, smoothing depth noise while preserving depth transitions at object edges. Furthermore, noise in the depth image may originate from measurement errors in the depth sensor itself or environmental interference. Depth hole filling is performed on the denoised depth image to obtain a preprocessed depth image. Depth hole filling can employ nearest-neighbor filling or interpolation methods to prevent the formation of holed point clouds due to missing depth values ​​during subsequent point cloudification. Further, the preprocessed depth image is point cloudified to obtain point cloud data. The point cloudification process involves converting two-dimensional depth pixels into points in three-dimensional space based on the pixel coordinates and depth values ​​of the depth image, combined with camera intrinsic parameters; each point has three-dimensional coordinates.

[0041] Further, the point cloud data is sampled from the farthest point to obtain a sampled point set. Farthest point sampling is a point cloud downsampling method. The process involves randomly selecting a point from the original point cloud as the initial sampling point, then iteratively selecting the point furthest from the selected point set until the number of sampling points reaches a preset value. Furthermore, farthest point sampling maintains the spatial uniformity of the point cloud coverage, resulting in a representative sampled point set. The preset number of sampling points can be determined based on the point cloud size and computing resources, for example, set to half the number of points in the original point cloud or a fixed value of 1024. Farthest point sampling avoids the problem of concentrated distribution of sampling points that may occur with random sampling.

[0042] Further, a local neighborhood graph is constructed based on the sampled point set, and graph convolution is performed on the local neighborhood graph to obtain local geometric features. The process of constructing the local neighborhood graph is as follows: for each sampled point, its k nearest neighbors are searched in the original point cloud, and the sampled point is connected to its nearest neighbors to form a graph structure. The value of k can be 16 or 32. Nodes in the graph structure are points, and edges are connections between the sampled point and its neighboring points. Graph convolution is a neural network operation that aggregates features on a graph. Further, for each node in the graph, graph convolution aggregates the node's features with the features of its neighboring nodes. The aggregation method can be mean aggregation, maximum aggregation, or attention aggregation. After graph convolution, the node's features are updated to include features containing local geometric structure information. Local geometric features reflect the surface shape around each sampled point in the point cloud and the relative positional relationships between points.

[0043] Furthermore, the local geometric features are extracted point-by-point using a multilayer perceptron, and the extracted point-by-point features are then max-pooled to obtain global geometric features. A multilayer perceptron is a feedforward neural network composed of multiple fully connected layers and activation functions. Point-by-point feature extraction involves performing a nonlinear transformation on each point in the local geometric features using the multilayer perceptron to obtain higher-dimensional features. Max-pooling further involves maximizing the features of all points channel-by-channel to obtain a global feature vector. Max-pooling is symmetric, meaning the output does not depend on the order of the input points, thus remaining invariant to the disorder of the point cloud. The global geometric features summarize the overall geometric shape information of the point cloud.

[0044] Further, a two-dimensional convolution is performed on the RGB image in the current frame's RGB-D image to obtain RGB features. Two-dimensional convolution is an operation that extracts local features by sliding a convolution kernel across the image. Multiple layers of two-dimensional convolution and pooling operations can extract edge, texture, and semantic information from the image. Further, the global geometric features are broadcast to the same spatial size as the RGB features and concatenated with them to obtain concatenated features. The broadcasting operation copies and expands the global geometric feature vector to the same width and height as the RGB features, aligning them spatially. The concatenation operation connects the broadcast global geometric features with the RGB features along the channel dimension to obtain the concatenated features. The concatenated features simultaneously contain local color information and global geometric information.

[0045] Furthermore, channel attention weighting is applied to the stitched features to obtain RGB-D fusion features. Channel attention weighting refers to learning the importance weight of each channel and weighting each channel of the stitched features according to this weight, highlighting feature channels that are more important for the pose estimation task and suppressing unimportant channels. Channel attention weights can be generated by a small neural network. Specifically, the stitched features are subjected to global average pooling to obtain a statistical vector of channel dimensions. This statistical vector is input into a two-layer fully connected network. The first layer performs dimensionality reduction, and the second layer restores the original dimensions. The attention weight of each channel is obtained through the sigmoid function. The sigmoid function is an activation function that maps the input to between 0 and 1. Each channel of the stitched features is multiplied by its corresponding weight to obtain the weighted features, i.e., the RGB-D fusion features. Channel attention enables the network to automatically learn the importance of each channel of color and geometric features, improving the adaptive ability of cross-modal fusion.

[0046] Furthermore, the RGB-D fusion features are downsampled at multiple scales to obtain downsampled features at multiple scales. Multi-scale downsampling refers to downsampling the feature map at different magnifications to obtain multiple feature maps with different resolutions. The downsampled features at multiple scales are concatenated, and the concatenated features are subjected to convolutional dimensionality reduction to obtain the current frame fusion features. The concatenation operation connects multiple features of different resolutions along the channel dimension. Before concatenation, the features at each scale need to be upsampled to a uniform resolution. Convolutional dimensionality reduction compresses the concatenated high-dimensional features into a feature representation of appropriate dimensions through 1x1 convolution. The current frame fusion features are used as the output of the shared feature encoding and are subsequently used by the initialization pose regression branch and the differentiable iterative nearest-point network layer.

[0047] This implementation effectively captures the local geometric structure information of point clouds through farthest point sampling and local neighborhood graph convolution, solving the problem of disordered original point clouds and the lack of explicit topological relationships. Multilayer perceptrons and max pooling ensure that global geometric features are invariant to point cloud permutations, enabling a stable representation of the overall shape of objects. Furthermore, channel attention weighting allows the model to automatically learn the importance of each channel of color and geometric features, improving the adaptive capability of cross-modal fusion. This encoding method has advantages in scenarios with complex geometric structures and missing textures, providing high-quality fused features for subsequent pose regression and optimization.

[0048] S3, initialize pose regression of the current frame fusion features to obtain the initial 6D pose.

[0049] In practice, the current frame's fused features are used to perform initial pose regression to obtain an initial 6D pose. Initial pose regression refers to using a lightweight regression branch to directly predict a coarse object pose based on the current frame's fused features; this coarse pose is called the initial 6D pose. Initial pose regression is computationally fast, but its prediction accuracy is usually insufficient to support fine-grained operations. Initial pose regression can be performed using either a voting-based approach or a direct regression approach. The voting-based approach first predicts the direction vector from each point to the object's keypoints, then aggregates the keypoint positions through voting, and finally solves for the pose based on the keypoint positions. The direct regression approach directly regresses the translation vector and rotation representation from the features. Furthermore, the initial 6D pose serves as the input to subsequent optimization branches, providing a good initial starting point for pose optimization. The significance of this initial starting point is that if a differentiable iterative nearest-point network layer is used for optimization without a good initial pose, the optimization process may require more iterations to converge and is prone to getting trapped in local optima. The initial pose regression branch provides an initial value close to the true pose with a small computational cost, which significantly reduces the optimization burden of the differentiable iterative nearest-point network layer.

[0050] In some preferred embodiments, the initial pose regression of the current frame fused features to obtain an initial 6D pose includes: performing semantic segmentation on the current frame fused features to obtain a foreground point set; predicting the direction vector for each point in the foreground point set to obtain the direction vector pointing from each point in the foreground point set to the object keypoint, and forming a direction vector set by the direction vectors pointing from each point in the foreground point set to the object keypoint; performing clustering voting on the direction vector set to obtain multiple keypoint positions; performing random sampling consensus filtering on the multiple keypoint positions to remove outlier keypoints and obtain valid keypoint positions; and obtaining the initial 6D pose by least squares fitting based on the valid keypoint positions and a pre-stored object model.

[0051] In practice, directly regressing the pose from the fused features of the entire frame is easily affected by background points and outliers, leading to a large deviation in the initial pose. Therefore, this implementation extracts a foreground point set through semantic segmentation, predicts direction vectors based on the foreground points, and obtains keypoints through cluster voting. Outlier keypoints are then removed using random sampling consensus filtering. Finally, least-squares fitting is performed between the valid keypoints and the object model to improve the robustness and accuracy of the initial pose. The specific implementation method is as follows: Semantic segmentation is performed on the fused features of the current frame to obtain a foreground point set. Semantic segmentation refers to predicting the semantic category (e.g., foreground or background) of each point or pixel in the feature map. Each point or pixel in the fused features of the current frame has a feature vector. A semantic segmentation prediction head outputs the probability that each point or pixel belongs to the foreground. Points or pixels with probabilities greater than a preset threshold are classified as foreground, resulting in the foreground point set. The foreground point set only contains feature points on the surface of the target object, reducing the interference of the background on the subsequent voting process. The semantic segmentation prediction head can be a lightweight convolutional network or a fully connected network.

[0052] Further, direction vector prediction is performed on each point in the foreground point set to obtain the direction vector pointing from each point in the foreground point set to a key point of the object. This direction vector set is formed by the direction vectors pointing from each point in the foreground point set to the key point of the object. Direction vector prediction is implemented through another prediction head network. The prediction head network receives the features of each point in the foreground point set and outputs a direction vector. This direction vector points from the foreground point to a key point in the object's coordinate system. Key points can be several predefined, identifiable points on the object's 3D model, such as corner points, edge endpoints, or center points. The number of key points can be set to 8 or 16. Each foreground point can predict the direction vector pointing to every key point, or it can predict only the direction vector pointing to the nearest key point. The direction vector set serves as the basis for subsequent voting.

[0053] Furthermore, cluster voting is performed on the set of direction vectors to obtain multiple keypoint locations. The cluster voting process is as follows: For each keypoint, based on the location of each foreground point and its predicted direction vector pointing to that keypoint, a ray can be generated in 3D space. Ideally, all rays pointing to the same keypoint should converge at that keypoint. In reality, due to prediction errors, the rays will not converge completely. Cluster voting selects the location with the highest voting density as the estimated keypoint location by statistically analyzing the distribution of these ray intersections. Cluster voting can employ mean shift or grid voting. Mean shift is an iterative clustering algorithm that finds the cluster center by continuously moving the search window to the location with the highest data point density. Grid voting divides the 3D space into a grid, counts the number of ray intersections within each grid, and the grid center with the most votes is the estimated keypoint location.

[0054] Furthermore, random sampling consensus filtering is applied to the multiple keypoint locations to eliminate outlier keypoints, resulting in valid keypoint locations. Random sampling consensus filtering is a robust model fitting method. The process involves randomly sampling a minimum sample set to fit model parameters, then counting the number of interior points consistent with the model across all data points, iterating multiple times to select the model with the most interior points. In keypoint location processing, the spatial relationships between multiple keypoint locations can be used for consistency checks. Specifically, several pairs of keypoints are randomly sampled from multiple keypoint locations. Based on the distance constraints of the corresponding keypoints in the object model, the sampled keypoint locations are checked to see if they satisfy the distance constraints. Keypoint locations that satisfy the constraints are interior points, and those that do not are exterior points. After multiple iterations, the model with the most interior points is selected, outlier keypoints are eliminated, and valid keypoint locations are obtained. This step effectively improves the reliability of keypoint locations.

[0055] Further, based on the effective keypoint positions and the pre-stored object model, the initial 6D pose is obtained through least-squares fitting. The pre-stored object model refers to an embodied intelligent physical model, which includes the keypoint positions and surface point cloud of the object in the object coordinate system. The object model can be obtained through computer-aided design or through 3D scanning reconstruction; this invention does not specifically limit this. Least-squares fitting solves for a rigid body transformation that minimizes the sum of squared distance errors between the transformed keypoints on the object model and the effective keypoint positions. Least-squares fitting can be solved using singular value decomposition or quaternion methods. Singular value decomposition is a matrix decomposition method commonly used to solve rigid body transformations. Least-squares fitting is a classic pose-finding method with high computational efficiency, and can obtain a high-precision initial pose when the effective keypoint positions are relatively accurate.

[0056] This implementation filters background interference through semantic segmentation, ensuring that direction vector prediction and voting are performed only on foreground points, reducing erroneous voting. Random sampling consensus filtering further eliminates outlier keypoints, improving the reliability of keypoint locations. Furthermore, least-squares fitting based on valid keypoints and the object model yields a relatively accurate initial 6D pose, providing a good starting point for subsequent optimization of the differentiable iterative nearest-point network layer. This initialization method is more robust in occlusion and complex background scenes.

[0057] Alternatively, in some preferred embodiments, the initial pose regression of the current frame fused features to obtain an initial 6D pose includes: performing global average pooling on the current frame fused features to obtain global features; inputting the global features into a translation regression layer to obtain a translation vector; inputting the global features into a rotation regression layer to obtain a rotation quaternion; normalizing the rotation quaternion to obtain a normalized rotation quaternion; and combining the translation vector and the normalized rotation quaternion to obtain the initial 6D pose.

[0058] In practice, when directly regressing 6D pose, the choice of rotation representation has a significant impact on regression accuracy and training stability. Therefore, this implementation obtains global features through global average pooling, predicts translation and rotation through independent regression layers, and normalizes the rotation quaternions to ensure the validity of the rotation representation, thereby improving the accuracy and stability of the initial pose regression. The specific implementation method is as follows: Global average pooling is performed on the fused features of the current frame to obtain global features. Global average pooling refers to calculating the average value of the feature map in its spatial dimension, compressing the high-dimensional feature map into a fixed-length feature vector. Furthermore, for a set of point features, global average pooling can be achieved by averaging the features of all points channel-by-channel. Global features contain comprehensive information from the entire input image, possess spatial invariance, and are suitable as input for pose regression. The dimensionality of the global features can be the same as the number of channels in the fused features of the current frame.

[0059] Furthermore, the global features are input into the translation regression layer to obtain the translation vector. The translation regression layer is a fully connected layer whose input is the global features, and its output dimension is 3, corresponding to the x-axis, y-axis, and z-axis translation components of the object in the camera coordinate system. A fully connected layer is a network layer that performs a fully connected linear transformation between the input and output features and then passes them through an activation function. The output of the translation regression layer is directly used as the translation vector, without requiring an additional activation function. The unit of the translation vector is typically meters or millimeters.

[0060] Furthermore, the global features are input into the rotation regression layer to obtain a rotation quaternion. The rotation regression layer is another fully connected layer with an output dimension of 4, corresponding to the four components of the rotation quaternion. A rotation quaternion is a mathematical tool for representing three-dimensional rotations, consisting of one real part and three imaginary parts. A quaternion is a mathematical tool for representing three-dimensional rotations, usually represented as q = w + xi + yj + zk. Here, w, x, y, and z are all real numbers; w is called the real part of the quaternion, and x, y, and z are called the imaginary parts. i, j, and k are the three imaginary units of the quaternion. The modulus of a quaternion is defined as the square root of w² + x² + y² + z². When a quaternion is used to represent rotations, its modulus is usually required to be 1, i.e., satisfying the normalization condition. Furthermore, compared to rotation matrices and Euler angles, quaternions do not have the gimbal lock problem and have fewer parameters. The gimbal lock problem refers to the loss of a rotational degree of freedom in certain poses when using Euler angles to represent rotation. Rotation regression layers and translation regression layers are independent, predicting translation and rotation separately, ensuring that their learning does not interfere with each other.

[0061] Furthermore, the rotated quaternion is normalized to obtain a normalized rotated quaternion. Normalization involves dividing the four components of the quaternion by its modulus, ensuring the normalized quaternion has a modulus of 1. The modulus of a quaternion is defined as the square root of w² + x² + y² + z². Only quaternions with a modulus of 1 are valid rotation representations. Normalization guarantees the validity of the rotated quaternion and avoids the problem of non-orthogonality of the rotation matrix caused by a quaternion modulus that is not 1. Normalization is a deterministic mathematical operation and does not involve learnable parameters.

[0062] Furthermore, the translation vector and the normalized rotation quaternion are combined to obtain the initial 6D pose. The combination method involves concatenating the translation vector and the normalized rotation quaternion into a 7-dimensional vector, where the first 3 dimensions are the translation components and the last 4 dimensions are the rotation quaternions. This 7-dimensional vector can represent the 6D pose of the object.

[0063] This implementation uses global average pooling to obtain compact global features, reducing the number of parameters and computational cost. Translation and rotation are regressed independently to avoid mutual interference between different geometric components. Furthermore, rotation quaternion normalization ensures the validity of the rotation representation, improving training stability and regression accuracy. This initialization method is simple in structure and computationally efficient, making it suitable for edge device scenarios with high real-time requirements.

[0064] S4, based on the initial 6D pose and the current frame fusion features, calculate the pose residual through a differentiable iterative nearest-neighbor network layer, and iteratively update the initial 6D pose based on the pose residual to obtain the accurate 6D pose; wherein, the differentiable iterative nearest-neighbor network layer is a differentiable neural network layer, the differentiable iterative nearest-neighbor network layer takes the initial 6D pose and the current frame fusion features as input, performs differentiable feature association on the current frame fusion features based on the initial 6D pose, and regresses and outputs the pose residual based on the differentiable feature association results.

[0065] In specific implementation, based on the initial 6D pose and the current frame fusion features, a differentiable iterative nearest-neighbor network layer is used to calculate the pose residual. The initial 6D pose is then iteratively updated based on the pose residual to obtain the accurate 6D pose. The differentiable iterative nearest-neighbor network layer is a differentiable neural network layer. Taking the initial 6D pose and the current frame fusion features as input, the differentiable iterative nearest-neighbor network layer performs differentiable feature association based on the initial 6D pose on the current frame fusion features, and regresses the pose residual based on the differentiable feature association results. The parameters of the differentiable iterative nearest-neighbor network layer are updated through backpropagation training.

[0066] Furthermore, the traditional iterative nearest-point algorithm minimizes the registration error between two point clouds by iteratively performing nearest-point search and rigid body transformation. Its basic process is as follows: given an initial pose, the source point cloud is transformed into the target point cloud coordinate system; the nearest point in the target point cloud is found for each point in the source point cloud; the optimal rigid body transformation is solved based on the corresponding point pairs; the pose is updated; and the above process is repeated until convergence. The traditional iterative nearest-point algorithm has high accuracy under ideal conditions, but nearest-point search is a discrete operation and not differentiable, making it impossible to embed into deep learning networks for end-to-end training. Therefore, the differentiable iterative nearest-point network layer in this embodiment transforms nearest-point association and residual regression into neural network operations, making them differentiable. Specifically, the differentiable iterative nearest-point network layer receives the initial 6D pose and the current frame fused features; it performs a spatial transformation on the point cloud corresponding to the current frame fused features based on the initial 6D pose; it performs differentiable feature association between the transformed point cloud and the object model point cloud to obtain the correspondence or soft-assigned weights between the point clouds; and then outputs the pose residual based on this correspondence through a lightweight regression network. Differentiable feature association refers to an association process entirely implemented by differentiable neural network operations. For example, similarity is calculated using feature inner products and soft-assigned weights are obtained through a normalized exponential function, rather than using non-differentiable hard nearest-point lookup. Further, the pose residual is the deviation between the current pose and the true pose, which can be represented as translation residual and rotation residual. Further, the translation residual is a three-dimensional vector representing the deviation between the current pose and the true pose in the x-axis, y-axis, and z-axis directions; further, the rotation residual represents the deviation between the current pose and the true pose in the orientation direction. The pose residual is applied to the initial 6D pose to obtain the updated pose. Further, the updated pose is input again into the differentiable iterative nearest-point network layer for a new round of residual calculation and update, obtaining the accurate 6D pose after a preset number of iterations. Further, since all operations of the differentiable iterative nearest-point network layer are differentiable, its network parameters and the network parameters of the shared encoder can be trained and updated using the backpropagation algorithm. Backpropagation refers to the process of passing gradients from the network output to the input layer by layer and updating parameters based on the gradients of the loss function with respect to each parameter, so that the entire network can be jointly optimized end-to-end.

[0067] Furthermore, the internal structure of the differentiable iterative nearest-neighbor network layer can be further explained. This network layer receives the fused features of the current frame and the initial 6D pose. First, the initial 6D pose is converted into a rigid body transformation matrix, which is used to transform the source point cloud corresponding to the fused features of the current frame to the coordinate system of the object model point cloud. Further, each point in the transformed source point cloud has a feature vector, and each point in the object model point cloud also has a pre-computed feature vector, which can be obtained through the same feature extraction network as the shared encoder, or pre-computed and stored before training. Further, feature similarity is calculated between each point in the transformed source point cloud and each point in the object model point cloud, using either the inner product of feature vectors or cosine similarity. The similarity is normalized using a normalized exponential function to obtain a soft-assigned weight matrix, where each element represents the correspondence strength between a point in the source point cloud and a point in the object model point cloud. Further, compared to hard nearest-neighbor lookup, the soft-assigned weight matrix is ​​not only differentiable but also capable of expressing uncertainty, enabling the network to learn more robust correspondences. Furthermore, based on the soft-assigned weight matrix, a weighted average coordinate is calculated for each point in the object model point cloud to obtain a soft corresponding point to each point in the source point cloud. The coordinate difference between this soft corresponding point and its corresponding point in the source point cloud forms a set of residual vectors. This set of residual vectors is input into a lightweight regression network, which can be a multilayer perceptron, and outputs translational and rotational residuals. The rotational residual can be represented by outputting a rotation vector or a quaternion increment. Further, the translational and rotational residuals are combined to form an incremental transformation, which is then applied to the current pose to obtain the updated pose. This process can be repeated multiple times.

[0068] Furthermore, the accurate 6D pose is the final estimation result of the current frame, which can be used for downstream tasks such as robot grasping and assembly. In this embodiment, shared feature encoding, initial pose regression, and a differentiable iterative nearest-point network layer constitute a complete network structure. Shared feature encoding provides the feature basis for the latter two branches, initial pose regression provides the starting point for the differentiable iterative nearest-point network layer, and the differentiable iterative nearest-point network layer iteratively refines the initial pose. This two-stage integrated design of initialization and optimization achieves a balance between accuracy and speed, while having a low overall network parameter count and computational overhead, making it suitable for deployment on edge devices.

[0069] This implementation provides a unified feature base for initializing pose regression and the differentiable iterative nearest-point network layer through shared feature encoding, enabling end-to-end network training. Furthermore, the initializing pose regression provides a good starting point for subsequent optimization, avoiding the problem of traditional iterative optimization requiring numerous iterations to converge. Moreover, the differentiable iterative nearest-point network layer achieves differentiability optimization through neural network implementation, allowing the optimization process to continuously learn during training and improve pose estimation accuracy. The entire network structure is compact and lightweight, enabling real-time processing on edge devices.

[0070] In some preferred embodiments, the step of calculating the pose residual through a differentiable iterative nearest-neighbor network layer based on the initial 6D pose and the fused features of the current frame, and iteratively updating the initial 6D pose based on the pose residual to obtain the accurate 6D pose, includes: determining whether the current frame is in tracking mode; if it is in tracking mode, obtaining the pose of the previous frame of the RGB-D image of the current frame, wherein the previous frame pose is the accurate 6D pose of the RGB-D image of the previous frame of the RGB-D image of the current frame, and converting the previous frame pose into... A prior transformation matrix is ​​applied to the current frame fusion features to obtain prior alignment features. The prior alignment features and the initial 6D pose are input into the differentiable iterative nearest-neighbor network layer to calculate the first pose residual. If it is not a tracking mode, the current frame fusion features and the initial 6D pose are input into the differentiable iterative nearest-neighbor network layer to calculate the second pose residual. The initial 6D pose is iteratively updated based on the first pose residual or the second pose residual to obtain the accurate 6D pose.

[0071] In practical implementation, in continuous video stream scenarios, the poses of objects between adjacent frames typically exhibit strong temporal continuity. Utilizing the pose information of the previous frame can improve the estimation accuracy and stability of the current frame under occlusion and noise conditions. This implementation, based on the initialization of pose regression and the differentiable iterative nearest-point network layer, further introduces tracking mode judgment. In tracking mode, the pose of the previous frame is used to perform prior alignment of the fused features of the current frame, improving the input quality of the differentiable iterative nearest-point network layer. In this implementation, the pose of the previous frame is the accurate 6D pose obtained by pose estimation of the RGB-D image of the previous frame of the current frame's RGB-D image. The specific implementation method is as follows: The system determines whether the current frame is in tracking mode. Tracking mode refers to a continuous tracking state where the system has obtained the precise 6D pose of the previous frame and can use that pose as prior information. The determination can be based on whether the system currently has a usable pose from the previous frame, or it can be specified by an external command to enter tracking mode. If the system is processing the first frame of the video sequence, or the pose from the previous frame is unavailable, or pose tracking has been lost and needs to be reinitialized, then the current frame does not belong to tracking mode. This determination is completed before the execution of the differentiable iterative nearest-point network layer to decide whether to incorporate the pose from the previous frame into the computation process.

[0072] Furthermore, if the current frame is in tracking mode, the pose of the previous frame of the RGB-D image of the current frame is obtained. The pose of the previous frame is the precise 6D pose obtained after processing the RGB-D image of the previous frame using this method, and it is stored in the system cache. The cache can be a variable in memory, a register, or a file storage unit, used to transfer pose information between adjacent frames. The pose of the previous frame can be obtained by reading it from the cache at the beginning of the current frame processing, or it can be read on demand after determining that tracking mode has been entered.

[0073] Further, after obtaining the pose of the previous frame, the pose is converted into a priori transformation matrix. The pose can be represented as a rotation matrix plus a translation vector, or as a rotation quaternion plus a translation vector. The priori transformation matrix is ​​a 4x4 rigid body transformation matrix, where the upper left 3x3 submatrix is ​​the rotation matrix, the upper right 3x1 submatrix is ​​the translation vector, and the last row is fixed at 0, 0, 0, 1. Converting the pose into matrix form facilitates spatial transformation of the features. If the pose of the previous frame is represented as a rotation quaternion plus a translation vector, the rotation quaternion is first converted into a rotation matrix, and then the priori transformation matrix is ​​constructed. The conversion from rotation quaternion to rotation matrix is ​​a common mathematical operation in rigid body transformation representation; the specific conversion relationship can be determined based on the components of the quaternion, which is not specifically limited in this invention.

[0074] Further, the prior transformation matrix is ​​applied to the current frame fusion feature to obtain the prior aligned feature. The point cloud or feature map in the current frame fusion feature contains spatial location information. The specific way to apply the prior transformation matrix to the current frame fusion feature is as follows: for each 3D spatial point corresponding to the current frame fusion feature, the 3D coordinates of the point are multiplied on the left by the prior transformation matrix to obtain the transformed 3D coordinates. After the transformation, the object point cloud in the current frame feature is aligned to the spatial region near the object pose of the previous frame. The prior aligned feature is the transformed feature representation. Since the motion amplitude of the object between adjacent frames is usually small, after prior alignment, the deviation between the current frame feature and the true pose of the object is reduced, which reduces the optimization difficulty of the differentiable iterative nearest-point network layer. The feature vector of each point in the prior aligned feature remains unchanged, only the spatial coordinates change, so color and geometric information are not lost.

[0075] Furthermore, the prior alignment features and the initial 6D pose are input into the differentiable iterative nearest-neighbor network layer to calculate the first pose residual. The first pose residual is calculated based on the prior aligned features. Since the prior alignment features are already close to the true pose of the object, the differentiable iterative nearest-neighbor network layer only needs to calculate a small residual to achieve high accuracy. The first pose residual also includes both translation and rotation residuals.

[0076] Furthermore, if the current frame is not in tracking mode, the fused features of the current frame and the initial 6D pose are input into the differentiable iterative nearest-point network layer to calculate the second pose residual. The second pose residual does not utilize prior information from the previous frame's pose; it is calculated solely based on the fused features of the current frame and the initial 6D pose. This branch is suitable for re-initialization scenarios after the first frame or pose tracking loss. The second pose residual also includes translational and rotational residuals.

[0077] Further, the initial 6D pose is iteratively updated based on the first pose residual or the second pose residual to obtain the accurate 6D pose. The iterative update process is as follows: the first pose residual or the second pose residual is applied to the initial 6D pose to obtain the pose after the first update; the pose after the first update is input again into the differentiable iterative nearest neighbor network layer to calculate a new pose residual and continue updating until a preset number of iterations is reached or the residual is less than a preset threshold. The preset number of iterations can be set according to accuracy and speed requirements, for example, 3 or 5 times. The preset threshold can be set to a small value, for example, the translation residual is less than 1 mm and the rotation residual is less than 0.5 degrees. The final pose obtained is the accurate 6D pose.

[0078] This implementation effectively utilizes inter-frame temporal consistency in continuous tracking scenarios by determining the tracking mode and aligning with the prior pose of the previous frame. This significantly improves the estimation accuracy and stability of the current frame when objects are occluded, the image contains noise, or the pose changes significantly. Furthermore, the prior alignment feature allows the differentiable iterative nearest-point network layer to be computed in a better initial state, reducing the convergence difficulty of iterative optimization and improving computational efficiency. Moreover, it maintains independent pose estimation capability in non-tracking mode, without affecting single-frame initialization. This design enables the method to maintain high accuracy and robustness in different operating modes.

[0079] In some preferred embodiments, the method further includes: acquiring a training dataset, the training dataset containing multiple sample RGB-D images and ground truth poses corresponding to each sample RGB-D image; encoding shared features for each sample RGB-D image to obtain sample fusion features; performing initial pose regression on the sample fusion features to obtain initial 6D poses of the samples; calculating sample pose residuals through the differentiable iterative nearest neighbor network layer based on the initial 6D poses of the samples and the sample fusion features, and iteratively updating the initial 6D poses of the samples based on the sample pose residuals to obtain accurate 6D poses of the samples; The pose loss is calculated based on the initial 6D pose of the sample and the ground truth pose, and the point-to-point distance loss is calculated based on the accurate 6D pose of the sample and the ground truth pose. A course-learning strategy is adopted: first, the network parameters used to perform the initial pose regression are trained only based on the pose loss; then, the network parameters used to perform the initial pose regression and the differentiable iterative nearest-point network layer are trained jointly based on the pose loss and the point-to-point distance loss, resulting in a trained integrated network parameter set. The trained integrated network parameter set is converted into an edge inference format and INT8 quantized to obtain the real-time inference model for edge devices.

[0080] In practice, the integrated network consisting of the initialization branch and the differentiable iterative nearest-point network layer requires a suitable training strategy to converge to a better state. This implementation uses a course-learning strategy to first train the initialization pose regression branch, then jointly train the initialization pose regression and the differentiable iterative nearest-point network layer, and uses pose loss and point-to-point distance loss for supervision. Finally, the trained model is converted to an edge inference format and deployed using INT8 quantization. The specific implementation method is as follows: A training dataset is obtained, comprising multiple sample RGB-D images and the ground truth pose corresponding to each sample RGB-D image. The training dataset can be a publicly available 6D pose estimation dataset, such as the YCB-Video dataset. The YCB-Video dataset is a publicly available dataset containing RGB-D video sequences of several common objects and their annotated ground truth poses. Further, the ground truth poses are the true 6D poses of the objects obtained through high-precision measurement or manual annotation, serving as training supervision signals. The training dataset can also include synthetic data, such as synthetic images with ground truth pose annotations generated by a rendering engine. Synthetic data can expand the diversity of the training set and improve the model's generalization ability.

[0081] Further, shared feature encoding is performed on each sample RGB-D image to obtain sample fusion features. The shared feature encoding process is consistent with the inference stage of this method, that is, features are extracted from the color image and depth image in the sample RGB-D image and fused to obtain sample fusion features. Further, the sample fusion features are used for initial pose regression to obtain the initial 6D pose of the sample. The initial pose regression process is consistent with the inference stage of this method. Based on the initial 6D pose of the sample and the sample fusion features, the sample pose residual is calculated through the differentiable iterative nearest-point network layer, and the initial 6D pose of the sample is iteratively updated based on the sample pose residual to obtain the accurate 6D pose of the sample. The above forward propagation process is consistent with the inference process, the difference being that the loss needs to be calculated simultaneously during training. Forward propagation refers to the process of data sequentially passing through each network layer from the network input layer to obtain the output.

[0082] Further, a pose loss is calculated based on the initial 6D pose of the sample and the ground truth pose, and a point-to-point distance loss is calculated based on the accurate 6D pose of the sample and the ground truth pose. The pose loss is used to supervise the output of the initial pose regression branch, measuring the deviation between the initial 6D pose of the sample and the ground truth pose. The pose loss can include translation loss and rotation loss. Further, the translation loss uses the L1 or L2 distance of the translation vector, where the L1 distance is the sum of the absolute differences of each component, and the L2 distance is the square root of the sum of the squares of the differences of each component. The rotation loss can use the angular difference between two rotation quaternions. The point-to-point distance loss is used to supervise the output of the differentiable iterative nearest-point network layer, measuring the average distance between the object model point cloud under the accurate 6D pose of the sample and the object model point cloud under the ground truth pose. The point-to-point distance loss is calculated by transforming the object model point cloud according to both the accurate 6D pose of the sample and the ground truth pose, and then calculating the average Euclidean distance between corresponding points. Point-to-point distance loss directly reflects the impact of pose estimation results on the registration accuracy of object surfaces, and is more consistent with the final pose accuracy index.

[0083] Furthermore, a course learning strategy is adopted. First, the network parameters used to perform the initial pose regression are trained only based on the pose loss. Then, the network parameters used to perform the initial pose regression and the differentiable iterative nearest point network layer are trained jointly based on the pose loss and the point-to-point distance loss, resulting in a unified network parameter set. Course learning is a training strategy that progresses from easy to difficult. In the first stage, only the initial pose regression branch and the shared encoder are trained using the pose loss; the differentiable iterative nearest point network layer is not involved in the training. This stage enables the initial pose regression branch to output a reasonable initial pose, providing a better input for the differentiable iterative nearest point network layer. Further, in the second stage, the entire network, including the shared encoder, the initial pose regression branch, and the differentiable iterative nearest point network layer, are trained jointly using the pose loss and the point-to-point distance loss. The gradient of the point-to-point distance loss is propagated to the differentiable iterative nearest point network layer and the shared encoder, allowing the shared feature encoding to adapt to the needs of the optimization branch. The gradient of the pose loss continues to propagate to the initial pose regression branch and the shared encoder, maintaining the reasonableness of the initial pose. The course learning strategy avoids the problem that the optimization branch is difficult to converge in the early stage of training due to poor initial pose of the input, thereby improving training stability and final model accuracy.

[0084] Furthermore, the trained integrated network parameters are converted into an edge inference format and INT8 quantized to obtain a real-time inference model for edge devices. The edge inference format can be an inference format optimized for edge devices, such as TensorRT. TensorRT is a high-performance deep learning inference optimizer and runtime library capable of performing layer fusion, accuracy calibration, and automatic kernel tuning on the trained network to generate an inference engine optimized for a specific graphics processing unit. INT8 quantization converts model parameters from 32-bit floating-point numbers to 8-bit integer representations to compress model size and accelerate inference. The quantized model is deployed on edge devices, such as platforms like Jetson, enabling real-time 6D pose estimation and tracking. Jetson is an embedded artificial intelligence computing platform with a graphics processing unit and a central processing unit, suitable for deploying machine vision models.

[0085] This implementation employs a course-based learning strategy, training the initialization branch first and then jointly training the entire network. This avoids the problem of the optimization branch failing to converge due to poor initial pose input in the early stages of training, thus improving training stability and final model accuracy. Pose loss and point-to-point distance loss supervise the initialization and optimization branches respectively, ensuring each branch performs its specific function. Furthermore, after training, through inference format conversion and INT8 quantization, the model is deployed to edge devices while maintaining accuracy that meets the ADD-S error requirement of less than or equal to 5mm, achieving a real-time inference speed of over 30FPS. ADD-S error is an evaluation metric for 6D pose estimation accuracy, calculated as the average distance between corresponding points in the object model's point cloud under the predicted and true poses. An ADD-S error of less than or equal to 5mm indicates high pose estimation accuracy. FPS stands for frames per second, representing the number of image frames processed per second. This training and deployment process has high practical value.

[0086] In some preferred embodiments, the step of converting the trained integrated network parameters into an edge inference format and performing INT8 quantization to obtain a real-time inference model for the edge device includes: acquiring a calibration dataset containing multiple RGB-D images; inputting each RGB-D image from the calibration dataset into an inference network loaded with the trained integrated network parameters, and statistically analyzing the activation value range of each network layer in the inference network; determining the quantization scale corresponding to each network layer in the inference network based on the activation value range of each network layer in the inference network; performing layer-by-layer INT8 quantization on the trained integrated network parameters based on the quantization scale corresponding to each network layer in the inference network to obtain quantized network parameters; converting the quantized network parameters into an edge inference format, and encapsulating the converted quantized network parameters and the quantization scale corresponding to each network layer in the inference network to obtain the real-time inference model for the edge device.

[0087] In practice, directly performing INT8 quantization may lead to excessive quantization errors due to not considering the actual distribution of activation values ​​in each network layer, thus affecting model accuracy. Therefore, this implementation method uses a calibration dataset to statistically analyze the activation value range of each network layer, thereby determining the quantization scale for each layer. INT8 quantization is then performed layer by layer. Finally, the quantization parameters and the quantized network parameters are encapsulated to obtain a high-precision real-time inference model for edge devices. The specific implementation method is as follows: A calibration dataset is obtained, which contains multiple RGB-D images. The calibration dataset is used to statistically analyze the numerical distribution of activation values ​​in each layer of the trained unibody network. The images in the calibration dataset can be a subset sampled from the training dataset or an independent dataset with a similar distribution to the training data. Furthermore, the calibration dataset does not need to contain ground truth pose annotations; only the input images are required, because the calibration process only performs forward propagation to statistically analyze activation values ​​and does not calculate loss. The calibration dataset typically contains tens to hundreds of images to ensure the representativeness of the statistical results.

[0088] Further, each RGB-D image from the calibration dataset is input into the inference network loaded with the trained unibody network parameters, and the activation value range of each network layer in the inference network is statistically analyzed. The inference network refers to the network structure of the trained unibody network in inference mode, where backpropagation and gradient calculation are not used. Further, each image from the calibration dataset is sequentially input into the inference network, and the activation value generated by each image in each network layer of the inference network is recorded. The activation value refers to the feature map value output by the network layer during forward propagation. Further, the activation value range of each network layer can be statistically analyzed by iterating through all activation values ​​to obtain the minimum and maximum activation values, or by using statistical distribution methods to determine a range covering most activation values, such as selecting the quantile of the distribution as the range boundary. The activation value range reflects the numerical fluctuation of each network layer under different inputs. Further, for some network layers, the activation value range may be very large, containing a few extreme values. In this case, a truncation method can be used, such as selecting the 99th quantile as the effective range, to balance quantization accuracy and representation range.

[0089] Further, based on the activation value range of each network layer in the inference network, the quantization scale corresponding to each network layer in the inference network is determined. The quantization scale refers to the scaling factor used when mapping floating-point activation values ​​to 8-bit integers. For each network layer, the quantization scale is calculated based on the activation value range width of that layer and the representation range of 8-bit integers. The representation range of 8-bit integers is -128 to +127, with a width of 255. The quantization scale can be obtained by dividing the activation value range width by 255. Further, the quantization zero point can be calculated based on the minimum value of the activation value range and the quantization scale, and is used to map the floating-point zero value to the corresponding position in the integer field. The quantization scale and quantization zero point together constitute the quantization parameters of this network layer.

[0090] Furthermore, based on the quantization scale corresponding to each network layer in the inference network, the trained integrated network parameters are subjected to layer-by-layer INT8 quantization to obtain quantized network parameters. Layer-by-layer INT8 quantization refers to affine quantization of the weight parameters of each network layer according to the quantization scale corresponding to that layer, converting floating-point weights into 8-bit integers. Further, the quantization scale of the weight parameters can be determined independently or shared with the activation values. Each floating-point weight value is subtracted from its quantization zero, divided by the quantization scale, and the result is rounded to obtain an 8-bit integer representation. The quantized network parameters are represented by 8-bit integers, and the model size is approximately one-quarter of the original floating-point model. The original floating-point parameters are typically stored using 32-bit floating-point numbers; after INT8 quantization, they are stored as 8-bit integers, resulting in a storage space compression ratio of 4:1.

[0091] Further, the quantized network parameters are converted into an edge inference format, and the converted quantized network parameters and the quantization scales corresponding to each network layer in the inference network are encapsulated to obtain the real-time inference model for the edge device. Further, the edge inference format refers to a model serialization format optimized for the target edge device, such as the TensorRT engine format or the ONNX format. ONNX is an open neural network exchange format used to exchange models between different deep learning frameworks. Further, the format conversion operation organizes the quantized network parameters according to the requirements of the inference framework, generating an executable inference model file. The encapsulation operation packages the quantized network parameters together with the quantization scales of each layer, enabling the deployed inference model to correctly perform quantization and dequantization operations during inference. Further, during inference, after receiving input data, the network layer quantizes the input activation values ​​into 8-bit integers according to the final activation value quantization scale and quantization zero point, performs integer operations, and then dequantizes the result into a floating-point number for output. The data type throughout the inference process is primarily integer, significantly improving computational efficiency.

[0092] This implementation method uses a calibration dataset to statistically analyze the activation value range of each network layer and determines the layer-by-layer quantization scale accordingly. This avoids the problem of excessive quantization errors in some layers that may result from a uniform quantization scale, significantly reducing the accuracy loss caused by INT8 quantization. Furthermore, layer-by-layer quantization allows each layer to fully utilize the representation range of 8-bit integers, improving quantization resolution. The quantized network parameters are converted into an edge inference format and the quantization scale is encapsulated, enabling the model to perform quantized inference directly after deployment without additional calibration. Furthermore, this deployment method achieves model volume compression and inference acceleration while maintaining model accuracy to meet the ADD-S error requirement of less than or equal to 5mm, enabling real-time 6D pose estimation and tracking on resource-constrained edge devices.

[0093] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0094] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0095] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0096] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to perform an embodied intelligent lightweight 6D pose estimation and tracking method.

[0097] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0098] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an embodied intelligent lightweight 6D pose estimation and tracking method.

[0099] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0100] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of an embodied intelligent lightweight 6D pose estimation and tracking method provided in any of the above method embodiments.

[0101] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0102] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0103] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of the embodied intelligent lightweight 6D pose estimation and tracking method provided in any of the above method embodiments.

[0104] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0106] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0107] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0109] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0110] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A lightweight, embodied intelligent 6D pose estimation and tracking method, characterized in that, include: Get the RGB-D image of the current frame; Shared feature encoding is performed on the current frame RGB-D image to obtain the current frame fusion feature; Initialize the pose regression of the current frame fusion features to obtain the initial 6D pose; Based on the initial 6D pose and the current frame fusion features, the pose residual is calculated through a differentiable iterative nearest-point network layer, and the initial 6D pose is iteratively updated based on the pose residual to obtain the accurate 6D pose. The differentiable iterative nearest point network layer is a differentiable neural network layer. The differentiable iterative nearest point network layer takes the initial 6D pose and the current frame fusion features as input, performs differentiable feature association on the current frame fusion features based on the initial 6D pose, and regresses and outputs the pose residual based on the differentiable feature association results.

2. The embodied intelligent lightweight 6D pose estimation and tracking method according to claim 1, characterized in that, The step of calculating the pose residual through a differentiable iterative nearest-neighbor network layer based on the initial 6D pose and the fused features of the current frame, and iteratively updating the initial 6D pose based on the pose residual to obtain the accurate 6D pose, includes: Determine if the current frame is in tracking mode; If it is in tracking mode, the pose of the previous frame of the current frame RGB-D image is obtained. The pose of the previous frame is the precise 6D pose of the previous frame RGB-D image of the current frame RGB-D image. The pose of the previous frame is converted into a prior transformation matrix. The prior transformation matrix is ​​applied to the current frame fusion feature to obtain the prior alignment feature. The prior alignment feature and the initial 6D pose are input into the differentiable iterative nearest point network layer to calculate the first pose residual. If it is not in tracking mode, the current frame fusion features and the initial 6D pose are input into the differentiable iterative nearest point network layer to calculate the second pose residual; The initial 6D pose is iteratively updated based on the first pose residual or the second pose residual to obtain the accurate 6D pose.

3. The embodied intelligent lightweight 6D pose estimation and tracking method according to claim 1, characterized in that, The step of encoding shared features in the current frame RGB-D image to obtain the current frame fused features includes: Bilateral filtering is performed on the depth image in the current frame RGB-D image to obtain a denoised depth image; The denoised depth image is filled with depth holes to obtain a preprocessed depth image; The preprocessed depth image is converted into point cloud data. Perform a two-dimensional convolution on the RGB image in the current frame's RGB-D image to obtain RGB features; Perform 3D sparse convolution on the point cloud data to obtain geometric features; The geometric features are upsampled so that the upsampled geometric features have the same spatial resolution as the RGB features, thus obtaining aligned geometric features; Attention-weighted fusion is performed on the RGB features and the alignment geometric features to obtain point-by-point fusion weights, and the RGB features and the alignment geometric features are weighted and summed according to the point-by-point fusion weights to obtain RGB-D fusion features; The RGB-D fusion features are downsampled at multiple scales to obtain downsampled features at multiple scales; The downsampled features from the multiple scales are concatenated, and the concatenated features are then subjected to convolutional dimensionality reduction to obtain the fused features of the current frame.

4. The embodied intelligent lightweight 6D pose estimation and tracking method according to claim 1, characterized in that, The step of encoding shared features in the current frame RGB-D image to obtain the current frame fused features includes: Bilateral filtering is performed on the depth image in the current frame RGB-D image to obtain a denoised depth image; The denoised depth image is filled with depth holes to obtain a preprocessed depth image; The preprocessed depth image is converted into point cloud data. The point cloud data is sampled from the farthest point to obtain a set of sampled points; A local neighborhood graph is constructed based on the set of sampling points, and graph convolution is performed on the local neighborhood graph to obtain local geometric features; The local geometric features are extracted point by point using a multilayer perceptron, and the extracted point by point features are max pooled to obtain the global geometric features. Perform a two-dimensional convolution on the RGB image in the current frame's RGB-D image to obtain RGB features; The global geometric features are broadcast to the same spatial size as the RGB features and then concatenated with the RGB features to obtain the concatenated features; Channel attention weighting is applied to the spliced ​​features to obtain RGB-D fusion features; The RGB-D fusion features are downsampled at multiple scales to obtain downsampled features at multiple scales; The downsampled features from the multiple scales are concatenated, and the concatenated features are then subjected to convolutional dimensionality reduction to obtain the fused features of the current frame.

5. The embodied intelligent lightweight 6D pose estimation and tracking method according to claim 1, characterized in that, The initial pose regression of the fused features of the current frame to obtain the initial 6D pose includes: Semantic segmentation is performed on the fused features of the current frame to obtain a foreground point set; For each point in the foreground point set, a direction vector prediction is performed to obtain the direction vector of each point in the foreground point set pointing to the key point of the object. The direction vector set is composed of the direction vectors of each point in the foreground point set pointing to the key point of the object. Clustering and voting are performed on the set of direction vectors to obtain the locations of multiple key points; Random sampling and consistency filtering are performed on the multiple key point locations to remove outlier key points and obtain the effective key point locations. Based on the effective key point locations and the pre-stored object model, the initial 6D pose is obtained through least squares fitting.

6. The embodied intelligent lightweight 6D pose estimation and tracking method according to claim 1, characterized in that, The initial pose regression of the fused features of the current frame to obtain the initial 6D pose includes: Global average pooling is performed on the fused features of the current frame to obtain global features; The global features are input into the translation regression layer to obtain the translation vector; The global features are input into the rotational regression layer to obtain rotational quaternions; The rotated quaternion is normalized to obtain a normalized rotated quaternion; The initial 6D pose is obtained by combining the translation vector and the normalized rotation quaternion.

7. The embodied intelligent lightweight 6D pose estimation and tracking method according to claim 1, characterized in that, The method further includes: Obtain a training dataset, which contains multiple sample RGB-D images and ground truth poses corresponding to each sample RGB-D image; Shared feature encoding is performed on each sample RGB-D image to obtain sample fusion features; The sample fusion features are initialized with pose regression to obtain the initial 6D pose of the sample; Based on the initial 6D pose of the sample and the fusion features of the sample, the sample pose residual is calculated through the differentiable iterative nearest point network layer, and the initial 6D pose of the sample is iteratively updated based on the sample pose residual to obtain the accurate 6D pose of the sample. The pose loss is calculated based on the initial 6D pose of the sample and the ground truth pose, and the point-to-point distance loss is calculated based on the accurate 6D pose of the sample and the ground truth pose. A course learning strategy is adopted. First, the network parameters used to perform the initial pose regression are trained only based on the pose loss. Then, the network parameters used to perform the initial pose regression and the differentiable iterative nearest point network layer are trained together based on the pose loss and the point-to-point distance loss to obtain the integrated network parameters after training. The trained integrated network parameters are converted into edge inference format and quantized using INT8 to obtain a real-time inference model for edge devices.

8. The embodied intelligent lightweight 6D pose estimation and tracking method according to claim 7, characterized in that, The step of converting the trained integrated network parameters into an edge inference format and performing INT8 quantization to obtain a real-time inference model for edge devices includes: Obtain a calibration dataset, which contains multiple RGB-D images; Each RGB-D image in the calibration dataset is input into an inference network loaded with the trained integrated network parameters, and the activation value range of each network layer in the inference network is calculated. Based on the activation value range of each network layer in the inference network, determine the quantization scale corresponding to each network layer in the inference network; Based on the quantization scale corresponding to each network layer in the inference network, the parameters of the trained integrated network are quantized layer by layer using INT8 to obtain the quantized network parameters. The quantized network parameters are converted into an edge inference format, and the converted quantized network parameters and the quantization scale corresponding to each network layer in the inference network are encapsulated to obtain the real-time inference model of the edge device.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images