A six-degree-of-freedom object pose estimation method and device based on a visual language model and conditioned on reference observations, and a storage medium

Abstract

The application discloses a six-degree-of-freedom object pose estimation method and device based on a visual language model and conditioned on reference observations and a storage medium. The method acquires a reference set containing one or more reference RGB-D observations and an unlabeled complete query RGB-D observation, and the first reference is a pose reference with a known pose. A multi-modal prompt is constructed, a standard coordinate axis is superimposed on the first reference, and an object center depth and a camera internal parameter are provided. A shared multi-modal encoding model is used for joint encoding, a rotation is output through a rotation decoding branch, a three-dimensional translation is output through a translation decoding branch, and a six-degree-of-freedom pose is combined. The application does not require a query bounding box, a mask and a CAD model, and the same model can uniformly process single-reference, multi-reference and tracking tasks.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, robot perception and spatial localization, and specifically to a method for estimating the six-degree-of-freedom object pose (6DoF pose) of a target object in a color-depth image (hereinafter referred to as a query RGB-D image) using a shared multimodal coding model, with a reference observation of known pose, optional additional reference observation or a historical frame with pose as conditions, as well as an apparatus, electronic device and computer-readable storage medium for implementing the method. Background Technology

[0002] Six-DOF object pose estimation is used to determine the 3D rotation and 3D translation of a target object relative to the camera coordinate system. Translation represents the position of the origin of the object's canonical coordinate system in the camera coordinate system; this origin is simply referred to as the object center. The pose has actual physical scale, with translation and depth expressed in physical units such as millimeters. Six-DOF object pose estimation is a fundamental capability in systems such as robot grasping and assembly, augmented reality, industrial inspection, warehouse sorting, and target tracking. For new objects appearing during the testing phase, it is typically necessary to provide the object's appearance and coordinate system information from a Computer-Aided Design (CAD) model, several reference observations, or historical video frames, and then reconstruct the object's pose in Color-Depth (RGB-D).

[0003] Existing methods are typically broken down into multiple independent technical approaches based on the reference evidence and task format. Computer-aided design model-based methods estimate pose through offline rendering, retrieval, and iterative alignment; multi-reference view-based methods obtain object representations through feature matching, point cloud registration, or reconstruction; single-reference methods estimate poses through segmentation, dense correspondence, coordinate graph prediction, and geometric solvers; while video tracking methods rely on pose propagation, filtering, mapping, or online reconstruction. Different approaches often use different input organizations, network structures, training data, and inference programs.

[0004] Visual-language models (VLMs) possess multi-image understanding, open-vocabulary recognition, and multi-step reasoning capabilities, and have been used in recent years for spatial relationship judgment, object recognition, candidate pose selection, and ambiguity interpretation. However, in most related schemes, visual-language models mainly play a semantic auxiliary role, and the final continuous six-DOF pose is still generated by a dedicated correspondence network, regression network, or geometric optimizer.

[0005] The existing technology has at least the following shortcomings:

[0006] First, there is a lack of a unified processing mechanism for different tasks and forms of reference information. Single-reference, multi-reference, and video tracking are treated as different tasks, requiring switching of models, input interfaces, or inference processes, making it difficult for the same model to work flexibly based on currently available evidence.

[0007] Second, it relies heavily on object models, reference data, and dedicated intermediate processing modules. Some solutions depend on computer-aided design models, offline rendering, target segmentation masks, querying bounding boxes of target objects in images, point cloud registration, online reconstruction, or iterative refinement, which increases deployment costs and limits applicability to new objects.

[0008] Third, there is the problem of adapting continuous rotations to multiple effective supervised targets. Rotations lie in the (Special Orthogonal Group, SO(3)) manifold; geometric tolerances and object symmetries allow the same target rotation to correspond to multiple effective coarse rotation supervised targets. Directly serializing these targets into text will produce competing symbol sequences, while a single shared residual cannot simultaneously adapt to representative rotations of different categories.

[0009] Fourth, the center depth error is significant. The RGB-D depth map provides the depth of visible surface points, while pose translation requires the object's center depth. The signed offset between the two is related to the visible surface, object orientation, and geometry, making direct regression particularly difficult under occlusion conditions.

[0010] Fifth, the tracking system modules are redundant and prone to accumulating errors. Existing tracking schemes typically require separate detection, filtering, mapping, reconstruction, or tracking heads; static estimation models are difficult to reuse directly, and historical evidence cannot be naturally extended to the same reference set. Summary of the Invention

[0011] The core technical problem to be solved by this invention is to establish a unified input and decoding mechanism based on reference observations, without requiring the bounding box of the target object in the query image, the segmentation mask of the target object in the query image, or the computer-aided design model. This mechanism enables the same shared multimodal coding model to directly output the metric six-degree-of-freedom pose of the target object in the query RGB-D frame based on a reference observation with known pose, one or more additional references, or historical frames with poses, and improves adaptability to symmetry, occlusion, and large viewpoint changes.

[0012] To achieve the above objectives, the present invention provides the following technical solution:

[0013] A six-DOF object pose estimation method based on a visual language model and conditioned on reference observations includes:

[0014] Obtain a reference set and a query observation. The reference set contains one or more reference RGB-D observations, and the query observation is a complete query RGB-D observation without labeling. The first reference in the reference set is a pose-known reference, and the remaining references are supplementary observations without pose labeling or historical observations with previously predicted poses.

[0015] Construct a multimodal cue for pose estimation by organizing each RGB-D observation in the reference set into an ordered image pair of an RGB image and a pseudo-color depth map encoded with the same fixed absolute depth scale. Superimpose the canonical coordinate axis markers determined by the known pose on the first reference RGB image and provide the object center depth and camera intrinsics of the first reference to the cue.

[0016] The reference set and the query observation are jointly encoded using a shared multimodal coding model to obtain a shared hidden representation;

[0017] From the shared hidden representation, the rotation of the target object in the query observation is output through the rotation decoding branch, and the three-dimensional translation of the target object is output through the translation decoding branch;

[0018] The rotation and the three-dimensional translation are combined to form a six-degree-of-freedom pose in the coordinate system of the query observation camera.

[0019] Furthermore, the rotation decoding branch is configured as follows:

[0020] SO(3) is discretized into K coarse-grained rotation categories by a fixed set of representative rotations, which are obtained by sampling the farthest point from the candidate rotation set represented by unit quaternions according to the geodesic distance of SO(3);

[0021] Based on the hidden state corresponding to the rotation-specific terminology, the probability of each of the K coarse-grained rotation categories is predicted, as well as the local rotation residual represented by a continuous six-dimensional rotation representation conditioned on each coarse-grained rotation category. The continuous six-dimensional rotation representation is mapped to SO(3) by an orthogonalization operator.

[0022] During inference, the coarse-grained rotation category with the highest probability and its corresponding local rotation residual are selected. The representative rotation of the selected category is multiplied by the orthogonalized local rotation residual to obtain the rotation of the target object.

[0023] Furthermore, during training, for each rotation category b, the rotations represented by that category are... To the closest symmetric equivalent truth value distance Determine the category with the smallest distance and with Construct thresholds based on geometric tolerances The effective coarse rotation target set is determined by the object symmetry transformation set S. ,in The coarse-grained classification loss increases the sum of prediction probabilities for each category within the effective coarse rotation target set by minimizing the negative log-likelihood, while the local rotation residual loss supervises the conditional residual for each rotation category within the effective coarse rotation target set.

[0024] Furthermore, for each rotation category b in the effective coarse rotation target set, the symmetric equivalent target closest to the rotation representing that category is selected. ,in And construct the residual target corresponding to this rotation category. Different effective rotation categories receive different residual targets, and each selected effective rotation category has its corresponding local rotation correction.

[0025] Furthermore, the translation decoding branch is configured to execute a depth-axis thought chain based on geometric information, including:

[0026] In a full query RGB-D observation without providing the bounding box and segmentation mask of the target object in the query image, infer the projected position of the target object's center and select visible object surface points that are close to the projection and have effective observation depth.

[0027] Read the surface depth of the selected surface points from the query depth map. ;

[0028] Locate the same physical surface point in the first reference, and combine the reference's canonical coordinate axes, the known object center depth, camera intrinsic parameters, and the relative orientation of the query viewpoint with respect to the reference viewpoint to infer the signed depth offset of the surface point from the object center along the query camera's depth axis. ;

[0029] according to The object's center depth is calculated, and the three-dimensional translation is obtained by combining the projected position with the back projection of the queried camera intrinsic parameters.

[0030] Furthermore, the intermediate supervised target with signed depth offset is constructed offline during the training data preparation phase: given a ground truth query pose, the object center is projected onto the query image; among pixels that belong to both the target's visible region and have effective observation depth, the surface pixel closest to the projection is selected, its depth is read, and it is back-projected into a 3D point in the query camera coordinate system; this 3D surface point is mapped to the object's normal coordinate system via the inverse transformation of the ground truth query pose, then transformed to the reference camera coordinate system via the known pose of the first reference, and projected onto the reference image; calculation... The center position, surface position and depth, reference corresponding position, signed offset and final center depth are serialized into language-supervised text; during the inference phase, no object mesh, visibility label, query truth pose or explicit correspondence are input.

[0031] Furthermore, single-reference samples, multiple-reference samples, and tracking samples are mixed and share the same set of model parameters in a single-stage training; the single-reference sample contains a reference RGB-D observation with known pose; the multiple-reference sample adds one or more reference RGB-D observations without pose annotation after the first reference; the tracking sample uses one or more historical frames with known or predicted poses as a reference set, and perturbs the historical poses during training and generates coordinate axes and center depth cues based on the perturbated poses;

[0032] During tracking inference, the predicted six-DOF pose at the current moment is re-encoded as a reference cue for the next moment, allowing the same model to run frame by frame without the need for additional detectors, maps, or online object reconstructors.

[0033] Furthermore, the pseudo-color depth map is generated in the following way: the effective depth of each depth map is normalized according to the same preset absolute metric interval, and then a pseudo-color map is generated using a fixed color mapping function so that the same color in different observations corresponds to the same physical depth; invalid depths are identified using fixed reserved values ​​or masks; the preset absolute metric interval is given through text prompts, and different observations use the same absolute depth mapping and the same depth unit.

[0034] The present invention also provides a six-degree-of-freedom object pose estimation device for implementing the above method, comprising:

[0035] The data acquisition module is used to acquire a reference set and a query observation. The reference set contains one or more reference RGB-D observations, and the query observation is a complete unlabeled query RGB-D observation. The first reference in the reference set is a pose-known reference.

[0036] The prompting construction module is used to organize each RGB-D observation into an ordered image pair of an RGB image and a pseudo-color depth map encoded with the same fixed absolute depth scale, overlay normalized coordinate axis markers on the first reference RGB image, and provide the object center depth and camera intrinsic parameters of the first reference to the prompting.

[0037] The shared encoding module is used to jointly encode the reference set and the query observation using a shared multimodal encoding model to obtain a shared hidden representation;

[0038] A rotation estimation module is used to output the rotation of the target object in the query observation from the shared hidden representation through a rotation decoding branch;

[0039] A translation estimation module is used to output the three-dimensional translation of the target object from the shared hidden representation through a translation decoding branch;

[0040] The pose fusion module is used to combine the rotation and the three-dimensional translation into a six-degree-of-freedom pose in the coordinate system of the query observation camera.

[0041] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by one or more processors, implement the above-described method.

[0042] Compared with the prior art, the present invention has at least the following beneficial effects:

[0043] One model covers multiple tasks: it unifies single-reference, multi-reference, and tracking through a variable set of references, avoids deploying different networks and intermediate processing flows for each form of evidence, and can absorb supplementary evidence as the number of available references increases.

[0044] Reduced testing dependencies: Queries use full RGB-D frames, eliminating the need to query bounding boxes or masks of target objects in images; core inference does not require computer-aided design models, fine-tuning of target datasets, online reconstruction, explicit correspondence, or iterative refinement, which is beneficial for deploying new objects.

[0045] Improve the structural adaptability of rotation decoding: coarse-grained rotation classification utilizes the global candidate discrimination ability of shared visual representation, and local rotation residuals are used for local rotation refinement; ensemble coarse supervision and residuals predicted separately for each rotation category can simultaneously handle nearest neighbor candidates and object symmetry.

[0046] Improve the accuracy and interpretability of center depth: Instead of treating center depth as a black box value, it uses the surface depth observed from the depth map and cross-view geometric inferences to infer the signed offset; it still has a clear geometric basis when the center is occluded.

[0047] High parameter and deployment efficiency: Most pre-trained backbones can be frozen, and only low-rank adaptation (LoRA) and lightweight rotation heads are trained; rotation coarsening and refinement are completed in a single forward computation, and tracking does not require building separate maps or object models.

[0048] Extended reference information: The same model can receive a single ground truth reference, multiple supplementary viewpoints, or historical frames with poses by changing the reference set. When the number of references and the task format change, the same model parameters, input organization rules, and rotation and translation decoding mechanisms are maintained. Attached Figure Description

[0049] Figure 1This illustrates the overall flow of the method of the present invention;

[0050] Figure 2 This illustrates a shared multimodal model and a two-branch data flow.

[0051] Figure 3 The offline sampled data represents a fixed rotation.

[0052] Figure 4 This demonstrates multi-objective supervision during training.

[0053] Figure 5 The four-stage inference process of z-CoT is shown. Detailed Implementation

[0054] The present invention will now be described in further detail with reference to the accompanying drawings.

[0055] This invention proposes a unified six-DOF object pose estimation scheme based on reference observations. This scheme reformulates different tasks as the same problem: "given a variable amount of reference evidence, infer the pose of a complete, unlabeled query frame." The first reference observation with known pose establishes the object's canonical coordinate system; additional references or historical frames with poses provide supplementary evidence; a shared multimodal coding model provides unified encoding; rotation and translation are recovered by two branches adapted to their output characteristics, and finally fused into the query pose.

[0056] 1. Overall Process

[0057] like Figure 1 As shown, in one embodiment, the method includes the following steps:

[0058] Step S101, Obtain the reference set and query observations: Obtain one or more reference RGB-D observations of the target object and one query RGB-D observation. The first reference has a known object pose; other references can be supplementary observations without pose annotations, or historical observations with previously predicted poses during tracking.

[0059] Step S102: Construct a multimodal cue for pose estimation: Organize each RGB-D observation into an ordered image pair of an RGB image and a pseudo-color depth map at a fixed absolute depth scale; overlay canonical coordinate axes onto the RGB image of the first reference observation with known pose, and provide the object center depth and camera intrinsics of that reference to the cue. For tracking samples, each historical frame with pose can carry its available pose in the same way. The query always uses the complete image, without providing the bounding box of the target object in the query image or the segmentation mask of the target object in the query image.

[0060] Step S103, Shared multimodal coding: Using a visual language model, the first reference, optional additional reference or historical frame and the query frame are jointly encoded to form a shared hidden representation that includes object semantics, cross-view relationship, local geometry and global orientation information.

[0061] Step S104, Rotation Decoding: Based on the hidden state corresponding to the rotation-specific term, simultaneously predict the probability of each coarse-grained rotation category in SO(3), as well as the local rotation residuals conditioned on each category and using Continuous 6DRotation Representation (6D Rotation); during training, retain all effective coarse rotation supervision targets determined by geometric tolerance and object symmetry.

[0062] Step S105, Translation Decoding: The language branch executes a geometry-grounded z-axis chain of thought (z-CoT) to locate the center projection of the object in the query image, read the depth of the nearby visible surface, and combine it with the first reference observation with pose to infer the signed depth offset from the surface to the center, obtain the center depth, and recover the three-dimensional translation through camera back projection.

[0063] Step S106, Pose Fusion and Task Reuse: Combine rotation and translation into a pose in the query camera coordinate system (Special Euclidean Group, SE(3)). In video tracking, the predicted pose at the current moment is re-encoded as a reference cue for the next moment, allowing the same model to run frame by frame.

[0064] 2. Unified RGB-D input interface based on reference observation

[0065] Let the reference set be , of which Reference Includes an RGB image, an aligned metric depth map, and camera intrinsics; the query observation is... Object pose This method maps the normalized object coordinates to the corresponding camera coordinate system. The output of this solution is the pose lookup query. When homogeneous coordinates are required, you can... The last line is supplemented with [0,0,0,1].

[0066] Each depth map is mapped to a pseudo-color depth map using the same fixed absolute metric range, ensuring that the same color corresponds to the same physical depth in different observations; the depth range used can be provided through text prompts. As one embodiment, the effective depth can be first normalized according to a preset interval, and then a pseudo-color map can be generated using a fixed color mapping function; invalid depths are identified using fixed reserved values ​​or masks. The mapping can also employ other deterministic encodings that maintain absolute scale consistency.

[0067] Since no pose convention is shared during the training phase for any new object, the first reference must simultaneously define a recognizable canonical coordinate system. Let the known pose of the first reference be... This plan will be based on The image projection is taken as the origin, with Oriented three-axis coordinate markers are rendered to the first reference RGB image, and the center depth is... And camera intrinsic parameter writing prompts. The three-axis markers express the mapping from the canonical coordinates to the reference camera coordinates, while the center depth and intrinsic parameters supplement the measurement information that cannot be provided by two-dimensional overlay.

[0068] The single-reference task includes only the first known pose reference; the multi-reference task adds one or more references without pose annotations; the tracking task uses one or more historical frames with known or predicted poses as a reference set. All three tasks query unannotated complete RGB-D frames, thus forming a unified interface. Reference images can be complete frames or images cropped around the target, while query frames remain complete, enabling the model to jointly perform target recognition, localization, and pose estimation.

[0069] 3. Shared multimodal coding model and dual-branch decoding

[0070] like Figure 2 As shown, this embodiment uses the Qwen3-VL-8B-Instruct visual language model. Its visual encoder processes the sequentially input RGB image and pseudo-color depth map. The multimodal feature merging module connects the visual features and the language model. The language model jointly processes image features, camera parameters, reference pose cues, and rotation-specific terms. <r>The pre-trained backbone parameters are kept frozen, LoRA parameters are trained in the language, vision, and feature merging modules, and new parameters are trained. <r>Lexical embedding and two lightweight rotation prediction modules.

[0071] Rotate branch read <r>In the hidden state of the last layer, the lexical term precedes the language response, so rotation prediction does not depend on the subsequently generated z-CoT text. The language branch retains the native autoregressive decoding mechanism, generating z-CoT inference text and object center fields. The two branches share inputs and model parameters, and their outputs are used to recover rotation and translation, respectively.

[0072] 4. Coarse-to-fine rotation decoding considering multiple solutions

[0073] 4.1 SO(3) coarse-grained rotational classification and category condition residuals

[0074] This section defines SO(3) as the set of third-order real rotation matrices satisfying RᵀR=I3 and det(R)=1, where I3 is the third-order identity matrix. The superscript * indicates the truth value or the supervised target determined by the truth value; the subscript pred indicates the model prediction; b and j are rotation categories or candidate indices, and K is the number of rotation categories. Matrix products are applied to column vectors from right to left.

[0075] SO(3) is discretized using a fixed set of representative rotations to establish coarse-grained rotation categories (bins). Pre-constructed A fixed category represents rotation ,and Representative rotations for each category are obtained by sampling the furthest point from a set of candidate rotations represented as unit quaternions, according to the geodesic distance of SO(3), to ensure that the selected representative rotations are as dispersed as possible on SO(3). This set is generated only once and subsequently shared by all objects, datasets, and tasks. The geodesic distance between any two rotations is defined as:

[0076] (1)

[0077] In equation (1), R1 and R2 are two rotation matrices to be compared, tr represents the trace of the matrix, ⊤ represents the transpose, and arccos is the inverse cosine function. The calculated angle and the tolerance in the following text should use the same unit; in this embodiment, δ=10° corresponds to π / 18 radians. During numerical calculation, the inverse cosine input should be limited to its domain [-1,1] to avoid out-of-bounds errors caused by floating-point rounding.

[0078] The iterative process of sampling the farthest point is as follows. 20,000 candidate rotations are uniformly and randomly generated on SO(3), denoted by the rotation matrix Qⱼ, with a total number of candidates M=20,000. In this embodiment, a random seed of 42 is used, and the first candidate in the generation order is taken as the initial point, i.e., j1=1, B1=Q1. Let the set of selected indices in the nth step be J. n For each unselected candidate, calculate its minimum geodesic distance to all selected representatives, and then select the candidate with the largest minimum distance.

[0079] (1a)

[0080] If the maximum and minimum distances are obtained simultaneously by multiple candidates, the candidate with the smallest index in the original candidate pool is selected. The selected candidate is added to the set, and the minimum distance from each unselected candidate to the selected set is updated. This process is repeated until K=128 representative rotations are selected and then terminated. Once the generated representative rotation set is fixed, it is shared by all objects, datasets, training, and inference tasks. Figure 3 Provide the iterative process.

[0081] For each rotation category b, the model predicts a local rotation residual conditioned on that rotation category. And form candidate rotations in a single forward computation. The local rotation residual is represented by a continuous six-dimensional rotation, consisting of a six-dimensional vector. Mapped to SO(3) via the orthogonalization operator. Here, six dimensions refer to the parameter representation dimension of the rotation; the three-dimensional rotation itself still has three degrees of freedom. Unlike the serial scheme of "first calling the model once to select the coarse rotation category, and then calling the model again to regress the local rotation residuals", this invention outputs the probability and conditional residuals of all rotation categories simultaneously.

[0082] The orthogonalization of the continuous six-dimensional rotation representation involves Gram-Schmidt orthogonalization of two three-dimensional vectors, followed by the construction of a third unit vector through the cross product. For any class b, its six-dimensional output is decomposed into... , ,but:

[0083] (1b)

[0084]

[0085] Where, ||·||² represents the Euclidean norm, and × represents the cross product of three-dimensional vectors; e1, e2, and e3 are the three columns of the matrix. The above expression requires... Non-zero, and Not with Collinear. The resulting matrix satisfies ΔRᵀΔR=I3, det(ΔR)=1, belonging to SO(3). The rotation residual is right-multiplied by the category representation according to the coordinate convention of equation (4).

[0086] 4.2 The effective set of supervised targets determined by geometric tolerance and object symmetry (e.g.) Figure 4 (As shown)

[0087] For asymmetric objects, representative rotations of multiple adjacent categories may all lie within a supportable local residual neighborhood; for symmetric objects, several rotations represent the same physical pose. Therefore, this invention does not mandate the use of a unique coarse label, but rather preserves all geometrically valid targets. Let the set of object symmetry transformations be... The set of asymmetric objects contains only unit transformations; continuous rotational symmetry is discretized at fixed 45° intervals, and the symmetry definition adopts the BOP convention.

[0088] Let R* be the truth rotation of the query object, and S∈𝒮 be one of the symmetric rotation transformations. The distance of category b from the rotation to the nearest symmetric equivalent truth rotation, and the category index that minimizes this distance, are respectively:

[0089] (2)

[0090] Given a nonnegative geometric tolerance δ, the effective coarse rotation supervision target set is defined as:

[0091] (3)

[0092] In equation (3), This represents the minimum distance obtained by substituting the optimal category index b* into the category distance; the meaning of b* in the subscript is different from that of * in the distance superscript. This formula retains all categories whose distance does not exceed "minimum distance plus tolerance δ". Therefore, the optimal category must belong to the effective supervised target set, which is not empty; symmetric equivalence is achieved by minimizing S in equation (2), and δ allows the retention of nearest neighbor categories.

[0093] For each b∈C(R) * Select the symmetric equivalent objective closest to the rotation representing that category, and construct the residual objective corresponding to that rotation category:

[0094]

[0095] (4)

[0096]

[0097] Equation (4) sequentially gives the optimal symmetric rotation transformation, the selected symmetric equivalent truth rotation, and the residual supervision objective for category b. The inverse of the representative rotation is equal to its transpose; therefore, multiplying the representative rotation by the residual objective recovers the selected symmetric equivalent truth rotation, with the multiplication order consistent with the combination of predicted candidate rotations.

[0098] From the construction, we can obtain Different effective rotation categories therefore receive different local rotation corrections and no longer share conflicting residual objectives.

[0099] 4.3 Rotation-specific terms <r>Predicting Head and Loss

[0100] Add trainable rotation-specific lexical units to multimodal cues. <r>This lexical unit is located before the language response, and is its last hidden state. It summarizes reference observations, queries observations, and pose cues without relying on pre-generated text. The first projection module outputs... One coarse-grained rotation category probability; the second projection module outputs one time. Group 6 residuals:

[0101] (5)

[0102] Let the dimension of the hidden state be d, then It is a d-dimensional real vector; Mapping these scores to K classification scores, softmax converts the scores into non-negative class probabilities that sum to 1. Output 6K real numbers, reshape them into a K×6 matrix A arranged by category, with each row corresponding to a six-dimensional residual representation of a category.

[0103] Among them, A's first After the above orthogonalization, we obtain .remember The coarse-grained classification loss improves the sum of predicted probabilities for each category within the effective supervised target set by minimizing the negative log-likelihood, while the local rotation residual loss supervises each effective rotation category separately.

[0104] (6)

[0105] In equation (6), |C*| is the number of elements in the effective class set, and the probability in the summation term corresponds to class b. The coarse-grained classification loss applies to the sum of probabilities of the effective classes; when the set contains only one class, it degenerates into ordinary single-objective cross-entropy. The residual loss calculates the rotation geodesic distance for each effective class and takes the average, without requiring different classes to share the same residual objective.

[0106] This design allows the model to retain non-uniform probabilities among symmetric or nearest-neighbor candidates without requiring an average distribution of probabilities; it also ensures that any selected valid rotation class has a corresponding local rotation correction. During inference, the rotation class with the highest probability and its conditional residual are selected.

[0107] (7)

[0108] 5. z-CoT translation decoding based on geometric information

[0109] 5.1 Central Projection and Back Projection

[0110] The language branch generates progressively deeper-axis inference text and ends with structured fields, such as: <origin>u n ,v n ,z center < / origin> Among them, (u n , v n ) are the normalized projected coordinates of the object's center in the complete query image, and both of them have a range of [0, 1000]. This is the depth of the object's center, expressed in millimeters. Let the width and height of the query image be W and H, respectively. By querying the camera's intrinsic parameters, the 3D translation is obtained through backprojection using the following formula:

[0111] (8)

[0112] In equation (8), K qry To query the camera's 3×3 intrinsic parameter matrix, (u,v) represents the pixel coordinates of the complete query image, [u,v,1]⊤ represents the homogeneous image coordinates, and t pred,qry This defines the 3D position of the object's normalized coordinate system origin within the query camera coordinate system. Here, the depth along the camera's depth axis is used, rather than the Euclidean distance from the camera center to a surface point.

[0113] 5.2 Explicit Decomposition of Surface Depth and Center Depth

[0114] Depth maps measure the depth of visible surface points. Pose translation requires the depth of the object's center. Both conditions are met:

[0115] (9)

[0116] in, Can be read from the query depth map; signed offset This depends on the selected surface, object orientation, and geometry. Directly outputting the center depth hides the aforementioned observables and geometric relationships, easily leading to significant depth errors. Therefore, this invention manifests it as a four-stage z-CoT (e.g., Figure 5 (as shown)

[0117] (1) Surface point selection: In a complete query RGB-D without providing a segmentation mask of the target object in the query image, first infer the center projection position of the object, and select a visible object surface point that is close to the projection and has an effective observation depth; when occluded, the surface point can be different from the center projection.

[0118] (2) Observation depth reading: Read the absolute depth of the selected surface point from the query depth map. .

[0119] (3) Geometric inference based on cross-viewpoint correspondence: Locate the same physical surface point in the first reference observation with known pose. Combine the object canonical coordinate axis in the reference image, the known object center depth, camera intrinsic parameters, and the relative orientation of the query viewpoint relative to the reference viewpoint inferred from the shared representation, estimate the signed depth offset of the surface point to the object center along the query camera depth axis.

[0120] (4) Calculation of center depth: Add the depth of the observed surface to the signed offset according to equation (9), output the object center projection and center depth, and then back-project according to equation (8) to obtain the three-dimensional translation.

[0121] The following example, using pose estimation under a single reference condition, illustrates the four-stage inference process and the values ​​of each output field. The example lists the model's inputs and outputs. The two-dimensional projection position of the object's center, the depth of the selected visible surface point, and the signed depth offset from the surface point to the object's center are all predicted by the model based on the input information and are not provided as pre-defined query inputs.

[0122] Input images and prompts: A is a reference RGB image with superimposed object coordinate axes, where +X, +Y, and +Z are represented by red, green, and blue, respectively; B is a reference pseudo-color depth map; C is the complete query RGB image; D is the query pseudo-color depth map. Both depth maps B and D together map 421 mm to a normalized depth of 0 and 1389 mm to a normalized depth of 1. The reference center field is O=(559,441,894), where the center depth is 894 mm; the camera parameter is f. x =f y =1000, c x =W / 2, c y =H / 2.

[0123] The inference instructions in the prompt are as follows: First, determine the projection of the center of the query object; within the visible object region, read the stable surface depth closest to this projection; then, using the canonical coordinate axes in the reference image, the object shape, and the viewpoint change from the reference to the query, estimate the signed offset of the surface to the center along the query depth axis; add the two to obtain the center depth. The object center may be located inside the object or occluded; therefore, the depth of the selected visible surface point is used for inference. Final output. <origin>u n ,v n ,z center < / origin> , where u n , v n z is an integer in the range [0, 1000]. center Camera depth is in millimeters; no spaces are included inside the label.

[0124] Phase 1, Center Positioning and Surface Point Selection: Query the normalized projection of the object center as (134, 276), and select a visible object surface point near this projection.

[0125] Phase 2, Read Surface Depth: The depth of the selected visible surface point in the query depth map is 804 mm.

[0126] Phase 3, Cross-viewpoint correspondence and offset estimation: The physical surface point corresponds to the position (568, 427) in the reference image. The reference center is located at (559, 441) with a depth of 894 mm. Combining the reference geometry and viewpoint changes, the signed offset from the surface point to the center along the query camera depth axis is -19 mm.

[0127] Phase 4, Center Depth Synthesis and Output: z_center = 804 + (-19) = 785 mm, final output <origin> 134,276,785< / origin> .

[0128] In this example, the signed offset is -19 mm, indicating that the depth of the object's center is less than the depth of the selected surface point. The normalized center coordinates (134, 276) are converted into query pixel coordinates according to the coordinate relationship in Section 5.1. Then, combined with the center depth of 785 mm and the query camera intrinsics, the three-dimensional translation is obtained by backprojection according to Equation (8).

[0129] 5.3 Offline Construction of z-CoT Supervised Targets

[0130] The z-CoT intermediate supervised targets are constructed offline during the training data preparation phase and used for supervision during model training; these intermediate annotations are not provided during the inference phase. Given a ground truth query pose, the object center is projected onto the query image. Among pixels that are both within the target's visible region and have effective observation depth, the surface pixel closest to this projection is selected, its depth is read, and it is back-projected into a 3D point in the query camera coordinates. If no valid sample meets the conditions, the query training sample is discarded. Subsequently, this 3D surface point is mapped to the object's canonical coordinates via the inverse transformation of the ground truth query pose, then transformed to the reference camera coordinates via the known pose of the first reference, and projected onto the reference image. Finally, the following calculations are performed:

[0131] (10)

[0132] Expanding the coordinate transformations in the offline construction, we can write:

[0133]

[0134] (10a)

[0135]

[0136] Among them, (u s ,v s ) represents the selected query surface pixels; , , These are the three-dimensional coordinates of the same physical surface point in the coordinate systems of the query camera, object specification, and reference camera, respectively. , To find the pose for the truth value query; , and The rotation, translation, and intrinsic parameters are the first reference. π([a,b,c]⊤)=(a / c,b / c) represents perspective division. For the corresponding pixel in the reference image. Equation (10) It is the component of the truth query translation along the depth axis.

[0137] The center position, surface position and depth, reference correspondence position, signed offset, and final center depth are serialized into language-supervised text. During inference, no object mesh, visibility labels, query truth poses, or explicit correspondences are input; this information is only used for offline generation of supervised targets.

[0138] 6. Unified training and reasoning

[0139] 6.1 Mixed Task Training

[0140] Single-reference, multi-reference, and tracking samples are mixed in a single training session, with roughly equal numbers of samples across the three classes, and they share the same set of model parameters. Single-reference samples contain one RGB-D reference observation with a known pose; multi-reference samples include a random number of additional RGB-D reference observations, with only the first reference carrying the object's canonical coordinate axes and center depth cue; tracking samples use a random number of historical frames as a reference set, each frame carrying its available pose. During training, the historical poses are perturbed, and coordinate axes and center depth cue are generated based on the perturbed pose to simulate accumulated tracking error. The current query frame does not contain pose cue.

[0141] The supervision sources for the training data were handled separately. FoundationPose data is synthetic rendering data, and its depth, object mesh, segmentation mask, and visibility are obtained from the rendering process. HOT3D uses the official high-resolution object scan model and pose annotations; Objectron uses the official 3D bounding box pose annotations and reconstructs the mesh of each object offline from the video and its sparse point cloud. On both HOT3D and Objectron, the object mesh is projected using the annotated poses to obtain the object mask and visibility; neither provides dense depth, so Depth Anything 3 is used to predict the depth metric for both the reference and query frames. These meshes, masks, and visibilitys are used to construct supervision offline, and the pose model does not receive query ground truth annotations during inference.

[0142] The training data processing sequence is as follows: read aligned RGB images, measure depth, camera intrinsics, and pose annotations; organize reference sets and query frames according to single-reference, multi-reference, or tracking methods; for each reference RGB-D, retain the complete frame or crop it around the target with a 50% probability; generate pseudo-color depth maps for each observation using a fixed absolute depth map; add coordinate axes, center depth, and camera parameters to the references with available poses; construct rotational and linguistic supervised targets according to Sections 4.2 and 5.3. Queries always retain complete frames.

[0143] When referencing cropping, the RGB values, depth, and cues in the image should use a consistent cropping region. Pixel coordinates and camera intrinsics used in conjunction with the cropped image should correspond to the same image coordinate definition to maintain projection and back-projection relationships. Different observations should use the same absolute depth mapping and the same depth unit; the color range should not be reset separately for each image's minimum and maximum depth.

[0144] Training references are uniformly and randomly selected from available references after excluding query frames; the references for tracking samples are limited to historical frames prior to the current query. When cropping, the width and height are each preserved at 50%–100% of the original image, and object masks from the training data are used to constrain the cropping region to avoid cropping the target object. The RGB image, depth map, and cue use the same cropping region, and the camera's coordinates are synchronized to correspond to the cropped image.

[0145] In this embodiment, the angles of the historical rotation perturbations are distributed using a zero-mean Gaussian distribution with a standard deviation of 6.87°. The canonical coordinate axis cues in the historical frames are updated based on the perturbed rotations to enhance the model's adaptability to historical rotation estimation errors.

[0146] 6.2 Joint Optimization

[0147] In the embodiment employing efficient parameter fine-tuning, the pre-trained visual language model backbone is frozen, and low-rank adaptation parameters are jointly trained and new parameters are added. <r>Lexical embeddings and rotation heads. The total loss is:

[0148] (11)

[0149] in, z-CoT inference text and final <origin>Next-word prediction loss for the field and The weights are used as the basis for prediction of the current word. During training, the current word is predicted conditioned on the previous target word. The coarse-grained classification loss and the category-conditional residual loss are calculated according to Equation (6). The LoRA parameters are jointly updated, new word embeddings are added, and the prediction head is rotated, while the remaining parameters of the pre-trained backbone are kept frozen.

[0150] 6.3 Unified Reasoning and Tracing Recursion

[0151] The language branch uses greedy decoding and parses the last valid center field; the rotation branch selects the rotation category with the highest predicted probability and its corresponding residual. The results from both branches constitute... During tracking, external initialization is only required for the first frame; subsequently, one or more recent predicted poses are overlaid onto the corresponding historical RGB image and a center depth is added, making it a reference observation with predicted pose for the next time step. This process does not require additional detectors, maps, online object reconstructors, or mission-specific tracking heads.

[0152] The length of the tracking reference history is denoted as H_hist, with a default value of 3. The tracking reference can be configured as a fixed initial reference, the most recent prediction history frame, or the most recent three prediction history frames. When sufficient prediction history has not yet been accumulated, the remaining reference positions are supplemented with an initial reference whose pose is known; the reference set only uses observations prior to the current time step, and subsequent frames are not reinitialized.

[0153] In another inference implementation, let the number of language generation iterations be N_s, where N_s≥1. Keeping the rotation result fixed, perform one greedy language generation and N_s-1 random sampling language generation iterations. Backproject the center projection and depth of each generation into a 3D translation, and then take the median of each coordinate component of the N_s translations to obtain the final translation. The experimental results in Section 8 use a single greedy decoding.

[0154] 7. Implementation methods for devices, electronic equipment, and storage media

[0155] The present invention also provides a six-degree-of-freedom object pose estimation device, comprising: a data acquisition module for acquiring reference RGB-D and query RGB-D; a cue construction module for generating a pseudo-color depth map encoded with a fixed absolute depth scale, and organizing the reference pose cue, the reference object center depth and camera intrinsic parameters into a multimodal cue; a shared encoding module for generating a shared representation through a visual language model; a rotation estimation module for outputting the predicted probability of each coarse-grained rotation category and the local rotation residual conditioned on the rotation category; a translation estimation module for performing z-CoT and back-projection; a pose fusion module for generating SE(3) pose; and a tracking recursion module for converting the predicted pose into a subsequent reference.

[0156] The device can be deployed on robots with RGB-D cameras, industrial camera workstations, augmented reality terminals, edge computing devices, or servers. The electronic device includes one or more processors, memory, graphics processors or neural network accelerators, and communication interfaces; instructions in the memory, when executed by the processor, implement the methods described above. The instructions, model parameters, and representative rotation and depth color mapping configurations for each category can be stored in a computer-readable storage medium.

[0157] 8. Training Configuration and Experimental Results

[0158] This embodiment uses Qwen3-VL-8B-Instruct to train the LoRA parameters in the language, vision, and multimodal feature merging modules, with rank r=16 and scaling parameter α=32, and then trains... <r>A word embedding and rotational multilayer perceptron was used. The trainable parameters are approximately 55.1 million, representing 0.62% of the model parameters. K=128, δ=10°, λcoarse=λres=0.2. Eight A100 GPUs were used, trained for 10 epochs with the AdamW optimizer, a global batch size of 32, BF16 precision, and a learning rate based on a cosine policy starting from 10⁻⁻⁶. 4 Attenuation to 10⁻ 5 The training images included 60,000 from the FoundationPose dataset, 30,000 from the Objectron dataset, and 30,000 from the HOT3D dataset. After training, the same set of model parameters was used to evaluate single-reference estimation, 16-reference estimation, and video tracking; fine-tuning was not performed on the target test dataset. Single-reference and multi-reference experiments used real RGB-D reference observations; tracking was initialized only with the ground truth pose in the first frame, and subsequently used the three most recent predicted historical frames by default. If historical data was insufficient, it was supplemented by the initial reference; subsequent frames were not re-initialized.

[0159] Single-reference evaluation uses a real RGB-D reference frame with known pose for each object. 16-reference evaluation uniformly selects 16 frames from the pose-bearing reference sequence, with only the first reference encoding its pose and the remaining references providing only RGB-D.

[0160] Evaluation metrics include ADD-0.1d and the area under the curve (AUC) for ADD and ADD-S. ADD represents the average distance between corresponding points of the object model after transformation between the predicted pose and the true pose; ADD-S uses the nearest point distance for matching to account for object symmetry. ADD-0.1d represents the proportion of samples with an error less than 10% of the object diameter. The results of single-reference and multi-reference pose estimation are shown in Table 1.

[0161] Table 1. Results of Single-Reference and Multi-Reference Pose Estimation

[0162] Single reference 1 real RGB-D reference LINEMOD ADD-0.1d 94.7 Single reference 1 real RGB-D reference YCB-Video ADD / ADD-S AUC 87.6 / 95.8 More references 16 real RGB-D references LINEMOD ADD-0.1d 99.6 More references 16 real RGB-D references YCB-Video ADD / ADD-S AUC 92.3 / 98.5

[0163] Table 2 lists the single-reference pose estimation results of the method of this invention compared with FoundationPose, One2Any, and CoordAR. All methods use a single real RGB-D reference observation; FoundationPose also uses a CAD model.

[0164] Table 2 Comparison of Single Reference Pose Estimation Results

[0165] FoundationPose RGB-D; CAD 1 real 48.3 76.1 90.4 One2Any RGB-D; None 1 real 52.6 84.4 93.7 CoordAR RGB-D; None 1 real 75.0 78.5 95.5 Method of the present invention RGB-D; None 1 real 94.7 87.6 95.8

[0166] Note: All values ​​are percentages, higher is better; LM stands for LINEMOD, YCB-V stands for YCB-Video. "None" indicates that no additional CAD model or lookup of pose ground truth components is used, and each method still accepts the reference pose. In LM evaluation, the method of this invention uses ADD-0.1d for all objects; the three comparison methods use ADD-S-0.1d for the two symmetrical objects eggbox and glue, and ADD-0.1d for the remaining objects.

[0167] 8.1 Component Ablation of the Complete Model

[0168] In the LINEMOD single-reference pose estimation task, model components were removed individually, and the impact of each component on pose accuracy and inference latency was evaluated. Each variant was retrained using the same data and training plan as the complete model, and the results are shown in Table 3. ADD-0.1d represents the percentage, rotation error is the average rotation angle error, and translation error is the average 3D translation error. Latency was measured on the same A100 sheet with a batch size of 1, including cue encoding and complete language generation.

[0169] Table 3. Component Ablation of the Complete Model

[0170] Complete model 94.7 4.95 7.61 0.07 Remove multiple coarse rotating targets 57.5 19.35 7.49 0.07 Remove coarse-to-fine rotation decoding 52.7 21.51 8.43 0.07 Remove z-CoT 75.9 4.91 13.35 0.05 Remove explicit metric geometric reasoning supervision 87.3 4.97 10.14 0.06 Remove the rotating head 43.6 23.42 7.57 0.07 Remove text translation decoding 62.8 4.69 17.48 0.05

[0171] The variant is constructed as follows. When removing multiple coarse rotation targets, only the single class closest to the target rotation is supervised, while the nearest neighbor class and symmetric equivalence class are removed; when removing coarse-to-fine rotation decoding, from... <r>The hidden state directly regresses continuous six-dimensional rotations; when z-CoT is removed, translation results are generated directly in a single step. When explicit metric geometric inference supervision is removed, the z-CoT supervised text, which replaces the geometric construction, is interpreted in a free form generated by zero-shot pre-trained VLM. When the rotation-specific head is removed, rotation category indices and residuals are generated sequentially from the text; when text translation decoding is removed, the center 2D projection and depth are predicted using the dedicated head and then back-projected.

[0172] Compared to the variant that only supervises a single coarse rotating target, the full model improves ADD-0.1d by 37.2 percentage points, reduces the rotation error from 19.35° to 4.95°, and shows little change in translation error. Compared to the variant that directly generates translation results in a single step, the full model improves ADD-0.1d by 18.8 percentage points, reduces the translation error from 13.35 mm to 7.61 mm, and maintains approximately no change in rotation error. These results indicate that multi-target rotation supervision helps reduce rotation error, and z-CoT translation decoding helps reduce translation error.

[0173] Table 4 Comparison of translation error components

[0174] Remove z-CoT 11.72 5.59 Remove explicit metric geometric reasoning supervision 8.04 5.53 Complete model 4.62 5.47

[0175] Table 4 further compares the depth axis component and the image plane direction component of the translation error. After applying z-CoT, the depth axis error decreased from 11.72 mm to 4.62 mm, and the translation error in the image plane direction decreased from 5.59 mm to 5.47 mm, indicating that the improvement in translation accuracy of this method is mainly reflected in the object center depth estimation.

[0176] 8.2 Tracking history configuration for the same model

[0177] Table 5 Comparison of historical tracking configurations for the same model

[0178] Fixed initial reference 89.62 96.54 87.6 95.8 The most recent prediction historical frame 91.25 97.18 89.3 96.6 The three most recent historical prediction frames 93.17 98.68 92.3 98.5

[0179] In Table 5, all configurations share the same set of model parameters, and all values ​​are percentages. Each configuration is initialized only with the ground truth pose in the first frame and is not re-initialized thereafter. When using the three most recent prediction history frames, the ADD AUC of YCBInEOAT increased from 89.62% to 93.17% with the fixed initial reference configuration, and the ADD AUC of YCB-Video increased from 87.6% to 92.3%, indicating that updating the history reference helps improve tracking accuracy.

[0180] Cross-dataset evaluation employed BOP (Balance of Surface) average recall (AR), and statistically analyzed using three error metrics: visible surface difference (VSD), maximum surface distance considering symmetry (MSSD), and maximum projected distance considering symmetry (MSPD). Under real single-reference input, the BOPAR of the proposed method on LM-O, TUD-L, IC-BIN, HB, YCB-Video, and T-LESS were 68.1%, 91.7%, 56.4%, 79.2%, 86.4%, and 87.1%, respectively.

[0181] 9. Reference Configuration and Implementation Method

[0182] The following implementation uses the same fixed representative rotation, class conditional residual, explicit z-CoT, and joint training model, reusing the same estimation method through the composition of the reference set.

[0183] Single-reference implementation: Obtain a real RGB-D reference observation with known pose, render the object's standard coordinate axes according to its pose, and provide the reference object's center depth and camera intrinsic parameters; input it into the model together with the complete query RGB-D, and restore the query rotation and translation according to Sections 4 and 5 of Part 5, respectively.

[0184] Multi-reference implementation: After the first reference observation with known pose, other real RGB-D reference observations are added. Only the first reference carries coordinate axis and center depth pose cues, while the additional references provide supplementary perspectives. The 16-reference experiment uses a total of 16 reference observations, and the model and its parameters are the same as in the single-reference implementation.

[0185] Tracking Implementation: The first frame is initialized with the ground truth pose; subsequently, historical RGB-D values ​​and their existing predicted poses are organized into a reference set. For each selected historical frame, corresponding coordinate axes are plotted and a center depth is provided; the current frame remains intact and no pose cues are provided. After the model predicts the current pose, this prediction is used as a reference cues for subsequent frames.

[0186] Historical tracking configuration: By default, the three most recent prediction history frames are used; if the history is insufficient, the initial pose reference is used to supplement it. Alternatively, only a fixed initial reference can be retained, or only the most recent prediction history frame can be updated; these configurations use the same pattern and decoding structure. A quantitative comparison is shown in Table 6 of Part 5.

[0187] Reference image format: During training, the reference RGB-D image is either preserved as a complete frame or cropped around the target with a 50% probability to adapt to different reference image ranges. Both formats use a consistent RGB-D coordinate relationship and absolute depth mapping, ensuring the query image remains intact.

[0188] Common Depth Input: All the above implementations use a depth map aligned with the RGB image as input. Each depth map generates a pseudo-color image using the same fixed absolute depth map, the range of which is explicitly given in the text prompt. The depth of the reference object center and the final output center depth are uniformly expressed in millimeters.

[0189] Common rotation decoding: All configurations use K pre-generated representative rotations, through the same <r>The hidden state outputs both the category probability and the local rotation residual for each category; the category with the highest probability is selected, and its representative rotation is multiplied by the orthogonalized residual to obtain the query rotation.

[0190] Common translation decoding: All configurations are explicitly output by the native language decoder as four-stage z-CoT text and final... <origin>Fields. Parse the last valid field to recover the pixel coordinates and center depth of the queried object, and then use the query camera intrinsic parameters for backprojection to obtain the 3D translation.

[0191] Backbone size configuration: The visual language model can adopt a 2B, 4B, or 8B model from the Qwen3-VL series, maintaining consistency in reference input organization, dual-branch decoding structure, training data, and supervision methods. The training configuration and experimental results described in Part 5 use the 8B model.

[0192] This invention can be used as a pose service in a robot operating system, a unified positioning module in a camera software development kit, a model plugin on an edge device, or a cloud vision interface. Its input and output definitions are clear, allowing it to interface with grasping planning, motion control, quality inspection, and 3D visualization modules; the low-rank adapter and lightweight rotating head also facilitate compression, quantization, and acceleration tailored to specific hardware.

[0193] The main application scenarios include:

[0194] Robot gripping, assembly, and teaching: Define a new part coordinate system using a known pose teaching diagram, and directly restore the pose in the complete RGB-D screen of the workpiece stage; adding a reference view can improve reliability under occlusion and large viewing angle changes.

[0195] Industrial inspection and flexible manufacturing: For small batches and multiple types of workpieces, there is no need to create precise CAD models for each one. Workpiece positioning, defect area alignment and measurement coordinate transformation can be completed quickly using real reference drawings.

[0196] Warehousing, sorting, and retail: For products with frequently changing packaging or long-tail products, on-site reference observation is used to complete the shelving and grab positioning, and the pose is maintained through continuous historical frames.

[0197] Augmented Reality and Digital Twins: Stably align real-world objects to virtual content, supporting single initialization, multi-view enhancement, and continuous tracking, reducing the need for creating dedicated object models.

[0198] Scientific data acquisition and human-computer interaction: Using the same interface to process static estimation and video sequences, it provides a unified pose estimation method for motion analysis, object interaction understanding and spatial intelligent data annotation.

[0199] The above embodiments are merely typical implementations of the present invention and are not intended to limit the present invention. Any equivalent substitutions or improvements made within the scope of the claims of the present invention are within the protection scope of the present invention.< / origin> < / r> < / r> < / r> < / origin> < / r> < / r> < / r> < / r> < / r> < / r>

Claims

1. A six-DOF object pose estimation method based on a visual language model and conditioned on reference observations, characterized in that, include: Obtain a reference set and a query observation. The reference set contains one or more reference RGB-D observations, and the query observation is a complete query RGB-D observation without labeling. The first reference in the reference set is a pose-known reference, and the remaining references are supplementary observations without pose labeling or historical observations with previously predicted poses. Construct a multimodal cue for pose estimation by organizing each RGB-D observation in the reference set into an ordered image pair of an RGB image and a pseudo-color depth map encoded with the same fixed absolute depth scale. Superimpose the canonical coordinate axis markers determined by the known pose on the first reference RGB image and provide the object center depth and camera intrinsics of the first reference to the cue. The reference set and the query observation are jointly encoded using a shared multimodal coding model to obtain a shared hidden representation; From the shared hidden representation, the rotation of the target object in the query observation is output through the rotation decoding branch, and the three-dimensional translation of the target object is output through the translation decoding branch; The rotation and the three-dimensional translation are combined to form a six-degree-of-freedom pose in the coordinate system of the query observation camera.

2. The method according to claim 1, characterized in that, The rotation decoding branch is configured as follows: SO(3) is discretized into K coarse-grained rotation categories by a fixed set of representative rotations, which are obtained by sampling the farthest point from the candidate rotation set represented by unit quaternions according to the geodesic distance of SO(3); Based on the hidden state corresponding to the rotation-specific terminology, the probability of each of the K coarse-grained rotation categories is predicted, as well as the local rotation residual represented by a continuous six-dimensional rotation representation conditioned on each coarse-grained rotation category. The continuous six-dimensional rotation representation is mapped to SO(3) by an orthogonalization operator. During inference, the coarse-grained rotation category with the highest probability and its corresponding local rotation residual are selected. The representative rotation of the selected category is multiplied by the orthogonalized local rotation residual to obtain the rotation of the target object.

3. The method according to claim 2, characterized in that, During training, for each rotation category b, the rotations represented by that category are... To the closest symmetric equivalent truth value distance Determine the category with the smallest distance and with Construct thresholds based on geometric tolerances The effective coarse rotation target set is determined by the object symmetry transformation set S. ,in The coarse-grained classification loss increases the sum of prediction probabilities for each category within the effective coarse rotation target set by minimizing the negative log-likelihood, while the local rotation residual loss supervises the conditional residual for each rotation category within the effective coarse rotation target set.

4. The method according to claim 3, characterized in that, For each rotation category b in the set of effective coarse rotation targets, select the symmetric equivalent target that is closest to the rotation represented by that category. ,in And construct the residual target corresponding to this rotation category. Different effective rotation categories receive different residual targets, and each selected effective rotation category has its corresponding local rotation correction.

5. The method according to claim 1, characterized in that, The translation decoding branch is configured to execute a depth-axis thought chain based on geometric information, including: In a full query RGB-D observation without providing the bounding box and segmentation mask of the target object in the query image, infer the projected position of the target object's center and select visible object surface points that are close to the projection and have effective observation depth. Read the surface depth of the selected surface points from the query depth map. ; Locate the same physical surface point in the first reference, and combine the reference's canonical coordinate axes, the known object center depth, camera intrinsic parameters, and the relative orientation of the query viewpoint with respect to the reference viewpoint to infer the signed depth offset of the surface point from the object center along the query camera's depth axis. ; according to The object's center depth is calculated, and the three-dimensional translation is obtained by combining the projected position with the back projection of the queried camera intrinsic parameters.

6. The method according to claim 5, characterized in that, The intermediate supervised target of the signed depth offset is constructed offline during the training data preparation stage: given the ground truth query pose, the object center is projected onto the query image, and the surface pixel closest to the projection is selected from the pixels that belong to both the visible area of ​​the target and have effective observation depth. Its depth is read and back-projected into a three-dimensional point in the query camera coordinate system. The 3D surface points are mapped to the object's canonical coordinate system via the inverse transformation of the truth-finding pose, then transformed to the reference camera coordinate system via the known pose of the first reference, and projected onto the reference image; calculation The center position, surface position and depth, reference corresponding position, signed offset and final center depth are serialized into language-supervised text; during the inference phase, no object mesh, visibility label, query truth pose or explicit correspondence are input.

7. The method according to claim 1, characterized in that, Single reference samples, multiple reference samples, and tracking samples are mixed and share the same set of model parameters in a single-stage training; the single reference sample contains a reference RGB-D observation with known pose; the multiple reference samples add one or more reference RGB-D observations without pose annotation after the first reference; the tracking sample uses one or more historical frames with known or predicted poses as a reference set, and perturbs the historical poses during training and generates coordinate axes and center depth cues based on the perturbated poses; During tracking inference, the predicted six-DOF pose at the current moment is re-encoded as a reference cue for the next moment, allowing the same model to run frame by frame without the need for additional detectors, maps, or online object reconstructors.

8. The method according to claim 1, characterized in that, The pseudo-color depth map is generated in the following way: the effective depth of each depth map is normalized according to the same preset absolute metric interval, and then a pseudo-color map is generated using a fixed color mapping function so that the same color in different observations corresponds to the same physical depth; invalid depths are identified by fixed reserved values ​​or masks; the preset absolute metric interval is given through text prompts, and different observations use the same absolute depth mapping and the same depth unit.

9. A six-degree-of-freedom object pose estimation device for implementing the method of any one of claims 1 to 8, characterized in that, include: The data acquisition module is used to acquire a reference set and a query observation. The reference set contains one or more reference RGB-D observations, and the query observation is a complete unlabeled query RGB-D observation. The first reference in the reference set is a pose-known reference. The prompting construction module is used to organize each RGB-D observation into an ordered image pair of an RGB image and a pseudo-color depth map encoded with the same fixed absolute depth scale, overlay normalized coordinate axis markers on the first reference RGB image, and provide the object center depth and camera intrinsic parameters of the first reference to the prompting. The shared encoding module is used to jointly encode the reference set and the query observation using a shared multimodal encoding model to obtain a shared hidden representation; A rotation estimation module is used to output the rotation of the target object in the query observation from the shared hidden representation through a rotation decoding branch; A translation estimation module is used to output the three-dimensional translation of the target object from the shared hidden representation through a translation decoding branch; The pose fusion module is used to combine the rotation and the three-dimensional translation into a six-degree-of-freedom pose in the coordinate system of the query observation camera.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by one or more processors, they implement the method of any one of claims 1 to 8.

Images