A priori-free articulated object state generation and joint estimation method

Abstract

The application discloses a priori-free hinge object state generation and joint estimation method, which is characterized by using a single closed state RGB image, through a feature reconstruction module, a new state generation module and a 3D perception joint estimation module, learning a cross-state feature mapping in a self-supervised manner, generating a high-fidelity new state image sequence, positioning a motion seed and accurately estimating a joint 3D parameter. The application does not require auxiliary inputs such as part masks, language prompts and camera external parameters during training, and in the test, a single closed state image is given, and a new state generation result and a joint 3D parameter set are output in real time through feedforward reasoning. Compared with the prior art, the application solves the problems of single-image joint 3D parameter estimation depending on prior information, depth ambiguity of view angle and poor generalization, realizes real-time joint 3D parameter estimation with generalization on task data, and improves the new view generation accuracy and joint 3D parameter prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of 3D articulated object generation technology in computer vision and computer graphics, and in particular to the construction of a feedforward joint 3D parameter estimation method based on implicit feature dynamic cross-state mapping and 3D point cloud reconstruction. Specifically, it is a method for generating hinged object states and estimating joints without prior knowledge. Background Technology

[0002] The perception of motion structure of 3D articulated objects is of great significance in fields such as embodied AI environmental interaction, robot manipulation, 3D game asset generation, and virtual reality (VR) scene construction. Modern intelligent systems rely on single-image input to understand the motion logic of objects, thereby supporting downstream interactive tasks. However, single-image joint parameter estimation faces many challenges, such as the lack of depth information due to sparse viewpoints, large differences in topological structures among different types of objects, and the real-time requirements of downstream tasks. These factors together constitute the main difficulties in achieving accurate and efficient joint estimation in open-world scenarios.

[0003] In recent years, although multi-state reconstruction methods and single-image joint estimation methods have made some progress in this field, they have significant limitations: multi-state methods rely on multi-state observation data of objects, including image pairs, video or interaction data, and it is difficult to obtain additional state information during testing; single-image methods either rely on category prior knowledge or require auxiliary inputs such as part masks and language prompts, resulting in poor generalization and inability to handle unlabeled open-world images; at the same time, existing methods generally suffer from large joint parameter prediction errors and high inference latency, making it difficult to meet the needs of real-time interaction.

[0004] To address these challenges, current research focuses on developing joint estimation methods that can achieve real-time performance and good generalization from single unlabeled images. These methods attempt to construct cross-state mappings using generative models or regression networks. However, due to the depth ambiguity of 2D images, the lack of explicit geometric constraints, and the fact that network architecture design does not fully consider the physical consistency of joint motion, existing methods suffer from insufficient accuracy in joint type identification, axis parameter localization, and motion range estimation. Furthermore, the synthesized open-state images exhibit problems such as texture drift and structural distortion.

[0005] Furthermore, existing methods typically employ fixed network input patterns, which cannot flexibly adapt to the estimation needs of images with different resolutions and objects with different topological structures. Moreover, the training process relies on camera extrinsic parameters or manually labeled joint information, resulting in limited model generalization ability and difficulty in fully utilizing unlabeled data in the open world.

[0006] In summary, developing a real-time joint parameter estimation technique that is applicable to single unlabeled closed-state image inputs and has generalization capabilities is a key problem that urgently needs to be solved in the field of jointed object perception. Summary of the Invention

[0007] The purpose of this invention is to address the limitation of existing technologies in estimating all joint parameters from a single static image, by designing a joint parameter estimation method applicable to a single closed-state image. It employs cross-state mapping based on latent dynamics and 3D perception aggregation technology. During training, a feature reconstruction module, a new state generation module, and a 3D perception joint estimation module are jointly trained to predict open-state images and joint parameters. During inference, real-time joint estimation from a single closed-state image is achieved through a feedforward approach.

[0008] The objective of this invention is achieved as follows:

[0009] A priori-free method for generating states and estimating joints in hinged objects is proposed. This method employs implicit feature dynamic cross-state mapping representation and uses a feedforward approach to achieve real-time generation of new states and estimation of 3D joint parameters using a single closed state image.

[0010] The specific steps are as follows:

[0011] Step 1: Dataset Preparation and Preprocessing

[0012] We collected single-image RGB images, 3D point clouds, and mesh data of 27,347 articulated objects in their closed state. Each object category contains different instances within that category. For each object category, we collected multiple different instances, calibrated the image resolution and pixel normalization range, and defined I_0 as a single image in the closed state and I_1 as a single image in the open state. The instances differ in shape, structure, and component combination, and we provide samples from multiple angles and states based on the instances.

[0013] The image resolution is calibrated to 224×224, and the pixel value normalization range is [0,1].

[0014] Step 2: Network Architecture Construction

[0015] Construct an end-to-end network consisting of three main functional modules:

[0016] 2-1: Feature reconstruction module, which uses a frozen pre-trained visual encoder and a trainable multi-level decoder to extract multi-scale image features and align the feature space through reconstruction task;

[0017] 2-2: The new state generation module uses a Transformer generator based on Adaptive Layer Normalization (AdaLN) to inject state index information and generate open state images;

[0018] 2-3: The 3D perception joint estimation module consists of a point cloud lifting submodule and an ensemble prediction submodule, which is used to extract motion seeds to achieve end-to-end estimation of joint parameters;

[0019] Step 3: Joint training, including:

[0020] 3-1: Input a single closed-state RGB image, perform data augmentation, and then input it into the feature reconstruction module;

[0021] 3-2: The feature reconstruction module outputs aligned multi-scale feature maps, which are used to reconstruct the input image through reconstruction loss, thus completing feature space alignment;

[0022] 3-3: The new state generation module embeds multi-scale feature maps and state indices. Generate an open-state image as input. The generation accuracy of the model is optimized by generating loss; the value of the state index embedding is adjusted. To realize other intermediate state images of an object The generation;

[0023] 3-4: The 3D perception joint estimation module receives the closure state image input in step 1. Compared with the open state image generated in step 3-3 The process involves upgrading the point cloud to 3D and calculating displacement differences, extracting filtered motion seeds, estimating joint parameters in the 3D coordinate space using an ensemble prediction paradigm, and optimizing prediction accuracy using classification loss and regression loss. The joint parameters include: joint type, axis origin, axis direction, and range of motion.

[0024] 3-5: Joint reconstruction loss, synthesis loss and joint estimation loss are used to train the relevant network end-to-end, and self-supervised optimization is achieved by utilizing cross-state geometric consistency and pixel consistency.

[0025] Step 4: Feedforward Inference

[0026] A single closed-state RGB image is input into the trained network. The network sequentially extracts features through the feature reconstruction module, generates a new state image through the new state generation module, and outputs joint 3D parameters through the 3D perception joint estimation module. The new state generation result and the joint 3D parameter set are output in real time in a feedforward manner.

[0027] This invention requires only a single closed-state image as input, without the need for category priors or additional annotations. Through end-to-end feedforward inference, it simultaneously generates new object states and accurately estimates joint 3D parameters. It has advantages such as real-time efficiency, strong generalization, and good geometric consistency, and can effectively meet the practical application needs of embodied intelligence and robot interaction scenarios in open worlds. Attached Figure Description

[0028] Figure 1 is a flowchart of the network reconstruction process;

[0029] Figure 2 is a flowchart of the network generation process;

[0030] Figure 3 shows the flowchart of the joint estimation network;

[0031] Figure 4 shows the overall flowchart of the joint parameter estimation method;

[0032] Figure 5 is a flowchart of feedforward inference;

[0033] Figure 6 is a flowchart of the method of the present invention;

[0034] Figure 7 shows the experimental results of generating new states;

[0035] Figure 8 shows the experimental results of generating new states.

[0036] Figure 9 shows the results of the joint estimation experiment.

[0037] Figure 10 shows the experimental results of joint estimation. Detailed Implementation

[0038] To facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0039] A novel method for generating new states and estimating joint 3D parameters of a 3D articulated object from a single static image without prior knowledge is proposed. This method employs implicit feature dynamic cross-state mapping representation and uses a feedforward approach to achieve real-time generation of new states and estimation of joint 3D parameters from a single closed-state image. The method includes the following steps:

[0040] Step 1: Dataset Preparation and Preprocessing

[0041] We collected a large number of single-image RGB images of articulated objects in their closed states, along with 3D point clouds and meshes. These images cover eight or more categories of everyday objects, including drawers, laptops, microwave ovens, ovens, refrigerators, storage furniture, closets, and trash cans, containing over 27,000 multi-angle samples of different instances. All images were adjusted to a resolution of 224×224, and pixel values ​​were normalized to the [0,1] range. This indicates a single, closed input image, eliminating the need for additional annotations such as part masks and joint positions.

[0042] Step 2: Network Architecture Construction

[0043] Construct an end-to-end network consisting of three main functional modules:

[0044] 2-1: Feature reconstruction module, consisting of a frozen pre-trained visual encoder and a trainable multi-level decoder, is used to extract multi-scale features of the image and align the feature space through the reconstruction task;

[0045] 2-2: The new state generation module is built on the Transformer generator based on adaptive layer normalization (AdaLN). By injecting state index information, it generates an open state image with the same geometry and texture as the input image.

[0046] 2-3: 3D perception joint estimation module, which includes point cloud uplifting submodule and ensemble prediction submodule. It realizes end-to-end estimation of joint parameters by uplifting 2D image pairs into 3D point clouds and extracting motion seeds.

[0047] Step 3: Joint Training

[0048] 3-1: Input a single closed-state RGB image, perform data augmentation, and then input it into the feature reconstruction module;

[0049] 3-2: The enhanced image is input into the feature reconstruction module. The frozen pre-trained visual encoder outputs aligned multi-scale feature maps, and the trainable multi-level decoder outputs the reconstructed image based on the multi-scale feature maps. Feature space alignment is optimized through reconstruction loss;

[0050] 3-3: The no-priority new state generation module receives multi-scale feature maps and state index embedding maps, which are then modulated by AdaLN and processed by visual... Figure 1 After consistency enhancement, an open-state image is generated. Image fidelity is optimized by generating loss;

[0051] 3-4: The 3D perception joint estimation module receives images of the closed state. With open state image The corresponding 3D point cloud is generated through the point cloud enhancement submodule, the cross-state displacement difference is calculated and the filtered effective motion seeds are extracted, and then the joint parameters are output through the ensemble prediction paradigm, including: joint type, axis origin, axis direction and range of motion. The prediction accuracy is optimized by using classification loss and regression loss.

[0052] 3-5: Joint reconstruction loss, generation loss and joint estimation loss are used to train the entire network end-to-end. Self-supervised optimization is achieved by utilizing cross-state geometric consistency and pixel consistency to ensure that each module works together.

[0053] Step 4: Feedforward Inference

[0054] A single closed-state RGB image is input into the trained network. The network sequentially extracts features through the feature reconstruction module, generates a new state image through the new state generation module, and outputs joint 3D parameters through the 3D perception joint estimation module. The new state generation result and the joint 3D parameter set are output in real time in a feedforward manner.

[0055] Example

[0056] Referring to Figure 1, the present invention first uses a large number of closed state images of articulated objects to train a feature reconstruction module, providing robust feature representation for subsequent synthesis and generation of new states;

[0057] A0: For the input closed-state RGB image The input format is uniformly resized to a resolution of 224×224, and the pixel values ​​are normalized to the [0,1] range.

[0058] A1: Using a frozen pre-trained visual encoder Extract the input image Multi-scale semantic features. Specifically, the encoder patches the image into a sequence of tokens, extracts multi-level image features, achieves feature learning without explicit camera extrinsic parameters, and avoids viewpoint drift during the synthesis and generation process;

[0059] A2: Employs a multi-level decoder The decoder receives multi-scale features from the encoder output and performs image reconstruction, achieving feature space alignment. The decoder extracts features from multiple intermediate layers of the DINOv2 encoder to reconstruct the image. ;

[0060] A3: Using the reconstructed image With the currently input image Self-supervised training is employed using a weighted sum of L1 loss and LPIPS (perceptual loss) to enable the deep network to learn pixel-level and semantically perceptual information, thereby ensuring pixel-level reconstruction accuracy. Reconstruction Loss The formula is as follows:

[0061]

[0062] in, For L1 loss, For perceptual loss. Considering the consistency of image information, this part of the training continues until... Below 1e-5.

[0063] See Figure 2 The present invention is based on the Transformer synthesis and generation module with adaptive layer normalization (AdaLN), which injects state index information to generate an open state image with the same geometry and texture as the input closed state image.

[0064] B0: Define the state index Indicates the degree of opening and closing of an articulated object, where Input image corresponding to closed state , Image generated for a target in a fully open state intermediate state, such as etc., used to assist training. State indexing. Sine embedding is performed to convert it into a high-dimensional feature vector, which facilitates fusion with image features;

[0065] B1: The multi-scale image features output by the feature reconstruction module are concatenated with the sinusoidal embedding of the state index, and then mapped to the latent state dimension through a linear projection layer to obtain the fused features. State index information is injected using AdaLN modulation. To achieve the fusion feature statistics The dynamic adjustment of AdaLN modulation is shown in the following formula:

[0066]

[0067] in Ratio parameters and bias parameters The modulated features are obtained by regression from the state index embedding using a 2-layer MLP, ensuring that the modulated features can accurately reflect the opening and closing degree corresponding to the state index.

[0068] B2: The new state generation module uses the adaptively adapted features as input to the decoder pre-trained by the feature reconstruction module. Generate target state image Or other intermediate state images;

[0069] B3: With the encoder and decoder parameters frozen, only the deep network parameters of this new state generation module are optimized. MSE loss is used as the synthesis generation loss, based on the closed-state input image. Constraints to generate images Image of a real open state Pixel-level error, generating loss The formula is as follows:

[0070]

[0071] in, For the new state generation module, For encoder, For feature reconstruction module, This is the state index for random sampling.

[0072] See Figure 3 This invention utilizes a 3D sensing joint estimation module to receive images of closed states. With the synthesized open-state image Extract motion vectors and estimate joint parameters;

[0073] C0: Employs a pre-trained 3D reconstruction model to convert 2D images into digital representations. Each was upgraded to a dense 3D point cloud of equal size. and And the geometric confidence of each point. This process does not require camera extrinsic parameters. By loading pre-trained deep network weights, it achieves end-to-end upscaling from 2D images to 3D point clouds, solving the joint localization error problem caused by depth blur in 2D images;

[0074] C1: Calculate the cross-state displacement difference of the 3D point cloud, and use the 3D coordinates of the point cloud in the region of significant displacement as the initial motion vector, i.e. the potential region of joint motion;

[0075] C2: Based on the initial motion vectors, 3D position embeddings are added. For each motion vector, 16 joint hypotheses are predicted individually. The number of hypotheses is always greater than the maximum number of joints in the standard dataset, adapting to jointed objects with different topologies. For each set of 16 joint hypotheses, four types of parameters are predicted for the corresponding joints: joint type, axis origin, axis direction, and range of motion.

[0076] C3: A weighted sum of classification and regression losses is used as the joint estimation loss, combined with the Hungarian matching algorithm to achieve accurate matching between joint assumptions and actual joints. Joint type matching loss. The formula is as follows:

[0077]

[0078] and joint prediction loss The formula is as follows:

[0079]

[0080] in, Cross-entropy loss is used to optimize the accuracy of joint type classification. As an indicator function, the regression loss is calculated only for the true joint type, i.e., for joints of rotation and translation; for the , Real joint information, The optimal permutation obtained from Hungarian matching is used to match the predicted joint coordinates. Compared with the actual joint coordinates Distance, predicted joint direction Angle with the actual joint direction and the predicted range of motion of the joints. Compared to the actual range of motion of joints The gap.

[0081] See Figure 5 With only a single closed-state RGB image as input, the deep network with only three modules is needed for inference to generate multiple open-state images and related joint parameter information in real time.

[0082] D0: Preprocess the single closed-state RGB image to be processed according to the standards of stage A0;

[0083] D1: Using an encoder Extract multi-scale features from the preprocessed image. Input the features into the new state generation module. ,injection Discrete state indices, after AdaLN modulation, are processed by the feature reconstruction module. Generate a fully open state image ;

[0084] D2: Pair 2D images Input the 3D perception joint estimation module and output the joint parameters.

[0085] See Figure 4 The overall flowchart of this invention shows that, firstly, steps two and three construct three functional modules: feature reconstruction, new state generation, and 3D perception joint estimation. Then, step four involves training these modules. These three modules learn cross-state geometric and semantic consistency in a self-supervised manner, achieving feature space alignment and accurate extraction of joint motion information, thus improving the fidelity of open state generation and the accuracy of joint parameter estimation. During inference, the feature reconstruction module, new state generation module, and 3D perception joint estimation module output the generated open state image and a complete set of joint parameters using a feedforward approach. Based on the input of a single closed-state RGB image, without any auxiliary information, the invention achieves the generation of open states and accurate estimation of 3D joint parameters for 3D articulated objects.

[0086] For the dataset partitioning module, please refer to [link / reference]. Figure 6This embodiment uses the academic dataset PartNet-Mobility, which includes more than 10 categories of common articulated objects such as drawers, cabinets, laptops, and microwave ovens, with each object involving up to 11 joints. The training data includes 27,000 objects, and the test data includes 347 objects. Each object is provided with 24 different RGB image samples from various viewing angles. This partitioning ensures that no two objects are repeated. This embodiment uses an RTXA100 GPU as the hardware support.

[0087] Network construction involves building a reconstruction network (feature reconstruction module), a generation network (new state generation module), and an estimation network (3D perception joint estimation module) based on steps two and three.

[0088] Basic training, see reference Figure 4 The model takes a single RGB image as input, and the reconstruction and generation networks learn image features in a self-supervised manner to generate new state images. The joint estimation network predicts various parameters of the object's joints. The specific implementation steps are as follows:

[0089] 1) A0 performs image processing;

[0090] 2) A1 extracts image features, A2 reconstructs the network to output the reconstructed image, and A3 uses the pixel and semantic losses of the reconstructed image and the current real image for self-supervised training.

[0091] 3) Input B0 into the image features extracted by A1 in 2), use B1 and B2 to generate the image features of the new target state through the network, and then use the reconstructed network in 2) to output the new state generated image.

[0092] 4) C0 aggregates the single input image and the generated open state image pair to reconstruct the corresponding point cloud; C1 calculates the cross-state displacement difference of the 3D point cloud and gives the initial motion vector; C2 predicts the joint information in 3D coordinates; C3 calculates the similarity loss between the predicted joint information and the real joint information for optimization.

[0093] This embodiment was tested on PartNet-Mobility; see [link / reference]. Figure 5 During testing, a single RGB image is input, the model performs feedforward inference, generates a new state viewpoint, and predicts joint information. The effectiveness and real-time performance of the invention are demonstrated by comparing the similarity (PSNR, SSIM, LPIPS, CLIP-T, FVD) between the generated new state image and the real new state image with a benchmark method, as well as comparing the total accuracy and minor accuracy of joint group predictions. The comparison results with the image generation benchmark method are shown below. Figure 7 This invention surpasses the benchmark method in all similarity metrics, proving that it can accurately generate new object states; the comparison results with the benchmark joint prediction method are shown below. Figure 9 The present invention surpasses the benchmark method in terms of accuracy across all parameters, proving that the present invention can accurately predict the joint parameters of an object.

[0094] See Figure 7 Indicator Results and Figure 8 The image results demonstrate that this invention achieves higher quality in generating new states. Other benchmark methods exhibit significant artifacts, while this invention significantly reduces artifacts and generates clearer images from new perspectives, proving that this invention can achieve higher generation quality. (See also...) Figure 9 Indicator Results and Figure 10 The image results show that the present invention achieves higher quality in joint parameter prediction. Other benchmark methods have obvious errors, while the present invention significantly reduces the errors and predicts more accurate joint information, proving that the present invention can achieve higher prediction quality.

Claims

1. A method for generating the state and estimating the joints of a hinged object without prior knowledge, comprising the following steps: Step 1: Dataset Preparation and Preprocessing Collect single RGB images, 3D point clouds, and mesh data of articulated objects in their closed state. Each object category contains different instances within that category. Define the image resolution and pixel normalization range. This represents a single, closed-state input image. This represents a single image in an open state; Step 2: Network Architecture Construction Construct an end-to-end network consisting of three main functional modules: 2-1: Feature reconstruction module, which uses a frozen pre-trained visual encoder and a trainable multi-level decoder to extract multi-scale image features and align the feature space through reconstruction task; 2-2: The new state generation module uses a Transformer generator based on Adaptive Layer Normalization (AdaLN) to inject state index information and generate open state images; 2-3: The 3D perception joint estimation module consists of a point cloud lifting submodule and an ensemble prediction submodule, which is used to extract motion seeds to achieve end-to-end estimation of joint parameters; Step 3: Joint training, including: 3-1: Input a single closed-state RGB image, perform data augmentation, and then input it into the feature reconstruction module; 3-2: The feature reconstruction module outputs aligned multi-scale feature maps, which are used to reconstruct the input image through reconstruction loss, thus completing feature space alignment; 3-3: The new state generation module embeds multi-scale feature maps and state indices. Generate an open-state image as input. The generation accuracy of the model is optimized by generating loss; the value of the state index embedding is adjusted. To realize other intermediate state images of an object The generation; 3-4: The 3D perception joint estimation module receives the closure state image input in step 1. Compared with the open state image generated in step 3-3 The process is improved to 3D point cloud and displacement differences are calculated. Filtered motion seeds are extracted, and joint parameters in 3D coordinate space are estimated through ensemble prediction paradigm. Classification loss and regression loss are used to optimize prediction accuracy. 3-5: Joint reconstruction loss, synthesis loss and joint estimation loss are used to train the relevant network end-to-end, and self-supervised optimization is achieved by utilizing cross-state geometric consistency and pixel consistency. Step 4: Feedforward Inference A single closed-state RGB image is input into the trained network. The network sequentially extracts features through the feature reconstruction module, generates a new state image through the new state generation module, and outputs joint 3D parameters through the 3D perception joint estimation module. The new state generation result and the joint 3D parameter set are output in real time in a feedforward manner.

2. The method for generating the state and estimating the joints of a hinged object without prior knowledge, as described in claim 1, is characterized in that... The image resolution is calibrated to 224×224, and the pixel value normalization range is [0,1].

3. The method for generating the state and estimating the joints of a hinged object without prior knowledge, as described in claim 1, is characterized in that... The joint parameters include: joint type, axis origin, axis direction, and range of motion.

4. The method for generating the state of a hinged object and estimating its joints without prior knowledge, as described in claim 1, is characterized in that... Each object category contains different instances. For each object category, multiple different instances are collected. These instances differ in size, material, shape, structure, and component combination. Instance samples from multiple angles and states are provided.

Images