A method and system for three-dimensional modeling of articulated objects

Abstract

The application provides a kind of articulated object three-dimensional modeling method and system, the method comprises: constructing the first Gaussian point cloud set of articulated object in its high visibility state and the second Gaussian point cloud set in its low visibility state;Based on the first Gaussian point cloud set and the second Gaussian point cloud set, a plurality of components of the articulated object are constructed, the initial motion parameters of each component are configured based on the high visibility state and the low visibility state of the articulated object, and the initial mask vector containing the probability of each Gaussian ball belonging to each component is configured for each Gaussian ball in the first Gaussian point cloud set;Based on the initial motion parameters of each component and the initial mask vector of each Gaussian ball, geometric modeling and motion modeling are alternately executed in multiple rounds to jointly optimize the mask vector of the first Gaussian point cloud and the motion parameters of each component to convergence;Based on the updated first Gaussian point cloud set, the mask vector of each Gaussian ball and the motion parameters of each component, a three-dimensional model of the articulated object is constructed.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to the field of 3D reconstruction technology, and more specifically to a method and system for 3D modeling of articulated objects. Background Technology

[0002] In applications such as embodied intelligence, robotics, and virtual / augmented reality, constructing high-fidelity and interactive digital assets is a crucial foundation for intelligent agents to understand and interact with the real world. Among various interactive objects, articulated objects (such as cabinet doors, laptops, and scissors) are particularly common, due to their complex structures and high interaction difficulty. Unlike typical rigid objects, articulated objects consist of multiple movable parts, and the movement between these parts causes significant overall geometric deformation, resulting in the object exhibiting a variety of different states. Therefore, existing technical solutions for modeling articulated objects all aim to reconstruct high-quality geometric structures at the part level using multi-view, multi-state observation data (such as RGB-D image sequences) and simultaneously estimate accurate joint motion parameters.

[0003] 3D reconstruction schemes for articulated objects typically address the problem of recovering physical properties from discrete observation states. A key challenge lies in the coupling between component geometry and joint motion parameters. Accurate component segmentation often relies on the analysis of the object's motion behavior, while precise motion parameter estimation, in turn, depends on reliable component segmentation results. Existing 3D reconstruction schemes for articulated objects mainly fall into two categories. The first is based on clustering priors and geometry-motion joint optimization. This scheme employs a coarse-to-fine framework, utilizing unsupervised clustering and edge optimization to refine component segmentation, and reconstructs the object by jointly optimizing the component-level geometry and motion states. However, this scheme suffers from the following drawbacks: First, this clustering-based prior is ill-suited for complex articulated bodies, especially when different dynamic components are spatially close or occluded, easily leading to segmentation errors or component adhesion. Second, this method attempts to perform strongly coupled joint optimization of geometry and motion in a single step. When dealing with complex multi-jointed objects, the optimization space is highly non-convex, often facing convergence instability and easily getting trapped in local minima, resulting in reconstruction failure. The second approach is based on fine-tuning the SAM prior and phased optimization. This approach fine-tunes the segmentation base model (such as SAM) on a specific dataset to accurately obtain dynamic component segmentation masks from various perspectives, thereby gaining a strong component segmentation prior. Simultaneously, the optimization process typically employs a phased (or decoupled) strategy, i.e., completing static geometric reconstruction and component segmentation first, followed by motion parameter estimation. However, this existing technique has the following drawbacks: First, this method heavily relies on a prior model fine-tuned with a large amount of specific data, which not only increases training costs but also severely limits the method's generalization ability to unseen categories of objects or out-of-domain data. Second, the phased approach of performing geometric reconstruction and motion estimation severs the physical connection between the two, easily leading to the accumulation of geometric errors from the previous stage and their propagation into subsequent motion estimation, resulting in insufficient utilization of inter-state information and ultimately affecting the physical consistency of the model and the accuracy of motion estimation.

[0004] In summary, existing 3D reconstruction schemes for articulated objects suffer from several drawbacks: difficulty adapting to complex structures; poor generalization ability of methods based on fine-tuning large models for unseen object categories; and unstable optimization convergence and easy error accumulation due to strong coupling between geometric reconstruction and motion estimation. Therefore, there is an urgent need for a 3D modeling method that can achieve high-fidelity geometric reconstruction and accurate motion parameter estimation for complex articulated objects without requiring specific data fine-tuning.

[0005] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solutions of the present invention, and does not imply that the relevant information is necessarily prior art. In the absence of evidence indicating that the relevant information was disclosed before the filing date of this invention, the relevant information should not be considered prior art. Summary of the Invention

[0006] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for three-dimensional modeling of articulated objects.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] According to a first aspect of the present invention, a method for three-dimensional modeling of an articulated object is proposed, the method comprising: S1, acquiring first multi-view RGB-D observation data of the articulated object in its high visibility state and second multi-view RGB-D observation data of the articulated object in its low visibility state, and constructing a first Gaussian point cloud set based on the first multi-view RGB-D observation data, and constructing a second Gaussian point cloud set based on the second multi-view RGB-D observation data, wherein the Gaussian point cloud set is composed of multiple Gaussian spheres; S2, constructing multiple components of the articulated object based on the first Gaussian point cloud set and the second Gaussian point cloud set, each component comprising multiple Gaussian spheres in the first Gaussian point cloud set, and configuring initial motion parameters for each component based on the high visibility state and the low visibility state of the articulated object, wherein the initial motion parameters for each component are configured based on the first Gaussian point cloud set. Each Gaussian sphere configuration includes an initial mask vector containing the probability that the Gaussian sphere belongs to each component; S3, based on the initial motion parameters of each component and the initial mask vector of each Gaussian sphere, perform multiple rounds of alternating geometric modeling and motion modeling to jointly optimize the mask vector of the first Gaussian point cloud and the motion parameters of each component until convergence. Specifically, performing geometric modeling involves fixing the motion parameters of all components and iteratively updating the properties of the first Gaussian point cloud set and the mask vector of each Gaussian sphere using a first preset method. Performing motion modeling involves fixing the mask vectors of all Gaussian spheres and iteratively updating the properties of the first Gaussian point cloud set and the motion parameters of each component using a second preset method; S4, construct a 3D model of the articulated object based on the first Gaussian point cloud set updated in S3, the mask vector of each Gaussian sphere, and the motion parameters of each component.

[0009] Optionally, S2 includes: S21, performing voxelization on the first Gaussian point cloud set to obtain a first voxel occupancy set, and performing voxelization on the second Gaussian point cloud set to obtain a second voxel occupancy set; S22, obtaining the difference regions between the first voxel occupancy set and the second voxel occupancy set, and the same regions between the first voxel occupancy set and the second voxel occupancy set, using a preset method; S23, constructing multiple components of the articulated object based on the difference regions and the same regions, including: performing a clustering operation on the extracted difference regions to obtain multiple voxel connected components and selecting voxel connected components whose volume exceeds a preset volume threshold, and using each selected voxel connected component as a movement of the articulated object. The components are obtained by generating multiple moving components. Each voxel connected component contains multiple Gaussian spheres in the first Gaussian point cloud set. The Gaussian spheres contained in the same extracted region are used to form the static components of the articulated object. All moving components and static components constitute all components of the articulated object. S24: Configure the initial motion parameters of each component in the first Gaussian point cloud set based on the high visibility state and low visibility state of the articulated object. S25: Configure a mask vector for each Gaussian sphere in the first Gaussian point cloud set based on all the obtained components. Each dimension of the mask vector corresponds to a component, and the value in each dimension of the mask vector represents the probability that the Gaussian sphere corresponding to the mask vector belongs to the component corresponding to that dimension.

[0010] Optionally, the voxel difference function is:

[0011]

[0012] in, This indicates the region of difference between the first voxel occupancy set and the second voxel occupancy set. Indicates the set occupied by the first voxel. Indicates that the second voxel occupies the set. This is a morphological dilation operation.

[0013] Optionally, in S3, the first preset method is as follows: A pre-trained visual basic segmentation model is pre-acquired, wherein the pre-trained visual basic segmentation model is a pre-trained model that takes the RGB observation image of the object as input and multiple segmented region images as output, wherein each segmented region image highlights its corresponding segmented region and masks other regions; based on the acquired pre-trained visual basic segmentation model, the following steps are iteratively executed: acquiring the RGB observation image of the articulated object in its high-visibility state, and using the visual basic segmentation model to generate multiple segmented region images based on the acquired RGB observation image; constructing a two-dimensional Gaussian image of each component using a preset method based on the mask vector of the Gaussian sphere within each component, wherein the value of each pixel in the two-dimensional Gaussian image of each component represents the probability that the pixel belongs to that component; sequentially... Calculate the similarity between the 2D Gaussian map of each component and the image of each segmented region; based on the calculated similarity, assign each segmented region image to the component with the highest similarity; for each component, aggregate all segmented region images assigned to that component to obtain the weakly supervised mask image of that component; render the rendered image and the 2D Gaussian probability map based on the updated Gaussian point cloud set; calculate the reconstruction loss between the rendered image and the real observed image, the semantic segmentation loss between the 2D segmentation probability map and the weakly supervised mask, and the consistency loss of the Gaussian sphere in the 3D spatial neighborhood using a first preset loss function; jointly determine the total geometric modeling loss based on the calculated reconstruction loss, semantic segmentation loss, and consistency loss, and update the self-attributes of the first Gaussian point cloud and the mask vector of each Gaussian sphere with the total geometric modeling loss.

[0014] Optionally, the preset method is to determine the value of each pixel in the two-dimensional Gaussian plot of each component in the following manner:

[0015]

[0016] in, Indicates the index of the pixel. Indicates the index of the component. Indicates the first The first part in the two-dimensional Gaussian plot of the component The value of each pixel. Indicates projection to the first The first pixel on the The number of Gaussian balls in each component. Indicates the first The Gaussian sphere belongs to the first... The probability of each component and Indicates the index of the Gaussian sphere. and The numbers represent the order of the numbers. The Gaussian sphere and the first The opacity of a Gaussian sphere and The numbers represent the order of the numbers. The Gaussian sphere and the first The Gaussian ball in the... The projected value on each pixel.

[0017] Optionally, the first preset loss function is:

[0018]

[0019] in, Represents the geometric modeling loss. The first-state reconstruction loss represents the reconstruction error between the global rendered image obtained by directly rendering the first Gaussian point cloud set in a high-visibility state and the overall true observed image of the hinged object in that high-visibility state. Let represent the second-state reconstruction loss, which indicates the reconstruction error between the global rendered image obtained by performing a rigid body transformation on the first Gaussian point cloud set based on the motion parameters of each component and the overall true observed image of the hinged object in a low-visibility state. The semantic segmentation loss is represented by the error between the 2D part segmentation probability map rendered from the first Gaussian point cloud set and the weakly supervised mask image generated from the output of the pre-trained visual base segmentation model. This represents the spatial consistency loss, which is the difference in mask vectors between adjacent Gaussian spheres in three-dimensional space. and This represents the preset weight parameters.

[0020] Optionally, in S3, the second preset method is as follows: fix the mask vector of each Gaussian sphere in the first Gaussian point cloud set; based on the mask vector of each Gaussian sphere, determine the component to which the Gaussian sphere belongs, and obtain the motion parameters corresponding to the component to obtain the motion parameters of each moving component; based on all the obtained motion parameters, perform a rigid body transformation on the first Gaussian point cloud set to obtain a rendered image in a low visibility state; render the first Gaussian point cloud set in a high visibility state (normal state) and the first Gaussian point cloud set after the rigid body transformation, respectively, and combine them with their corresponding real observation images to determine the motion modeling loss using a second preset loss function; update the self-attributes of the first Gaussian point cloud set and the motion parameters of each component based on the motion modeling loss.

[0021] Optionally, the preset rigid body motion representation method can be any one of the dual quaternion method, the Lie algebra rotation matrix method, or the Euler angle and translation vector combination method.

[0022] Optionally, the second preset loss function is:

[0023]

[0024] in, Indicates motion modeling loss, The first-state reconstruction loss represents the reconstruction error between the rendered image obtained from the first Gaussian point cloud set under high visibility conditions and the actual observed image of the hinged object under the same high visibility conditions. The second state reconstruction loss is the reconstruction error between the rendered image obtained by performing a rigid body transformation on the first Gaussian point cloud set based on the motion parameters a of each component and the actual observed image of the hinged object in a low visibility state.

[0025] According to a second aspect of the present invention, a three-dimensional modeling system for articulated objects is proposed for implementing the method described in the first aspect of the present invention. The system comprises: a data acquisition and basic representation construction module, configured to acquire first multi-view RGB-D observation data of the articulated object in its high-visibility state and second multi-view RGB-D observation data of the articulated object in its low-visibility state, and construct a first Gaussian point cloud set based on the first multi-view RGB-D observation data, and construct a second Gaussian point cloud set based on the second multi-view RGB-D observation data, wherein the Gaussian point cloud set is composed of multiple Gaussian spheres; and a coarse reconstruction module, configured to extract differential voxels between the first and second Gaussian point cloud sets and perform connectivity clustering operations to construct multiple components of the articulated object, and configure initial motion parameters for each component based on the high-visibility and low-visibility states of the articulated object. ; and, configure a mask vector for each Gaussian sphere in the first Gaussian point cloud set, wherein the mask vector contains the probability that the Gaussian sphere belongs to each component; a 3D model construction module is used to perform geometric modeling and motion modeling alternately in multiple rounds based on the initial motion parameters of each component and the initial mask vector of each Gaussian sphere, so as to jointly optimize the self-properties of the first Gaussian point cloud, the mask vector of each Gaussian sphere and the motion parameters of each component until convergence; and construct a 3D model of the articulated object based on the finally obtained first Gaussian point cloud set, the mask vector of each Gaussian sphere and the motion parameters of each component, wherein performing geometric modeling involves fixing the motion parameters of all components and iteratively updating the self-properties of the first Gaussian point cloud and the mask vector of each Gaussian sphere, and performing motion modeling involves fixing the mask vector of all Gaussian spheres and iteratively updating the self-properties of the first Gaussian point cloud and the motion parameters of each component.

[0026] Compared with the prior art, the advantages of the present invention are as follows:

[0027] This invention centers on a Gaussian splash-based geometry-motion alternating optimization scheme, decomposing the modeling of articulated objects into sub-processes that alternate between geometric and motion modeling. This effectively decouples the highly coupled variables of component attribution and joint motion parameters, reducing optimization difficulty, avoiding local minima, and promoting stable convergence of the optimization. It significantly improves the geometric fidelity and motion estimation accuracy of complex multi-joint articulated object reconstruction. Simultaneously, it employs a similarity-based mask aggregation mechanism to address the issues of inconsistent granularity and lack of consistency across multiple perspectives. Component-level masks are extracted as weakly supervised signals to guide 3D model optimization, further improving component segmentation accuracy while ensuring generalization. Attached Figure Description

[0028] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0029] Figure 1 This is a schematic diagram of a hinged object in a highly visible state according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of a hinged object in a low-visibility state according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram illustrating the steps of a three-dimensional modeling method for an articulated object according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of geometric modeling according to an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of a three-dimensional modeling method for articulated objects according to an embodiment of the present invention;

[0034] Figure 6 This is a flowchart illustrating a three-dimensional modeling method for articulated objects according to an embodiment of the present invention.

[0035] Figure 7 This is a schematic diagram of the experimental results of comparative experiment 1 according to an embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the experimental results of comparative experiment 2 according to an embodiment of the present invention;

[0037] Figure 9 This is a schematic diagram of the experimental results of comparative experiment 2 according to an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] As mentioned in the background section, existing 3D reconstruction schemes for articulated objects are mainly divided into two categories: clustering prior and geometry-motion joint optimization schemes, and methods based on fine-tuned SAM prior and staged optimization. The former reconstructs the object by jointly optimizing the geometric and motion states at the component level. This scheme attempts to simultaneously optimize the strongly coupled geometry and motion to convergence in one step. However, when dealing with complex multi-joint objects, the optimization space is highly non-convex, often facing convergence instability and easily getting trapped in local minima, leading to reconstruction failure. The latter relies heavily on prior models fine-tuned with a large amount of specific data. This not only increases training costs but also severely limits the method's generalization ability to unseen categories of objects or out-of-domain data. Furthermore, the strategy of performing geometric reconstruction and motion estimation in stages severs the physical connection between the two, which can easily lead to the accumulation of geometric errors in the previous stage and their transmission to subsequent motion estimation, resulting in insufficient utilization of inter-state information and ultimately affecting the physical consistency of the model and the accuracy of motion estimation. Therefore, existing 3D reconstruction schemes for articulated objects have problems such as difficulty in adapting to complex structures, poor generalization ability of methods based on fine-tuning large models for unseen categories of objects, and unstable optimization convergence and easy accumulation of errors due to strong coupling between geometric reconstruction and motion estimation.

[0040] After in-depth research on high-fidelity 3D reconstruction and motion estimation of complex articulated objects, the inventors found that existing technologies often suffer from defects such as fragmented reconstruction geometry, large deviations in motion parameter estimation, and unstable convergence when dealing with objects with complex structures (such as multiple closely adjacent parts) or those that exceed the training data distribution. The core reasons for the defects of existing technologies are as follows: (1) It is difficult to achieve both the accuracy and generalization of geometric priors: Traditional unsupervised clustering-based methods are difficult to distinguish between spatially closely adjacent or textured parts; while methods based on fine-tuning the 2D segmentation base model (such as fine-tuning SAM) can segment more accurately in specific categories, but the fine-tuning process causes the model to lose its ability to generalize to zero samples of unseen objects, and it is difficult to directly provide consistent supervision signals in 3D space. (2) The strong coupling between geometry and motion optimization leads to convergence difficulties: The geometry of the parts is highly coupled with the motion parameters (geometry depends on motion analysis, and motion depends on geometric segmentation). Existing "joint optimization" strategies are prone to getting trapped in local minima due to the highly non-convex solution space, while "phased optimization" strategies (geometry first, motion later) sever the physical connection, causing the geometric error of the previous stage to accumulate and worsen the subsequent motion estimation.

[0041] To address the aforementioned technical challenges, the inventors, after research, proposed the following improvement approach: First, regarding the difficulty of simultaneously achieving geometric prior accuracy and generalization, although part segmentation is inherently a 3D segmentation task, the development of current 3D segmentation models is relatively lagging, while general 2D vision models (such as the original SAM) are extremely powerful in both segmentation and generalization capabilities. To avoid the loss of generalization caused by fine-tuning, the inventors proposed a scoring-based fine-grained mask aggregation mechanism, leveraging the characteristic that Gaussian splashing can render coarse masks consistent across multiple viewpoints. This method directly utilizes masks generated by the un-fine-tuned 2D segmentation model, and aggregates the Gaussian mask and SAM mask by voting based on their overlap, thereby achieving high-quality, 3D consistent part-level supervised mask acquisition under zero-sample conditions. Secondly, in response to the optimization problem, the inventors proposed a geometry-motion alternating optimization scheme, which decomposes the modeling of articulated objects into sub-processes that alternate between geometric modeling and motion modeling. This effectively decouples the two highly coupled variables of component attribution and joint motion parameters, and decomposes the complex non-convex optimization problem into two more easily convergent sub-problems. Thus, while ensuring physical consistency, it significantly improves the convergence stability and reconstruction accuracy of the model under complex structures.

[0042] Before describing the embodiments of the present invention in detail, some of the terms used therein are explained as follows:

[0043] High visibility state (canonical state):

[0044] For articulated objects, this refers to a state in which the moving parts of the object are in a relatively unfolded, open, or dispersed position. In this state, the articulated object exhibits less self-occlusion, exposing a more complete geometric topology and surface texture, and serves as a reference state for three-dimensional geometric initialization in this invention; see attached figure. Figure 1 As shown in the image, the USB flash drive, utility knife, and chair are displayed in an exemplary high-visibility state.

[0045] Low visibility state (target state): This refers to a state where, relative to the high visibility state, at least one moving part of the articulated object has undergone relative movement, and is in a relatively closed, folded, or increasingly occluded position. In this state, the articulated object exhibits more self-occlusion compared to the high visibility state, and in this invention, it serves as the target state for driving component movement and evaluating motion modeling accuracy; such as... Figure 2 As shown in the image, the USB flash drive, utility knife, and chair are displayed in an exemplary low-visibility state.

[0046] According to one embodiment of the present invention, a method for three-dimensional reconstruction of articulated objects is proposed, as shown in the attached figure. Figure 3 In summary, the method includes steps S1, S2, S3, and S4. To better understand the present invention, each step will be described in detail below with reference to the accompanying drawings and embodiments.

[0047] 1. Step S1

[0048] In step S1, first multi-view RGB-D observation data of the articulated object in its high visibility state and second multi-view RGB-D observation data of the articulated object in its low visibility state are obtained. A first Gaussian point cloud set is constructed based on the first multi-view RGB-D observation data, and a second Gaussian point cloud set is constructed based on the second multi-view RGB-D observation data. The Gaussian point cloud set is composed of multiple Gaussian spheres.

[0049] According to one embodiment of the present invention, in step S1, it is first necessary to acquire multi-view RGB-D observation data (RGB-D observation data refers to a type of image or sensor data that simultaneously contains color (red, green, and blue) and depth information) of the target articulated object in the target state and the canonical state. The present invention defines the state with higher visibility as the canonical state. The state with low visibility is defined as the target state. The input data includes multi-view RGB-D images. Depth map and the corresponding camera parameters Based on these acquired observational data, a two-dimensional Gaussian splatting technique was used to independently train the two states, resulting in two independent sets of Gaussian point clouds. and Each Gaussian sphere All include location Rotation matrix Attributes such as scale, opacity, and spherical harmonics are included. Specifically, to achieve explicit modeling of object parts, this embodiment uses a Gaussian set in a normalized state. In the middle, for each Gaussian sphere Explicitly bound to a learnable mask vector ,in Given the known number of movable parts, The value represents the probability distribution of the Gaussian sphere belonging to each component (including moving components, static components, and static background) under normal conditions. It should be noted that the process of constructing the Gaussian point cloud set (first Gaussian point cloud set) and the Gaussian point cloud set (second Gaussian point cloud set) of the articulated object under high visibility conditions and low visibility conditions based on the multi-view RGB-D observation data of the articulated object in step S1 is a well-known technique to those skilled in the art and will not be described in detail here.

[0050] II. Step S2

[0051] In step S2, multiple components of the articulated object are constructed based on the first Gaussian point cloud set and the second Gaussian point cloud set. Each component contains multiple Gaussian spheres in the first Gaussian point cloud set. The initial motion parameters of each component are configured based on the high visibility state and low visibility state of the articulated object. An initial mask vector containing the probability that the Gaussian sphere belongs to each component is configured for each Gaussian sphere in the first Gaussian point cloud set.

[0052] According to an embodiment of the present invention, step S2 includes:

[0053] S21. Perform voxelization on the first Gaussian point cloud set to obtain a first voxel occupancy set, and perform voxelization on the second Gaussian point cloud set to obtain a second voxel occupancy set; S22. Obtain the difference regions between the first voxel occupancy set and the second voxel occupancy set, and the common regions between the first voxel occupancy set and the second voxel occupancy set, using a preset method; S23. Construct multiple components of the articulated object based on the difference regions and the common regions, including: performing clustering operations on the extracted difference regions to obtain multiple voxel connected components and selecting voxel connected components whose volume exceeds a preset volume threshold, and using each selected voxel connected component as a moving component of the articulated object to obtain... Multiple moving parts, each voxel connected component containing multiple Gaussian spheres from a first Gaussian point cloud set, the Gaussian spheres contained in the extracted same region form the static parts of the articulated object, wherein all moving parts and static parts constitute all parts of the articulated object; S24, configure the initial motion parameters of each part in the first Gaussian point cloud set based on the high visibility state and low visibility state of the articulated object; S25, configure a mask vector for each Gaussian sphere in the first Gaussian point cloud set based on all obtained parts, wherein each dimension of the mask vector corresponds to a part, and the value in each dimension of the mask vector represents the probability that the Gaussian sphere corresponding to the mask vector belongs to the part corresponding to that dimension.

[0054] According to one embodiment of the present invention, in S21, firstly, a unified voxel space is established, and the first Gaussian point cloud set is... Second Gaussian point cloud set Voxelization is performed separately to obtain the voxel occupancy set of the first voxel occupancy set. Second voxel occupying set To accommodate objects of varying complexity, different voxel scales are used. Voxelization is performed, for example, using a coarser scale (e.g., v=0.1) for single-joint articulated objects and a finer scale (e.g., v=0.01) for complex multi-joint articulated objects. It should be understood that the voxelization scale directly determines the spatial resolution of the voxelization. A larger voxel scale means a larger space covered by a single voxel, resulting in a coarser resolution; conversely, a smaller voxel scale means a smaller space covered by a single voxel, resulting in a finer resolution. Therefore, the voxel scale needs to be adaptively selected according to the complexity of the articulated object. Simple structures with sparse components use a larger scale to compress the data volume; complex structures with dense components or small kinematic pairs use a smaller scale to ensure the distinguishability of adjacent components.

[0055] According to one embodiment of the present invention, in S22, the voxel difference function is calculated. To extract dynamic regions (difference regions), where, This indicates the region of difference between the first voxel occupancy set and the second voxel occupancy set. Indicates the set occupied by the first voxel Indicates that the second voxel occupies the set. This is a morphological dilation operation. The aforementioned voxel difference function can quickly eliminate minute noise and preserve the true motion region, improving the accuracy and stability of subsequent component clustering.

[0056] According to one embodiment of the present invention, in S23, connected component clustering is performed on the differential regions, and the largest volume is selected. Clusters (voxel connected components) as For the initial component, it should be understood that the algorithm used to cluster the differential regions is not limited to connected component analysis; K-Means, DBSCAN, or spectral clustering algorithms can also be used to divide the differential regions. Step S23 divides the differential regions into spatially independent motion units, filters the main components, and suppresses debris interference, providing a reliable initial partition for subsequent component-level modeling.

[0057] According to one embodiment of the present invention, in S24, the initial motion parameters of the component are the transformation matrix of the component from the normal state to the target state. Each moving component learns a transformation matrix, and each static component uses an identity matrix. This transformation matrix is ​​represented as a dual quaternion. Regarding the initialization of motion parameters, this embodiment employs a structure-aware strategy: the aspect ratio of the component is calculated using principal component analysis (PCA). If the aspect ratio is greater than 3.0, it is determined to be a planar structure (such as a door panel), and the rotation axis is approximated using the intersection line fitted to the planes containing the source and target voxels; otherwise, it is initialized with an identity transformation matrix, thereby obtaining the initial motion parameters. .

[0058] III. Step S3

[0059] In step S3, based on the initial motion parameters of each component and the initial mask vector of each Gaussian sphere, geometric modeling and motion modeling are performed alternately in multiple rounds to jointly optimize the mask vector of the first Gaussian point cloud and the motion parameters of each component until convergence. Specifically, geometric modeling involves fixing the motion parameters of all components and iteratively updating the properties of the first Gaussian point cloud set and the mask vector of each Gaussian sphere using a first preset method. Motion modeling involves fixing the mask vectors of all Gaussian spheres and iteratively updating the properties of the first Gaussian point cloud set and the motion parameters of each component using a second preset method.

[0060] It should be noted that the inherent properties of the Gaussian point cloud set refer to a set of optimizable mathematical parameters used to express and render the two-dimensional Gaussian in three-dimensional space, mainly including the following five items: (1) Center position: the center coordinates of the two-dimensional Gaussian disk in the three-dimensional world coordinate system. (2) Rotation orientation: determined by two principal tangent vectors ( (3) Scaling scale: Unlike the three-directional scaling of the three-dimensional Gaussian disk, the scaling scale of the two-dimensional Gaussian disk is a two-dimensional vector. , only used to control the variance of the two-dimensional disk in the two principal axis directions in its local tangent plane (i.e., the size and stretching of the disk); (4) Color features: represented by spherical harmonic function coefficients, used to calculate and render the color of the two-dimensional Gaussian under different viewing angles; (5) Opacity: the value is between 0 and 1, representing the transparency of the two-dimensional Gaussian primitive, used for transparency mixing calculation during ray tracing or rasterization rendering. It should be understood that the inherent properties of the Gaussian point cloud set are well known to those skilled in the art and will not be described in detail here.

[0061] According to an embodiment of the present invention, the first preset method is as follows: A pre-trained visual basic segmentation model is pre-acquired, wherein the pre-trained visual basic segmentation model is a pre-trained model that takes the RGB observation image of an object as input and multiple segmented region images as output, wherein each segmented region image highlights its corresponding segmented region and masks other regions; based on the acquired pre-trained visual basic segmentation model, the following steps are iteratively executed: acquiring the RGB observation image of the articulated object in its high-visibility state, and using the visual basic segmentation model to generate multiple segmented region images based on the acquired RGB observation image; constructing a two-dimensional Gaussian image of each component using a preset method based on the mask vector of the Gaussian sphere within each component, wherein the two-dimensional Gaussian image of each component... The value of each pixel in the image represents the probability that the pixel belongs to that component. The similarity between the two-dimensional Gaussian map of each component and each segmented region image is calculated one by one. Based on the calculated similarity, each segmented region image is assigned to the component with the highest similarity. For each component, all segmented region images assigned to that component are aggregated to obtain the weakly supervised mask image of that component. The reconstruction loss between the rendered image and the real observed image, the semantic segmentation loss between the two-dimensional segmentation probability map and the weakly supervised mask, and the consistency loss of the Gaussian sphere in the three-dimensional spatial neighborhood are calculated using a first preset loss function. Based on the calculated reconstruction loss, semantic segmentation loss, and consistency loss, the total geometric modeling loss is jointly determined, and the mask vector of each Gaussian sphere is updated with the total geometric modeling loss.

[0062] According to an embodiment of the present invention, in a first preset mode, the preset method is as follows: determining the value of each pixel in the two-dimensional Gaussian plot of each component by means of the following:

[0063]

[0064] in, Indicates the index of the pixel. Indicates the index of the component. Indicates the first The first part in the two-dimensional Gaussian plot of the component The value of each pixel. Indicates projection to the first The first pixel on the The number of Gaussian balls in each component. Indicates the first The Gaussian sphere belongs to the first... The probability of each component and Indicates the index of the Gaussian sphere. and The numbers represent the order of the numbers. The Gaussian sphere and the first The opacity of a Gaussian sphere and The numbers represent the order of the numbers. The Gaussian sphere and the first The Gaussian ball in the... The projected value on each pixel.

[0065] According to one embodiment of the present invention, the pre-trained visual basic segmentation model used in the present invention can be SAM, FastSAM, MobileSAM, SAM 3D, or a more advanced general-purpose segmentation model that will emerge in the future. It should be understood that the present invention does not limit the pre-trained visual basic segmentation model used; any visual basic model capable of zero-shot segmentation is acceptable. The following will be discussed in conjunction with the appendix... Figure 4 Taking the SAM model as an example, the geometric modeling process is described in detail. Please refer to the appendix. Figure 4 The system utilizes the native SAM model to generate fine-grained segmented region image sets. each A segmented region image is represented by a two-dimensional image with the same pixel count as the RGB observation image. Each pixel in the segmented region image has a value of 1 or 0. If an element belongs to the segmented region corresponding to the segmented region in the segmented region image, the value of that element is 1; otherwise, the value of that element is 0. The segmented region image can also be viewed as a two-dimensional binary mask. Each segmented region image corresponds to one segmented region. This represents the total number of segmented region images. For each component, the similarity between each segmented region image and the 2D Gaussian map of that component is calculated, and each segmented region image is assigned to the component with the highest similarity. For each component, all segmented region images assigned to that component are aggregated to obtain the weakly supervised mask image of that component. Based on the 2D Gaussian maps and weakly supervised masks of all components, a first preset loss function is used to calculate the geometric modeling loss of each component, and the mask vector of the Gaussian sphere within that component is updated based on the calculated loss to adjust the geometric modeling results.

[0066] According to an embodiment of the present invention, the first preset loss function is:

[0067]

[0068] in, Represents the geometric modeling loss. The first-state reconstruction loss represents the reconstruction error between the global rendered image obtained by directly rendering the first Gaussian point cloud set in a high-visibility state and the overall true observed image of the hinged object in that high-visibility state. Let represent the second-state reconstruction loss, which indicates the reconstruction error between the global rendered image obtained by performing a rigid body transformation on the first Gaussian point cloud set based on the motion parameters of each component and the overall true observed image of the hinged object in a low-visibility state. The semantic segmentation loss is represented by the error between the 2D part segmentation probability map rendered from the first Gaussian point cloud set and the weakly supervised mask image generated from the output of the pre-trained visual base segmentation model. This represents the spatial consistency loss, which is the difference in mask vectors between adjacent Gaussian spheres in three-dimensional space. and This represents the preset weight parameters.

[0069] According to an embodiment of the present invention, in S3, the second preset method is as follows: The mask vector of each Gaussian sphere in the first Gaussian point cloud set is fixed; based on the mask vector of each Gaussian sphere, the component to which the Gaussian sphere belongs is determined, and the motion parameters corresponding to the component are obtained to obtain the motion parameters of each moving component; based on all the obtained motion parameters, a rigid body transformation is performed on the first Gaussian point cloud set to obtain a rendered image in a low visibility state; the first Gaussian point cloud set in a high visibility state (normal state) and the first Gaussian point cloud set after the rigid body transformation are rendered respectively, and combined with their respective corresponding real observation images, a second preset loss function is used to determine the motion modeling loss; the motion parameters of each component are updated based on the motion modeling loss.

[0070] According to one embodiment of the present invention, the preset rigid body motion representation method is any one of the dual quaternion method, the Lie algebra rotation matrix method, and the Euler angle and translation vector combination method. For example, the transformation process of a moving part from the standard state to the target state constructed by the dual quaternion method is represented as follows:

[0071]

[0072]

[0073] in, Let the center coordinates and rotation angle of the Gaussian sphere be given. The motion parameters of the k-th moving part are derived from the dual quaternions. This represents the center coordinates of the Gaussian sphere. Perform rotation and translation. This means that the rotation of the component is superimposed on the rotation angle of the Gaussian sphere. The component undergoes a transformation from the normal state to the target state. The Gaussian sphere of the component in the original normal state is "moved" to the position it should be in the target state, thus obtaining the object image of the component in the target state.

[0074] According to an embodiment of the present invention, in step S3, the second preset loss function is:

[0075]

[0076] in, Indicates motion modeling loss, The first-state reconstruction loss represents the reconstruction error between the rendered image obtained from the first Gaussian point cloud set under high visibility conditions and the actual observed image of the hinged object under the same high visibility conditions. The second-state reconstruction loss represents the reconstruction error between the rendered image obtained by performing a rigid body transformation on the first Gaussian point cloud set based on the motion parameters of each component and the actual observed image of the hinged object in a low-visibility state. During motion modeling, only the motion parameters of the components are adjusted: precise optimization of the motion parameters is achieved by minimizing the difference between the rendered image and the actual image of the object in the normal and target states.

[0077] According to an embodiment of the present invention, in step S3, in order to improve the estimation accuracy of translational joints (such as drawers), during the motion modeling process, when iterating to a specific cycle, it is determined whether the rotation angle of the component is lower than a preset threshold (e.g., 5 degrees). If it is lower than the preset threshold, the component is determined to be a translational component, the rotation parameters of the translational component are reset and forcibly locked, and only the translation parameters are optimized. For the case represented by the dual quaternion method, the rotation component of the quaternion is forcibly locked for the translational component, and only the translation component of the quaternion is optimized. Other rigid body motion representation methods are similar and will not be described in detail here.

[0078] According to an embodiment of the present invention, in the first preset loss function and and the second preset loss function and The first-state reconstruction loss is the reconstruction error between the rendered image obtained from the first Gaussian point cloud set in a high-visibility state and the actual observed image of the hinged object in the same high-visibility state. The above four losses can be expressed by the formula: ,in, This represents the pixel error between the real image and the rendered image, i.e., the color difference; This represents the structural similarity error between the real image and the rendered image, i.e., the difference in texture and edges; This represents the depth error between the real image and the rendered image, i.e., the difference in geometric shape. , and This represents the hyperparameter used to balance the weights of the various loss terms.

[0079] IV. Step S4

[0080] In step S4, a 3D model of the articulated object is constructed based on the first Gaussian point cloud set updated in S3, the mask vector of each Gaussian sphere, and the motion parameters of each component. Specifically, the Gaussian splash model of each component (i.e., the parameters of the Gaussian sphere of each component) and the motion parameters (rotation axis, angle, and translation vector) of each component are converted into URDF format to construct the 3D model of the articulated object. The constructed 3D model can be used to support the interactive operation or physical simulation of the robot.

[0081] Reference Appendix Figure 5 The figure illustrates the process of creating a 3D model of a cabinet using a 3D modeling method for hinged objects proposed in this invention. In summary, the process includes: First, acquiring multi-view RGB-D observation data (RGB-D observation data sequence) of the target hinged object in two different states. The state with higher visibility (such as a fully open drawer) is defined as the canonical state. A state with low visibility (such as a closed drawer) is defined as the target state. The second step is initialization, which involves constructing a Gaussian point cloud set based on the RGB-D observation data in a normalized state. A Gaussian point cloud set is constructed based on the RGB-D observation data of the target state. And collect Gaussian point clouds and Voxelization is performed separately to obtain the voxel occupancy set. and Extraction is achieved by calculating voxel difference functions. and For the difference regions, connected component clustering is performed on the difference regions, and the clusters with the largest volumes are selected as initial components, namely drawer 1 (yellow), drawer 2 (yellow), cabinet door 1 (green), and cabinet door 2 (red) shown in the figure. Additionally, initial mask vectors are configured for all Gaussian spheres in all Gaussian point cloud sets. With initial motion parameters The third step involves repeatedly alternating between the geometric modeling and motion modeling steps. The geometric modeling step can be expressed by the following formula: ,in, and These represent the mask vectors of all Gaussian spheres before and after geometric modeling, respectively. This represents the first preset loss function. This represents the motion parameters of all components after the last motion modeling execution. The geometric modeling step freezes the motion parameters and only adjusts the geometric attributes and mask vectors. The geometric modeling step includes multiple iterations. In each iteration, a binary mask set S is generated using the SAM model, and the similarity between each binary mask and the Gaussian mask (two-dimensional Gaussian map) of the currently modeled component is calculated. All binary masks with a similarity greater than a preset similarity threshold are aggregated to generate a weakly supervised mask for that component. Based on the component's Gaussian mask and the weakly supervised mask, the first loss function (geometric modeling loss function) is used to optimize the component's geometric model, i.e., optimizing the attributes of the first Gaussian point cloud set and the mask vector of each Gaussian sphere. The motion modeling step can be expressed by the formula: ,in, and These represent the motion parameters of all moving parts before and after motion modeling, respectively. This represents the second preset loss function (the loss function for motion modeling). This represents the result of the previous geometric modeling, i.e., the mask vectors of Gaussian spheres within all components. The motion modeling step freezes the mask vectors, adjusting only the intrinsic properties and motion parameters of the first Gaussian point cloud set. The motion modeling step first converts the mask vectors to hard masks, ensuring that each Gaussian sphere belongs only to the component with the highest probability, avoiding the same Gaussian sphere belonging to multiple components. This prevents the Gaussian sphere from being affected by the motion transformations of different components during motion modeling. After converting the mask vectors to hard masks, the following steps are iteratively executed: Constructing the transformation process from the normal state to the target state for each component; using the constructed transformation process to transform the Gaussian spheres of each component from the normal state to the target state; rendering the transformed target state as an image; comparing the difference between the rendered image and the real image using a preset second loss function; and optimizing the intrinsic properties of the first Gaussian point cloud set and the motion parameters of each component accordingly. The fourth step involves constructing a 3D model of the articulated object based on the first Gaussian point cloud set obtained in the third step, the mask vectors of each Gaussian sphere, and the motion parameters of each component.

[0082] like Figure 6As shown in the figure, this invention presents a flowchart of a 3D modeling method for articulated objects, which can be summarized as follows: a data acquisition and preprocessing step, acquiring multi-view RGB-D observation data of the target articulated object in two target states and a normal state, and constructing Gaussian point cloud sets for the target state and the normal state; a coarse reconstruction initialization step, extracting components of the articulated object based on the Gaussian point cloud sets for the target state and the normal state, and configuring initial motion parameters and geometric parameters (geometric parameters are the mask vectors of Gaussian spheres) for each component; and 3D modeling, alternately performing geometric modeling (E-step) and motion modeling (M-step), wherein the geometric modeling step includes multiple iterations, during which motion parameters are fixed and optimized, and the motion modeling step includes multiple iterations, during which geometric parameters are fixed and optimized. When the sum of the iterations of geometric modeling and motion modeling reaches a preset number of iterations, the 3D modeling ends and the modeling result is output to obtain the 3D modeling result of the articulated object.

[0083] According to an embodiment of the present invention, a 3D modeling system for articulated objects is proposed. The system includes a data acquisition and basic representation construction module, a coarse reconstruction module, and a 3D model construction module. The data acquisition and basic representation construction module acquires first multi-view RGB-D observation data of the articulated object in its high-visibility state and second multi-view RGB-D observation data of the articulated object in its low-visibility state. Based on the first multi-view RGB-D observation data, it constructs a first Gaussian point cloud set, and based on the second multi-view RGB-D observation data, it constructs a second Gaussian point cloud set, wherein the Gaussian point cloud set is composed of multiple Gaussian spheres. The coarse reconstruction module extracts the difference voxels between the first and second Gaussian point cloud sets and performs connectivity clustering operations to construct multiple components of the articulated object. Furthermore, based on the articulated object... The system configures the initial motion parameters of each component in both high-visibility and low-visibility states; and configures a mask vector for each Gaussian sphere in the first Gaussian point cloud set, wherein the mask vector contains the probability that the Gaussian sphere belongs to each component; a 3D model building module is used to perform geometric modeling and motion modeling alternately in multiple rounds based on the initial motion parameters of each component and the initial mask vector of each Gaussian sphere to jointly optimize the mask vector of the first Gaussian point cloud and the motion parameters of each component until convergence. Based on the final obtained first Gaussian point cloud set and the mask vector of each Gaussian sphere and the motion parameters of each component, a 3D model of the articulated object is constructed, wherein performing geometric modeling involves fixing the motion parameters of all components and iteratively updating the mask vector of each Gaussian sphere, and performing motion modeling involves fixing the mask vector of all Gaussian spheres and iteratively updating the motion parameters of each component.

[0084] To demonstrate the beneficial effects of this invention, the inventors designed two sets of comparative experiments. These experiments used different datasets to compare the performance of the proposed 3D modeling method for articulated objects with other 3D modeling methods. The parameters used for evaluation in the comparative experiments included three motion modeling prediction error evaluation parameters and three geometric modeling error evaluation parameters. The motion modeling error evaluation parameters included: Axis Angle (representing the axis angle prediction error); Axis Position (representing the axis position error); and Geo Dist (representing the geometric motion path distance). The geometric modeling error evaluation parameters were collectively referred to as CD (Chamfer Distance). These parameters measured the difference between the mesh surface of the generated 3D articulated object's 3D model and the actual mesh; the smaller the difference, the smaller the geometric modeling error evaluation parameter. The geometric modeling error assessment parameters include: CD-s (CD-static), which is the chamfer distance of static components, analogous to the reconstruction quality of immovable cabinet areas; CD-m (CD-movable), which is the chamfer distance of dynamic components; and CD-w (CD-whole), which is the overall chamfer distance. The following will combine this with... Figure 7 Appendix Figure 8 Appendix Figure 9 The results of the two sets of comparative experiments were analyzed separately.

[0085] (1) Comparative Experiment 1

[0086] Comparative Experiment 1 compares the 3D modeling method proposed in this invention with DTA, ArtGS, and the proposed method on the ArtGS-Multi dataset. (See attached figures for experimental results.) Figure 7 As shown, the numbers in parentheses (e.g., table (4)) represent tables with four moving dynamic parts. The experimental data is the average of three trials, and the lower the value, the better the performance.

[0087] (2) Comparative Experiment 2

[0088] Quantitative comparisons were performed on the inventor's self-built GEAR-Multi dataset. Numbers in parentheses (e.g., Box(5)) indicate the number of dynamic parts. Experimental results are attached. Figure 8 With appendix Figure 9 As shown, the experimental data is the average of three experiments, and the lower the value, the better the performance.

[0089] Analysis of the data from comparative experiments 1 and 2 shows that the present invention performs well in three motion evaluation indicators: axis angle, axis position, and geodesic motion distance. At the same time, the static, movable, and overall chamfer distances are generally lower than those of the comparative methods in all categories, showing a consistent and stable convergence trend across objects, demonstrating higher reconstruction consistency and modeling accuracy.

[0090] In summary, this invention, with a Gaussian splash-based geometry-motion alternating optimization scheme as its core, decomposes the modeling of articulated objects into sub-processes that alternate between geometric and motion modeling. This effectively decouples the highly coupled variables of component attribution and joint motion parameters, reducing optimization difficulty, avoiding local minima, promoting stable convergence of the optimization, and significantly improving the geometric fidelity and motion estimation accuracy of complex multi-joint articulated object reconstruction. Simultaneously, a similarity-based mask aggregation mechanism is employed to address the issues of inconsistent granularity and lack of consistency across multiple perspectives. Component-level masks are extracted as weakly supervised signals to guide 3D model optimization, further improving component segmentation accuracy while ensuring generalization.

[0091] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0092] This invention can be a system, method, electronic device, computing device, computer-readable medium, and / or computer program product. A computer program product primarily refers to a software product that implements this solution through a computer program.

[0093] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0094] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for three-dimensional modeling of articulated objects, characterized in that, The method includes: S1. Obtain first multi-view RGB-D observation data of the articulated object in its high visibility state and second multi-view RGB-D observation data of the articulated object in its low visibility state, and construct a first Gaussian point cloud set based on the first multi-view RGB-D observation data, and construct a second Gaussian point cloud set based on the second multi-view RGB-D observation data, wherein the Gaussian point cloud set is composed of multiple Gaussian spheres. S2. Construct multiple components of the articulated object based on the first Gaussian point cloud set and the second Gaussian point cloud set. Each component contains multiple Gaussian spheres in the first Gaussian point cloud set. Also, configure the initial motion parameters of each component based on the high visibility state and low visibility state of the articulated object. Configure an initial mask vector containing the probability that the Gaussian sphere belongs to each component for each Gaussian sphere in the first Gaussian point cloud set. S3. Based on the initial motion parameters of each component and the initial mask vector of each Gaussian sphere, perform geometric modeling and motion modeling alternately in multiple rounds to jointly optimize the mask vector of the first Gaussian point cloud and the motion parameters of each component until convergence. Specifically, performing geometric modeling involves fixing the motion parameters of all components and iteratively updating the properties of the first Gaussian point cloud set and the mask vector of each Gaussian sphere using a first preset method. Performing motion modeling involves fixing the mask vectors of all Gaussian spheres and iteratively updating the properties of the first Gaussian point cloud set and the motion parameters of each component using a second preset method. S4. Based on the first Gaussian point cloud set updated in S3, the mask vector of each Gaussian sphere, and the motion parameters of each component, construct a 3D model of the articulated object.

2. The method according to claim 1, characterized in that, S2 includes: S21. Voxelize the first Gaussian point cloud set to obtain the first voxel occupancy set, and voxelize the second Gaussian point cloud set to obtain the second voxel occupancy set. S22. Obtain the difference region between the first voxel occupancy set and the second voxel occupancy set, and the same region between the first voxel occupancy set and the second voxel occupancy set using a preset method; S23. Constructing multiple components of an articulated object based on difference regions and similar regions, including: performing a clustering operation on the extracted difference regions to obtain multiple voxel connected blocks and selecting voxel connected blocks whose volume exceeds a preset volume threshold; using each selected voxel connected block as a moving component of the articulated object to obtain multiple moving components; each voxel connected block contains multiple Gaussian spheres in a first Gaussian point cloud set; and using the Gaussian spheres contained in the extracted similar regions to form the static components of the articulated object, wherein all moving components and static components constitute all components of the articulated object; S24. Configure the initial motion parameters of each component in the first Gaussian point cloud set based on the high visibility state and low visibility state of the articulated object. S25. Based on all the obtained components, configure a mask vector for each Gaussian sphere in the first Gaussian point cloud set, where each dimension of the mask vector corresponds to a component, and the value in each dimension of the mask vector represents the probability that the Gaussian sphere corresponding to the mask vector belongs to the component corresponding to that dimension.

3. The method according to claim 2, characterized in that, The voxel difference function is: in, This indicates the region of difference between the first voxel occupancy set and the second voxel occupancy set. Indicates the set occupied by the first voxel. Indicates that the second voxel occupies the set. This is a morphological dilation operation.

4. The method according to claim 1, characterized in that, In S3, the first preset method is: A pre-trained visual basic segmentation model is obtained in advance, wherein the pre-trained visual basic segmentation model is a pre-trained model that takes the RGB observation image of the object as input and multiple segmented region images as output, wherein each segmented region image highlights its corresponding segmented region and masks other regions. Based on the acquired pre-trained visual base segmentation model, the following steps are performed iteratively: The RGB observation image of the articulated object in its high-visibility state is obtained, and a visual basic segmentation model is used to generate multiple segmented region images based on the obtained RGB observation image. A two-dimensional Gaussian map of each component is constructed using a pre-defined method based on the mask vector of the Gaussian sphere within each component. The value of each pixel in the two-dimensional Gaussian map of each component represents the probability that the pixel belongs to that component. Calculate the similarity between the two-dimensional Gaussian plot of each component and the image of each segmented region; Based on the calculated similarity, each segmented region image is assigned to the component with the highest similarity. For each component, all segmented region images belonging to that component are aggregated to obtain the weakly supervised mask image of that component; The rendered image and the two-dimensional Gaussian probability map are obtained by rendering based on the updated Gaussian point cloud set; The reconstruction loss between the rendered image and the real observed image, the semantic segmentation loss between the two-dimensional segmentation probability map and the weakly supervised mask, and the consistency loss of the Gaussian sphere in the three-dimensional spatial neighborhood are calculated using a first preset loss function. The total geometric modeling loss is determined by jointly calculating the reconstruction loss, semantic segmentation loss, and consistency loss, and then the properties of the first Gaussian point cloud and the mask vector of each Gaussian sphere are updated with the total geometric modeling loss.

5. The method according to claim 4, characterized in that, The preset method is as follows: The value of each pixel in the two-dimensional Gaussian plot of each component is determined as follows: in, Indicates the index of the pixel. Indicates the index of the component. Indicates the first The first part in the two-dimensional Gaussian plot of the component The value of each pixel. Indicates projection to the first The first pixel on the The number of Gaussian balls in each component. Indicates the first The Gaussian sphere belongs to the first... The probability of each component and Indicates the index of the Gaussian sphere. and The numbers represent the order of the numbers. The Gaussian sphere and the first The opacity of a Gaussian sphere and The numbers represent the order of the numbers. The Gaussian sphere and the first The Gaussian ball in the... The projected value on each pixel.

6. The method according to claim 4, characterized in that, The first preset loss function is: in, Represents the geometric modeling loss. The first-state reconstruction loss represents the reconstruction error between the global rendered image obtained by directly rendering the first Gaussian point cloud set in a high-visibility state and the overall true observed image of the hinged object in that high-visibility state. Let represent the second-state reconstruction loss, which indicates the reconstruction error between the global rendered image obtained by performing a rigid body transformation on the first Gaussian point cloud set based on the motion parameters of each component and the overall true observed image of the hinged object in a low-visibility state. The semantic segmentation loss is represented by the error between the 2D part segmentation probability map rendered from the first Gaussian point cloud set and the weakly supervised mask image generated from the output of the pre-trained visual base segmentation model. This represents the spatial consistency loss, which is the difference in mask vectors between adjacent Gaussian spheres in three-dimensional space. and This represents the preset weight parameters.

7. The method according to claim 1, characterized in that, In S3, the second preset method is: Fix the mask vector of each Gaussian sphere in the first Gaussian point cloud set; Based on the mask vector of each Gaussian sphere, determine the component to which the Gaussian sphere belongs, and obtain the motion parameters corresponding to that component, so as to obtain the motion parameters of each moving component; Based on all the acquired motion parameters, a rigid body transformation is performed on the first Gaussian point cloud set to obtain a rendered image in a low visibility state. The first Gaussian point cloud set in the high visibility state (normal state) and the first Gaussian point cloud set after the rigid body transformation are rendered respectively, and the motion modeling loss is determined by the second preset loss function in combination with their corresponding real observation images. The properties of the first Gaussian point cloud set and the motion parameters of each component are updated based on the motion modeling loss.

8. The method according to claim 7, characterized in that, The preset rigid body motion representation method is any one of the following: dual quaternion method, Lie algebra rotation matrix method, and Euler angle and translation vector combination method.

9. The method according to claim 7, characterized in that, The second preset loss function is: in, Indicates motion modeling loss, The first-state reconstruction loss represents the reconstruction error between the rendered image obtained from the first Gaussian point cloud set under high visibility conditions and the actual observed image of the hinged object under the same high visibility conditions. The second state reconstruction loss is the reconstruction error between the rendered image obtained by performing a rigid body transformation on the first Gaussian point cloud set based on the motion parameters a of each component and the actual observed image of the hinged object in a low visibility state.

10. A three-dimensional modeling system for articulated objects for implementing the method as described in any one of claims 1-9, characterized in that, The system includes: The data acquisition and basic representation construction module is used to acquire first multi-view RGB-D observation data of the articulated object in its high visibility state and second multi-view RGB-D observation data of the articulated object in its low visibility state, and to construct a first Gaussian point cloud set based on the first multi-view RGB-D observation data, and to construct a second Gaussian point cloud set based on the second multi-view RGB-D observation data, wherein the Gaussian point cloud set is composed of multiple Gaussian spheres; A coarse reconstruction module is used to extract the difference voxels between the first Gaussian point cloud set and the second Gaussian point cloud set and perform connectivity clustering operations to construct multiple parts of the articulated object; and to configure the initial motion parameters of each part based on the high visibility state and low visibility state of the articulated object; and to configure a mask vector for each Gaussian sphere in the first Gaussian point cloud set, wherein the mask vector contains the probability that the Gaussian sphere belongs to each part; The 3D model building module is used to perform geometric modeling and motion modeling alternately in multiple rounds based on the initial motion parameters of each component and the initial mask vector of each Gaussian sphere. This is done to jointly optimize the properties of the first Gaussian point cloud, the mask vector of each Gaussian sphere, and the motion parameters of each component until convergence. Based on the final set of the first Gaussian point cloud, the mask vector of each Gaussian sphere, and the motion parameters of each component, a 3D model of the articulated object is constructed. Specifically, geometric modeling involves fixing the motion parameters of all components and iteratively updating the properties of the first Gaussian point cloud and the mask vector of each Gaussian sphere. Motion modeling involves fixing the mask vectors of all Gaussian spheres and iteratively updating the properties of the first Gaussian point cloud and the motion parameters of each component.

11. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-9.

12. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-9.

Images