Modeling methods and electronic devices

Abstract

This application provides a modeling method and an electronic device. The method includes: acquiring standard images of a joint object from multiple perspectives and in multiple states; generating deformed Gaussian fields of the joint object in each state based on the standard images, wherein the deformed Gaussian fields are used to characterize the geometric structure, appearance, and motion characteristics of the joint object in the corresponding state; performing component segmentation processing based on the deformed Gaussian fields to generate target Gaussian fields of each moving component in each state; determining the motion parameters of each moving component in each target state based on the target Gaussian fields of each moving component in the initial state and in each target state; modeling each moving component based on the motion parameters of each moving component in each target state; and obtaining the target model of the joint object in each target state based on the modeling results of each moving component, thereby improving the modeling accuracy and efficiency of the joint object in multiple states.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a modeling method and an electronic device. Background Technology

[0002] In the field of computer vision, it is often necessary to perform 3D modeling and motion analysis on objects with movable parts, such as doors, drawers, and notebooks.

[0003] Currently, 3D modeling typically employs methods based on explicit geometric representation, implicit shape representation, or a combination of Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). However, methods based on explicit geometric representation suffer from significant noise in actual sensor data and lack annotations, limiting their application in real-world scenarios. Methods based on implicit shape representation suffer from simplified appearance models and coarse geometric priors, failing to accurately decompose the independent motion of multiple parts. The combination of NeRF and 3DGS cannot handle scenarios with multiple moving parts, multiple interactive states, and continuous motion. Additionally, while comparing the reconstructed geometry of two states can analyze object structure, this requires knowledge of the number of parts and ensures high consistency between the two states, making it unsuitable for real-world applications.

[0004] Therefore, existing technologies have certain limitations when performing 3D modeling and motion analysis on objects with movable parts in real-world scenarios. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a modeling method and electronic device to solve the practical problem of limitations in the prior art when performing 3D modeling and motion analysis of objects with movable parts in real-world scenes.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide a modeling method, the method comprising:

[0008] Acquire standard images of jointed objects from multiple perspectives and in multiple states;

[0009] Based on the standard image, a deformed Gaussian field is generated for the joint object in each state. The deformed Gaussian field is used to characterize the geometric structure, appearance and motion characteristics of the joint object in the corresponding state.

[0010] Based on the deformed Gaussian field, component segmentation is performed to generate the target Gaussian field of each moving component in the joint object under each state.

[0011] Based on the target Gaussian field of each moving part in the initial state and each target state, the motion parameters of each moving part in each target state are determined, and each moving part is modeled based on the motion parameters of each moving part in each target state. Based on the modeling results of each moving part, the target model of the joint object in each target state is obtained.

[0012] As an optional implementation, generating the deformation Gaussian field of the joint object in each state based on the standard image includes:

[0013] Based on the standard image, a standardized Gaussian field is generated for the joint object in each state. The standardized Gaussian field includes multiple standard Gaussian elements, and the Gaussian parameters of the standard Gaussian elements include the center position of the Gaussian distribution, the covariance matrix, color information, and opacity.

[0014] Based on the target latent variables of the joint object in each state, the standardized Gaussian fields are shifted to obtain the deformed Gaussian fields of the joint object in each state.

[0015] As an optional implementation, the step of shifting each standardized Gaussian field according to the target latent variable of the joint object in each state to obtain the deformed Gaussian field of the joint object in each state includes:

[0016] The latent variables of the target under each state and the standardized Gaussian field are input into the pre-trained target deformation network. The target deformation network predicts and generates the target offset of each standard Gaussian element in each standardized Gaussian field. The target offset includes the target offset at the center position, the target offset in the direction, and the target offset in the scale.

[0017] Based on the target offset, the normalized Gaussian field is offset to obtain the deformed Gaussian field of the joint object in each state.

[0018] As an optional implementation, before the target offset of each standard Gaussian element in each of the normalized Gaussian fields is predicted and generated by the target deformation network, the following steps are included:

[0019] Based on the standard image, an initial normalized Gaussian field of the joint object in each state is generated, and the initial latent variables of the joint object in each state and each initial normalized Gaussian field are input into the initial deformation network. The initial deformation network predicts and generates the initial offset of each initial normalized Gaussian element in each initial normalized Gaussian field.

[0020] Based on each initial offset, each initial normalized Gaussian field is offset to obtain an initial deformed Gaussian field in each state. A first initial rendered image is obtained based on the initial deformed Gaussian field, and the photometric loss value between the first initial rendered image and the standard image is calculated.

[0021] Based on the photometric loss value, the model parameters of the initial deformable network, each of the initial latent variables, and the initial Gaussian parameters of each of the initial standard Gaussian units are iteratively adjusted until the photometric loss value meets the preset conditions or the number of iterations reaches the prediction number, thereby obtaining the target deformable network, the target latent variables in each state, and the target Gaussian parameters of each of the standard Gaussian units.

[0022] As an optional implementation, the step of performing component segmentation based on the deformed Gaussian field to generate the target Gaussian field of each moving component in the joint object under each state includes:

[0023] Based on the deformed Gaussian field, coarse-grained component segmentation is performed to obtain the number of moving parts of the joint object and the dynamic Gaussian elements of each moving part.

[0024] Based on the dynamic Gaussian elements of each moving component, fine component segmentation is performed to obtain the boundary candidate Gaussian elements of each moving component. Each boundary candidate Gaussian element is split and the split Gaussian elements are corrected to generate the target Gaussian field of each moving component in the joint object under each state.

[0025] As an optional implementation, the step of performing coarse-grained component segmentation based on the deformed Gaussian field to obtain the number of moving parts of the joint object and the dynamic Gaussian elements of each moving part includes:

[0026] Dynamic Gaussian detection is performed on each deformed Gaussian element in the deformed Gaussian field to determine the dynamic Gaussian element set and the static Gaussian element set, wherein the maximum displacement of each deformed Gaussian element in the dynamic Gaussian element set between each state is greater than or equal to a preset threshold.

[0027] Multiple pairs of images in different states from different perspectives are sampled from the standard image as state image pairs. The state image pairs and preset prompt words are input into the visual language model, and the visual language model predicts the number of moving parts of the joint object.

[0028] For each deformable Gaussian element in the dynamic Gaussian element set, a corresponding motion descriptor is assembled, and trajectory clustering is performed based on each motion descriptor. Based on the clustering results, the dynamic Gaussian elements of each moving component are determined.

[0029] As an optional implementation, the step of performing fine component segmentation based on the dynamic Gaussian elements of each moving component to obtain candidate boundary Gaussian elements for each moving component, splitting each candidate boundary Gaussian element, and correcting each split Gaussian element to generate the target Gaussian field of each moving component in the joint object under each state includes:

[0030] Determine the cumulative weight of each pixel in each moving part under each viewpoint, and based on the cumulative weight, determine the set of visible pixels and the set of non-contributing pixels of each moving part under each viewpoint;

[0031] A first number of positive cue points are sampled from the set of visible pixels, and a second number of negative cue points are sampled from the set of non-contributing pixels. Each of the positive cue points and each of the negative cue points are input into a general segmentation model, and the general segmentation model generates a mask for each moving part under each viewpoint.

[0032] The dynamic Gaussian elements of each moving component are projected into two-dimensional ellipses under the corresponding viewpoints. Based on the major axis of each two-dimensional ellipse and the mask of each moving component under each viewpoint, the boundary candidate Gaussian elements of each moving component are determined, and each boundary candidate Gaussian element is split into sub-Gaussian elements within each adjacent moving component.

[0033] Based on the sub-Gaussian elements in each adjacent moving part after splitting and each non-boundary candidate Gaussian element, a second initial rendered image is obtained. Based on the photometric loss value between the second initial rendered image and the standard image, the orientation and opacity of the sub-Gaussian elements in each adjacent moving part are iteratively corrected to obtain the corrected sub-Gaussian elements in each adjacent moving part.

[0034] Based on the modified sub-Gaussian elements within each adjacent moving component and each non-boundary candidate Gaussian element, the target Gaussian field of each moving component in the joint object under each state is generated.

[0035] As an optional implementation, determining the motion parameters of each moving component in each target state based on the target Gaussian field of each moving component in the initial state and each target state includes:

[0036] Based on the Gaussian parameters of each target Gaussian element in the target Gaussian field, the first Gaussian point cloud of the moving component in the initial state and the second Gaussian point cloud in the target state are determined.

[0037] Based on the first Gaussian point cloud and the second Gaussian point cloud, the joint type of each moving component is determined, wherein the joint type is either translational or rotational.

[0038] If the joint type of the moving component is translational, then the translation axis direction and translation distance of the moving component in the target state are determined based on the first Gaussian point cloud and the second Gaussian point cloud.

[0039] If the joint type of the moving component is rotational, then the rotation axis direction, rotation center, and rotation angle of the moving component in the target state are determined based on the first Gaussian point cloud and the second Gaussian point cloud.

[0040] As an optional implementation, determining the joint type of each moving component based on the first Gaussian point cloud and the second Gaussian point cloud includes:

[0041] Based on the first Gaussian point cloud and the second Gaussian point cloud, determine the rotation angle of the moving component from the initial state to the target state;

[0042] If the rotation angle is greater than or equal to the preset angle, then the joint type of the moving part is determined to be rotational.

[0043] Otherwise, the joint type of the moving part is determined to be translational.

[0044] Secondly, embodiments of this application provide a modeling apparatus, the apparatus comprising:

[0045] The acquisition module is used to acquire standard images of the jointed object from multiple perspectives and in multiple states.

[0046] The generation module is used to generate a deformed Gaussian field of the joint object in each state based on the standard image. The deformed Gaussian field is used to characterize the geometric structure, appearance and motion characteristics of the joint object in the corresponding state.

[0047] The generation module is also used to perform component segmentation processing based on the deformed Gaussian field to generate the target Gaussian field of each moving component in the joint object under each state.

[0048] The modeling module is used to determine the motion parameters of each moving part in each target state based on the target Gaussian field of each moving part in the initial state and each target state, and to model each moving part based on the motion parameters of each moving part in each target state, and to obtain the target model of the joint object in each target state based on the modeling results of each moving part.

[0049] As an optional implementation, the generation module is specifically used for:

[0050] Based on the standard image, a standardized Gaussian field is generated for the joint object in each state. The standardized Gaussian field includes multiple standard Gaussian elements, and the Gaussian parameters of the standard Gaussian elements include the center position of the Gaussian distribution, the covariance matrix, color information, and opacity.

[0051] Based on the target latent variables of the joint object in each state, the standardized Gaussian fields are shifted to obtain the deformed Gaussian fields of the joint object in each state.

[0052] As an optional implementation, the generation module is specifically used for:

[0053] The latent variables of the target under each state and the standardized Gaussian field are input into the pre-trained target deformation network. The target deformation network predicts and generates the target offset of each standard Gaussian element in each standardized Gaussian field. The target offset includes the target offset at the center position, the target offset in the direction, and the target offset in the scale.

[0054] Based on the target offset, the normalized Gaussian field is offset to obtain the deformed Gaussian field of the joint object in each state.

[0055] As an optional implementation, the method further includes: a training module; the training module is used for:

[0056] Based on the standard image, an initial normalized Gaussian field of the joint object in each state is generated, and the initial latent variables of the joint object in each state and each initial normalized Gaussian field are input into the initial deformation network. The initial deformation network predicts and generates the initial offset of each initial normalized Gaussian element in each initial normalized Gaussian field.

[0057] Based on each initial offset, each initial normalized Gaussian field is offset to obtain an initial deformed Gaussian field in each state. A first initial rendered image is obtained based on the initial deformed Gaussian field, and the photometric loss value between the first initial rendered image and the standard image is calculated.

[0058] Based on the photometric loss value, the model parameters of the initial deformable network, each of the initial latent variables, and the initial Gaussian parameters of each of the initial standard Gaussian units are iteratively adjusted until the photometric loss value meets the preset conditions or the number of iterations reaches the prediction number, thereby obtaining the target deformable network, the target latent variables in each state, and the target Gaussian parameters of each of the standard Gaussian units.

[0059] As an optional implementation, the generation module is specifically used for:

[0060] Based on the deformed Gaussian field, coarse-grained component segmentation is performed to obtain the number of moving parts of the joint object and the dynamic Gaussian elements of each moving part.

[0061] Based on the dynamic Gaussian elements of each moving component, fine component segmentation is performed to obtain the boundary candidate Gaussian elements of each moving component. Each boundary candidate Gaussian element is split and the split Gaussian elements are corrected to generate the target Gaussian field of each moving component in the joint object under each state.

[0062] As an optional implementation, the generation module is specifically used for:

[0063] Dynamic Gaussian detection is performed on each deformed Gaussian element in the deformed Gaussian field to determine the dynamic Gaussian element set and the static Gaussian element set, wherein the maximum displacement of each deformed Gaussian element in the dynamic Gaussian element set between each state is greater than or equal to a preset threshold.

[0064] Multiple pairs of images in different states from different perspectives are sampled from the standard image as state image pairs. The state image pairs and preset prompt words are input into the visual language model, and the visual language model predicts the number of moving parts of the joint object.

[0065] For each deformable Gaussian element in the dynamic Gaussian element set, a corresponding motion descriptor is assembled, and trajectory clustering is performed based on each motion descriptor. Based on the clustering results, the dynamic Gaussian elements of each moving component are determined.

[0066] As an optional implementation, the generation module is specifically used for:

[0067] Determine the cumulative weight of each pixel in each moving part under each viewpoint, and based on the cumulative weight, determine the set of visible pixels and the set of non-contributing pixels of each moving part under each viewpoint;

[0068] A first number of positive cue points are sampled from the set of visible pixels, and a second number of negative cue points are sampled from the set of non-contributing pixels. Each of the positive cue points and each of the negative cue points are input into a general segmentation model, and the general segmentation model generates a mask for each moving part under each viewpoint.

[0069] The dynamic Gaussian elements of each moving component are projected into two-dimensional ellipses under the corresponding viewpoints. Based on the major axis of each two-dimensional ellipse and the mask of each moving component under each viewpoint, the boundary candidate Gaussian elements of each moving component are determined, and each boundary candidate Gaussian element is split into sub-Gaussian elements within each adjacent moving component.

[0070] Based on the sub-Gaussian elements in each adjacent moving part after splitting and each non-boundary candidate Gaussian element, a second initial rendered image is obtained. Based on the photometric loss value between the second initial rendered image and the standard image, the orientation and opacity of the sub-Gaussian elements in each adjacent moving part are iteratively corrected to obtain the corrected sub-Gaussian elements in each adjacent moving part.

[0071] Based on the modified sub-Gaussian elements within each adjacent moving component and each non-boundary candidate Gaussian element, the target Gaussian field of each moving component in the joint object under each state is generated.

[0072] As an optional implementation, the modeling module is specifically used for:

[0073] Based on the Gaussian parameters of each target Gaussian element in the target Gaussian field, the first Gaussian point cloud of the moving component in the initial state and the second Gaussian point cloud in the target state are determined.

[0074] Based on the first Gaussian point cloud and the second Gaussian point cloud, the joint type of each moving component is determined, wherein the joint type is either translational or rotational.

[0075] If the joint type of the moving component is translational, then the translation axis direction and translation distance of the moving component in the target state are determined based on the first Gaussian point cloud and the second Gaussian point cloud.

[0076] If the joint type of the moving component is rotational, then the rotation axis direction, rotation center, and rotation angle of the moving component in the target state are determined based on the first Gaussian point cloud and the second Gaussian point cloud.

[0077] As an optional implementation, the modeling module is specifically used for:

[0078] Based on the first Gaussian point cloud and the second Gaussian point cloud, determine the rotation angle of the moving component from the initial state to the target state;

[0079] If the rotation angle is greater than or equal to the preset angle, then the joint type of the moving part is determined to be rotational.

[0080] Otherwise, the joint type of the moving part is determined to be translational.

[0081] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the modeling method described in the first aspect above.

[0082] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the modeling method described in the first aspect above.

[0083] The beneficial effects of this application are:

[0084] This application provides a modeling method and electronic device that acquires standard images of a jointed object from multiple perspectives and in multiple states. Through multi-view, multi-state acquisition, complete visual information of the jointed object in different motion states is obtained. Based on the standard images, deformable 3D Gaussian splashing is performed on the jointed object in multiple states to generate deformed Gaussian fields of the jointed object in each state, characterizing the geometric structure, appearance, and motion characteristics of the jointed object in the corresponding states. Based on the deformed Gaussian fields of the jointed object in each state, progressive component segmentation is performed on the jointed object to generate target Gaussian fields of each moving component in each state. Based on the target Gaussian fields of each moving component in the initial state and each target state, the motion parameters of each moving component in each target state are determined. Based on the motion parameters of each moving component in each target state, each moving component is accurately modeled to simulate its position and motion posture in each target state, obtaining the modeling results of each moving component, and thus obtaining the target model of the jointed object in each target state. Based on multiple states of the jointed object, deformable 3D Gaussian splashing and progressive component segmentation are performed to obtain a target Gaussian field with clear and well-defined boundaries for the moving parts in multiple states and multiple parts. This provides accurate motion parameters for component modeling, thereby improving modeling accuracy and efficiency. Attached Figure Description

[0085] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 Flowchart of the modeling method provided in the embodiments of this application Figure 1 ;

[0087] Figure 2 A schematic diagram of standard images of a jointed object from multiple perspectives and in multiple states, provided in the embodiments of this application;

[0088] Figure 3 Flowchart of the modeling method provided in the embodiments of this application Figure 2 ;

[0089] Figure 4 Flowchart of the modeling method provided in the embodiments of this application Figure 3 ;

[0090] Figure 5 Flowchart of the modeling method provided in the embodiments of this application Figure 4 ;

[0091] Figure 6 Flowchart of the modeling method provided in the embodiments of this application Figure 5 ;

[0092] Figure 7 Flowchart of the modeling method provided in the embodiments of this application Figure 6 ;

[0093] Figure 8 Flowchart of the modeling method provided in the embodiments of this application Figure 7 ;

[0094] Figure 9 This is a schematic diagram of the process for generating the target Gaussian field of each moving part in a jointed object under various states, as provided in the embodiments of this application.

[0095] Figure 10 Flowchart of the modeling method provided in the embodiments of this application Figure 8 ;

[0096] Figure 11 Flowchart of the modeling method provided in the embodiments of this application Figure 9 ;

[0097] Figure 12 A modular structure diagram of the modeling apparatus provided in the embodiments of this application;

[0098] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0099] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0100] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0101] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0102] In the field of computer vision, it is often necessary to perform 3D modeling and motion analysis on objects with movable parts, such as doors, drawers, and laptops. These movable parts can be considered as the object's joint structure. Currently, 3D modeling typically employs methods based on explicit geometric representation, implicit shape representation, or a combination of NeRF and 3DGS. However, methods based on explicit geometric representation have limitations in real-world applications; methods based on implicit shape representation cannot accurately decompose the independent motion of multiple parts. Methods combining NeRF and 3DGS cannot handle scenarios with multiple moving parts, multiple interaction states, and continuous actions. In other words, existing technologies for 3D modeling and motion analysis of jointed objects in real-world scenes have certain limitations.

[0103] This application proposes a modeling method to address the aforementioned problems. Based on standard images of a jointed object from multiple perspectives and in multiple states, it performs state-deformation-based 3D Gaussian splashing, coarse-grained component segmentation, and fine-grained component segmentation to generate target Gaussian fields for each moving component in the jointed object under each state. Without requiring manual annotation or pre-defining the number of moving components, it models each moving component in each state of the jointed object based on the target Gaussian fields of clearly defined and well-defined moving components under multiple states, thereby improving the modeling accuracy and efficiency of the jointed object under various states.

[0104] Figure 1 Flowchart of the modeling method provided in the embodiments of this application Figure 1 The subject executing this method can be any electronic device with computing power. For example... Figure 1 As shown, the method includes:

[0105] S101. Obtain standard images of the jointed object from multiple perspectives and in multiple states.

[0106] Optionally, a jointed object is an object having at least one movable part, i.e., a moving part, with the parts connected by joints, such as hinges or slide rails, and the parts can move relative to each other. For example, doors, drawers, laptops, and robotic arms are all jointed objects.

[0107] The jointed object is placed statically on an image acquisition platform, and red, green, and blue (RGB) color images of the jointed object in multiple states are captured from multiple perspectives. Through multi-view, multi-state acquisition, complete visual information of the jointed object in different motion states is obtained. Each state can represent the different degrees of motion of the jointed object throughout the entire motion process. The acquired images of the jointed object in multiple perspectives and states are standardized to unify the format and resolution, resulting in standard RGB images of the jointed object in multiple perspectives and states.

[0108] Figure 2 A schematic diagram of standard images of a jointed object from multiple perspectives and in multiple states, as provided in the embodiments of this application, such as... Figure 2 As shown, taking a jointed object as a locker as an example, the locker is obtained from multiple perspectives, from Z1 to Z... K A standard RGB image in K states. For example, Z1 to Z... K The K states can represent the different degrees of opening and closing of the drawers and doors of the locker as they move from fully closed to fully open. Multiple perspectives can cover the front, back, left, right, top, and bottom of the locker.

[0109] S102. Based on the standard image, generate the deformed Gaussian field of the joint object in each state. The deformed Gaussian field is used to characterize the geometric structure, appearance and motion characteristics of the joint object in the corresponding state.

[0110] Optionally, based on the visual and motion information of the joint object contained in the standard RGB images of the joint object from multiple perspectives and in multiple states, deformable three-dimensional Gaussian splatting is performed on the joint object in multiple states to generate deformable Gaussian fields of the joint object in each state.

[0111] Specifically, the jointed object has a corresponding Gaussian deformation field in each state, for example, based on the locker Z1 to Z2 state. K Deformable 3D Gaussian splashing is performed on K states to generate a locker in Z1 to Z2. K K deformed Gaussian fields in K states.

[0112] The deformed Gaussian field generated after splashing the state of a joint object can characterize the geometry, appearance, and motion characteristics of the joint object in the corresponding state, which better reflects the actual motion and shape of the joint object in each state. Specifically, the geometry reflects local areas of the joint object, such as drawer handles and door edges of a locker. The appearance reflects the RGB color and texture information of the joint object, and the motion characteristics reflect the motion posture of each moving part of the joint object in the corresponding state.

[0113] S103. Based on the deformed Gaussian field, perform component segmentation processing to generate the target Gaussian field of each moving component in the joint object under each state.

[0114] Optionally, based on the deformed Gaussian field of the joint object in each state, the joint object is progressively segmented into components. By segmenting components of different fine granularities, the joint object is first roughly divided into moving components and static components, and then the boundaries of each moving component are further subdivided. For each segmented moving component, a target Gaussian field in each state is generated, that is, the target Gaussian field of each moving component in the joint object in each state is generated.

[0115] Among them, the target Gaussian field of each moving part in the joint object in each state is a multi-state, multi-part Gaussian field with clear moving parts and well-defined boundaries, which can characterize the geometric structure, appearance and motion characteristics of each moving part in the joint object in the corresponding state.

[0116] For example, continue to refer to Figure 2 According to the lockers located in Z1 to Z K K deformed Gaussian fields in K states are used to progressively segment the locker into parts, determining the moving parts as drawers and doors, and generating the drawers in Z1 to Z2 states.K K target Gaussian fields in K states and cabinet doors in Z1 to Z K K target Gaussian fields in K states are used to characterize the drawer and cabinet door in Z1 to Z2, respectively. K The geometric structure, appearance, and motion characteristics of the Z states.

[0117] S104. Based on the target Gaussian field of each moving part in the initial state and each target state, determine the motion parameters of each moving part in each target state, and model each moving part according to the motion parameters of each moving part in each target state. Based on the modeling results of each moving part, obtain the target model of the joint object in each target state.

[0118] Optionally, based on the target Gaussian field of each moving component in the initial state and the target Gaussian field in each target state, the motion parameters of each moving component in each target state are determined. The motion parameters of the moving component in the target state can characterize the motion characteristics of the moving component from the initial state to the target state. The target state can be any state other than the initial state.

[0119] Based on the motion parameters of each moving part under each target state, a kinematic model is constructed. The kinematic model can simulate the position and motion posture of each moving part under each target state, accurately model each moving part, obtain the modeling results of each moving part, and then obtain the target model of the joint object under each target state.

[0120] For example, the target state is Z2 to Z. K Given any one of the K states, if the target state is Z2, the motion parameters of the drawer in the target state Z2 are used to characterize the motion characteristics of the drawer from the initial state Z1 to the target state Z2, and the motion parameters of the cabinet door in the target state Z2 are used to characterize the motion characteristics of the cabinet door from the initial state Z1 to the target state Z2. The kinematic model models the drawer and cabinet door separately based on their motion parameters in the target state Z2, simulating their positions and motion postures in the target state Z2, thus obtaining the target model of the storage cabinet in the target state Z2.

[0121] In this embodiment, standard images of the joint object are acquired from multiple perspectives and in multiple states. Through multi-view, multi-state acquisition, complete visual information of the joint object in different motion states is obtained. Based on the standard images, deformable 3D Gaussian splashing is performed on the joint object in multiple states to generate deformed Gaussian fields of the joint object in each state, characterizing the geometric structure, appearance, and motion characteristics of the joint object in the corresponding state. Based on the deformed Gaussian fields of the joint object in each state, progressive component segmentation is performed on the joint object to generate target Gaussian fields of each moving component in each state. Based on the target Gaussian fields of each moving component in the initial state and each target state, the motion parameters of each moving component in each target state are determined. Based on the motion parameters of each moving component in each target state, each moving component is accurately modeled to simulate the position and motion posture of each moving component in each target state, obtaining the modeling results of each moving component, and thus obtaining the target model of the joint object in each target state. Based on multiple states of the jointed object, deformable 3D Gaussian splashing and progressive component segmentation are performed to obtain a target Gaussian field with clear and well-defined boundaries for the moving parts in multiple states and multiple parts. This provides accurate motion parameters for component modeling, thereby improving modeling accuracy and efficiency.

[0122] Figure 3 Flowchart of the modeling method provided in the embodiments of this application Figure 2 ,like Figure 3 As shown, in step S102 above, generating the deformation Gaussian field of the joint object in each state based on the standard image includes:

[0123] S201. Based on the standard image, generate a normalized Gaussian field for the joint object in each state. The normalized Gaussian field includes multiple standard Gaussian elements. The Gaussian parameters of the standard Gaussian elements include the center position of the Gaussian distribution, the covariance matrix, color information, and opacity.

[0124] Optionally, Structure From Motion (SFM) is used to recover motion parameters and 3D structure from standard RGB images of the joint object in multiple views and states, generating point clouds of multiple Gaussian ellipsoids of the joint object in each state, i.e., Gaussian point clouds of the joint object in each state. Based on the multiple Gaussian point clouds of the joint object in each state, normalized 3D Gaussian splashing is performed to generate a normalized Gaussian field of the joint object in each state.

[0125] In this normalized Gaussian field, there are multiple standard Gaussian elements. For any standard Gaussian element i, the standard Gaussian element g... i The Gaussian parameters include the center position μ of the Gaussian distribution. i , covariance matrix ∑ i Color information ci and opacity α i Wherein, the covariance matrix ∑ i Characterization scale s i and direction q i Color information c i Represents RGB color. The normalized Gaussian field is represented as... Where N is the normalized Gaussian field The total number of medium standard high-level elements.

[0126] S202. Based on the target latent variables of the joint object in each state, the standardized Gaussian fields are shifted to obtain the deformed Gaussian fields of the joint object in each state.

[0127] Optionally, the motion posture of the joint object in each state is represented by a corresponding target latent variable. The target latent variable of the joint object in each state is obtained. The target latent variable is based on the offset information of the standardized Gaussian field indicating the corresponding state, and is determined according to the target latent variable α of the joint object in each state. k′ The indicated offset information is for each standardized Gaussian field. The standard Gaussian elements are used for offsetting to obtain the deformed Gaussian field of the jointed object in each state. Wherein, the target latent variable can be β k′ k is any state from the initial state to the final state.

[0128] Specifically, offset each standardized Gaussian field When using standard Gaussian elements, maintain the standard Gaussian fields. Color information of each standard high-resolution element c i and opacity α i Unchanged for each standardized Gaussian field The central position μ of the Gaussian distribution of each standard Gaussian element i and the covariance matrix ∑ i The scale of representation s i and direction q i By performing offsetting, the deformed Gaussian field of the offset joint object in each state is obtained.

[0129] In this embodiment, based on standard images of the joint object from multiple perspectives and in multiple states, a motion reconstruction algorithm is used to generate Gaussian point clouds of the joint object in each state. Based on these multiple Gaussian point clouds, a standardized 3D Gaussian splash is performed to generate a standardized Gaussian field for the joint object in each state. The Gaussian parameters of the standardized Gaussian elements include the center position of the Gaussian distribution, the covariance matrix, color information, and opacity. The target latent variables of the joint object in each state are obtained. Based on the offset information indicated by these target latent variables, each standardized Gaussian element in each standardized Gaussian field is offset to obtain the offset deformed Gaussian field of the joint object in each state. This ensures that the deformed Gaussian field accurately represents the geometric structure, appearance, and motion characteristics of the joint object in each state.

[0130] Figure 4 Flowchart of the modeling method provided in the embodiments of this application Figure 3 ,like Figure 4 As shown, in step S202 above, the standardized Gaussian fields are shifted according to the target latent variables of the joint object in each state to obtain the deformed Gaussian fields of the joint object in each state, including:

[0131] S301. Input the latent variables of the target under each state and the standardized Gaussian field into the pre-trained target deformation network. The target deformation network predicts and generates the target offset of each standard Gaussian element in each standardized Gaussian field. The target offset includes the target offset at the center position, the target offset in the direction, and the target offset in the scale.

[0132] Optionally, the center position μ of the Gaussian distribution is obtained based on the Gaussian parameters of each standard Gaussian element in the normalized Gaussian field of the joint object in each state. i , covariance matrix ∑ i The scale of representation s i and direction q i The center position μ of the Gaussian distribution i Scale s i , direction q i The latent variables of the target under each state are positively input into the pre-trained target deformation network. The target deformation network includes a multilayer perceptron (MLP). The MLP in the target deformation network predicts and generates the target offset of each standard Gaussian element in each standardized Gaussian field based on the latent variables of the target under each state, including the target offset Δμ at the center position of the Gaussian distribution. i Target offset Δq i and the target offset Δs i The target offset is the offset corresponding to the state.

[0133] S302. Based on the offset of each target, the normalized Gaussian field is offset to obtain the deformed Gaussian field of the joint object in each state.

[0134] Optionally, the target offset Δμ is determined based on the center position of the Gaussian distribution in each target offset. i Target offset Δq i and the target offset Δs i The center position μ of the Gaussian distribution of each standard Gaussian element in each standardized Gaussian field under state k is determined by addition. i and the covariance matrix ∑ i The scale of representation s i By performing an offset, the center position of the Gaussian distribution of the deformed Gaussian element is obtained. and scale The orientation q of each standard Gaussian element in each standardized Gaussian field under state k is determined by quaternion multiplication. i By performing offsetting, the orientation of the deformed Gaussian element is obtained.

[0135] Specifically, based on the following formula, the center position μ of the Gaussian distribution of each standard Gaussian element in each standardized Gaussian field under state k is determined. i and covariance matrix ∑ i The scale of representation s i and direction q i Perform offset:

[0136]

[0137] in, This represents the center position of the Gaussian distribution of the deformed Gaussian element. The orientation of the deformed Gaussian element. The scale of the deformed Gaussian element.

[0138] Each deformed Gaussian element in the deformed Gaussian field under state k The Gaussian parameters include: the center position of the Gaussian distribution after deformation. Transformed covariance matrix Color information c i and opacity α i Among them, the transformed covariance matrix Characterizing the scale after deformation and the direction after deformation Based on the Gaussian parameters of the Gaussian elements of the joint object under various states of deformation, the deformation Gaussian field of the joint object under each state is obtained. Deformed Gaussian Field It includes Gaussian elements of the jointed object after deformation in various states.

[0139] In this embodiment, the target latent variables and standardized Gaussian fields for each state are input into a pre-trained target deformation network. The multilayer perceptron in the target deformation network predicts and generates target offsets for each standard Gaussian element in each standardized Gaussian field based on the target latent variables for each state. These offsets include the target offset at the center position of the Gaussian distribution, the target offset in direction, and the target offset in scale. Based on these target offsets, the center position, scale, and orientation of the Gaussian distribution of each standard Gaussian element in each standardized Gaussian field for each state are shifted, resulting in the deformed Gaussian elements of the joint object in each state. Based on the Gaussian parameters of the deformed Gaussian elements of the joint object in each state, the deformed Gaussian field of the joint object in each state is obtained. The target deformation network controls the shifts of each standard Gaussian element in each standardized Gaussian field according to the target latent variables for each state, improving the accuracy of the deformed Gaussian field of the joint object in each state.

[0140] Figure 5 Flowchart of the modeling method provided in the embodiments of this application Figure 4 ,like Figure 5 As shown, before the target offset of each standard Gaussian element in each normalized Gaussian field is predicted and generated by the target deformation network in step S301 above, the following steps are included:

[0141] S401. Based on the standard image, generate the initial normalized Gaussian field of the joint object in each state, and input the initial latent variables of the joint object in each state and each initial normalized Gaussian field into the initial deformation network. The initial deformation network predicts and generates the initial offset of each initial normalized Gaussian element in each initial normalized Gaussian field.

[0142] Optionally, based on multiple Gaussian point clouds of the joint object in each state obtained from a standard image, an initial normalized 3D Gaussian splash is performed to generate an initial normalized Gaussian field of the joint object in each state. The initial normalized Gaussian field includes multiple initial normalized Gaussian elements, and the Gaussian parameters of each initial normalized Gaussian element include the center position of the initial Gaussian distribution, the initial covariance matrix, initial color information, and initial opacity.

[0143] The initial standard Gaussian distribution center position, initial scale and initial orientation represented by the initial covariance matrix of each initial standard Gaussian element in the initial normalized Gaussian field of the joint object in each state, and the randomly generated initial latent variables of the joint object in each state are input into the initial deformation network. The MLP in the initial deformation network predicts and generates the initial offset of each initial standard Gaussian element in each initial normalized Gaussian field according to the initial latent variables in each state under the initial model parameters, including the initial offset of the center position of the Gaussian distribution, the initial offset of the orientation, and the initial offset of the scale.

[0144] S402. Based on each initial offset, offset each initial normalized Gaussian field to obtain the initial deformed Gaussian field in each state. Render the first initial rendered image based on the initial deformed Gaussian field and calculate the photometric loss value between the first initial rendered image and the standard image.

[0145] Optionally, based on the initial offset of the center position, initial offset of the orientation, and initial offset of the Gaussian distribution in each initial offset, the center position, initial scale, and initial orientation of the initial Gaussian distribution of each initial normalized Gaussian element in each initial normalized Gaussian field are offset to obtain the initial deformation Gaussian field of the joint object in each state. The process of offsetting each initial normalized Gaussian field can be referred to the description of step S302 above, and will not be repeated here.

[0146] Based on the initial deformation Gaussian field of the joint object in each state, the first initial rendered image of the joint object in each state is rendered, and the photometric loss value between the first initial rendered image and the standard image is calculated. The photometric loss value can characterize the degree of difference between the first initial rendered image and the standard image.

[0147] S403. Based on the photometric loss value, iteratively adjust the model parameters of the initial deformable network, each initial latent variable, and the initial Gaussian parameters of each initial standard Gaussian unit until the photometric loss value meets the preset conditions or the number of iterations reaches the prediction number, to obtain the target deformable network, the target latent variables in each state, and the target Gaussian parameters of each standard Gaussian unit.

[0148] Optionally, during training, the photometric loss values ​​of the first initial rendered image and the standard image are backpropagated to the MLP in the initial deformable network, and the model parameters of the initial deformable network, the initial latent variables in each state, and the initial Gaussian parameters of each initial standard Gaussian unit are iteratively adjusted using the error gradient descent algorithm.

[0149] Specifically, with the optimization objective of minimizing the photometric loss between the first initial rendered image and the standard image, the steps of Gaussian field deformation, image rendering, loss calculation, and backpropagation are repeatedly executed. Training stops when the photometric loss meets the preset condition or the number of iterations reaches the prediction number, thereby obtaining the target deformation network, the target latent variables in each state, and the target Gaussian parameters of each standard Gaussian unit.

[0150] The model parameters of the initial deformable network, the initial latent variables in each state, and the initial Gaussian parameters of each initial standard Gaussian element are jointly optimized to improve the accuracy of the target deformable network, the target latent variables in each state, and the target Gaussian parameters of each standard Gaussian element. This, in turn, improves the accuracy of the deformable Gaussian field of the joint object in each state, enabling the deformable Gaussian field of the joint object in each state to more accurately represent the geometric structure, appearance, and motion characteristics of the joint object in each state.

[0151] In this embodiment, initial normalized Gaussian fields of the joint object in each state are generated based on the standard image. The initial latent variables of the joint object in each state and each initial normalized Gaussian field are input into the initial deformation network. The initial deformation network predicts and generates the initial offset of each initial standard Gaussian element in each initial normalized Gaussian field. Based on each initial offset, each initial normalized Gaussian field is offset to obtain the initial deformed Gaussian field in each state. A first initial rendered image is obtained based on the initial deformed Gaussian field. The photometric loss value between the first initial rendered image and the standard image is calculated. The photometric loss value characterizes the degree of difference between the first initial rendered image and the standard image. With minimizing the photometric loss value between the first initial rendered image and the standard image as the optimization objective, the model parameters of the initial deformation network, the initial latent variables in each state, and the initial Gaussian parameters of each initial standard Gaussian element are iteratively adjusted. Training stops when the photometric loss value meets a preset condition or when the number of iterations reaches the prediction number, thus obtaining the target deformation network, the target latent variables in each state, and the target Gaussian parameters of each standard Gaussian element. By jointly optimizing the initial deformable network, the initial latent variables in each state, and the initial Gaussian parameters of each initial standard Gaussian element, the accuracy of the target deformable network, the target latent variables in each state, and the target Gaussian parameters of each standard Gaussian element is improved.

[0152] Figure 6 Flowchart of the modeling method provided in the embodiments of this application Figure 5 ,like Figure 6 As shown, in step S103 above, component segmentation is performed based on the deformed Gaussian field to generate the target Gaussian field of each moving component in the joint object under each state, including:

[0153] S501. Based on the deformed Gaussian field, perform coarse-grained component segmentation to obtain the number of moving parts of the joint object and the dynamic Gaussian elements of each moving part.

[0154] Optionally, in the progressive component segmentation process of the joint object based on the deformed Gaussian field of the joint object in each state, the joint object is first subjected to coarse-grained component segmentation. Based on the motion characteristics of the joint object in the corresponding state characterized by the deformed Gaussian field, the joint object is roughly divided into moving parts and static parts, and the number n of moving parts of the joint object is estimated. parts .

[0155] The dynamic Gaussian elements of each moving part are determined from the deformed Gaussian elements of the deformed Gaussian field under each state, and each deformed Gaussian element is categorized into the label of each moving part. In other words, during coarse-grained part segmentation, without manual intervention or pre-setting the number of moving parts, the number of moving parts of the joint object (n) can be determined self-supervisedly based on the motion characteristics of the joint object in the corresponding state, as characterized by the deformed Gaussian field. parts Based on this, the deformed Gaussian elements of the deformed Gaussian field under each state are assigned to different moving parts, and the relationship between each dynamic Gaussian element and each moving part is roughly determined.

[0156] S502. Based on the dynamic Gaussian elements of each moving part, perform fine part segmentation processing to obtain the boundary candidate Gaussian elements of each moving part. Split each boundary candidate Gaussian element and correct each split Gaussian element to generate the target Gaussian field of each moving part in each state in the joint object.

[0157] Optionally, after coarse-grained component segmentation, based on the dynamic Gaussian elements of each moving component, further fine-grained component segmentation is performed using visual guidance and image segmentation. This refines the boundaries of each moving component, ensuring that the dynamic Gaussian elements of each moving component are accurately assigned to each moving component, avoiding blurred component boundaries. This generates a multi-state, multi-component Gaussian field with clear and well-defined boundaries for each moving component, serving as the target Gaussian field for each moving component in each state within the joint object.

[0158] Specifically, firstly, candidate boundary Gaussian elements for each moving component are determined from the dynamic Gaussian elements of each moving component. These candidate boundary Gaussian elements are then used as splitting targets, and recursively split to obtain the split Gaussian elements for different moving components. Through a small number of iterations, local optimization is performed on each split Gaussian element. Combined with the unsplit dynamic Gaussian elements of each moving component, a multi-state, multi-component Gaussian field with clear and well-defined boundaries for each moving component is generated.

[0159] In this embodiment, based on the deformed Gaussian field of the joint object in each state, coarse-grained component segmentation is performed on the joint object, roughly dividing it into moving and static components, and estimating the number of moving components. Dynamic Gaussian elements of each moving component are determined from the deformed Gaussian elements of the deformed Gaussian field in each state, and these deformed Gaussian elements are categorized into labels for each moving component. During the coarse-grained component segmentation process, no manual intervention or pre-setting of the number of moving components is required. Based on the motion characteristics of the joint object in the corresponding state represented by the deformed Gaussian field, the number of moving components of the joint object can be determined self-supervised, and the relationship between each dynamic Gaussian element and each moving component can be roughly determined. Based on the dynamic Gaussian elements of each moving component, fine-grained component segmentation is further performed, subdividing the boundaries of each moving component. Candidate boundary Gaussian elements of each moving component are determined from the dynamic Gaussian elements of each moving component. These candidate boundary Gaussian elements are used as splitting objects, and recursively splitting them to obtain the split Gaussian elements of different moving components. This process ensures the precise allocation of dynamic Gaussian elements to each moving component, avoiding blurred component boundaries and generating a multi-state, multi-component Gaussian field with clear and well-defined boundaries for each moving component. Progressive segmentation further enhances the boundary differentiation of moving components within the jointed object.

[0160] Figure 7 Flowchart of the modeling method provided in the embodiments of this application Figure 6 ,like Figure 7 As shown, in step S501 above, coarse-grained component segmentation is performed based on the deformed Gaussian field to obtain the number of moving parts of the joint object and the dynamic Gaussian elements of each moving part, including:

[0161] S601. Perform dynamic Gaussian detection on each deformed Gaussian element in the deformed Gaussian field to determine the dynamic Gaussian element set and the static Gaussian element set, wherein the maximum displacement of each deformed Gaussian element in the dynamic Gaussian element set between each state is greater than or equal to a preset threshold.

[0162] Optionally, dynamic Gaussian detection is performed on each deformed Gaussian element in the deformed Gaussian field, and the deformed Gaussian elements in the deformed Gaussian field are divided into a set of dynamic Gaussian elements based on the dynamic Gaussian detection results. and static Gaussian set

[0163] Specifically, the maximum displacement of each deformed Gaussian element in the Gaussian deformation field between states is calculated and normalized, based on the condition that the maximum displacement between states is greater than or equal to a preset threshold τ. mot The various deformed Gaussian elements form a dynamic Gaussian element set. Based on the maximum displacement between each state being less than a preset threshold τ motThe various deformed Gaussian elements form a set of static Gaussian elements.

[0164] The deformable Gaussian element g in the Gaussian deformation field is calculated based on the following formula. i Find the maximum displacement between each state and normalize it:

[0165]

[0166] Where, d i ′ is a deformed Gaussian element g i The maximum displacement when moving from state j to state k. For the deformed Gaussian element g i The normalized maximum displacement when moving from state j to state k. For the deformed Gaussian element g in state j i The central location of the Gaussian distribution at ′ For the deformable Gaussian element g in state k i The central location of the Gaussian distribution.

[0167] Preset threshold τ mot This indicates the proportion of the maximum displacement in the entire scene that can be considered as dynamic Gaussian elements. In other words, it refers to the Gaussian deformation field that satisfies... Deformed Gaussian elements form a dynamic Gaussian element set. Satisfying the conditions in a Gaussian deformation field Deformed Gaussian elements form a set of static Gaussian elements.

[0168] S602. Sample multiple pairs of images from different perspectives and states from the standard image as state image pairs. Input the state image pairs and preset prompt words into the visual language model, and the visual language model predicts the number of moving parts of the joint object.

[0169] Optionally, two images from different perspectives and states of a jointed object are sampled from standard RGB images of the object in multiple viewpoints and states to obtain a Y-pair of state image pairs. Where, k i ≠k j And v h ≠v f .

[0170] The Y-pair of state images and the preset prompt are input into the Visual Language Model (VLM). Under the guidance of the preset prompt, the VLM analyzes the M-pair of state images and predicts the number n of moving parts of the jointed object. partsFor example, VLM can be a Bootstrapping Language-Image Pre-training (BLIP) model, with a preset prompt such as: "Compare the two images. How many components moved?"

[0171] "Output: 'Number of moving components: [Number]'" ("Answer: 'Number of moved components: [Number]'.")

[0172] Select the most frequent integer from the Y results predicted by BLIP as the number of moving parts, n. parts The mode of the Y-th prediction from BLIP is taken as the number of moving parts, n. parts This avoids accidental misjudgments caused by single predictions and enables self-supervised estimation of the number of moving parts in jointed objects.

[0173] S603. Assemble the corresponding motion descriptors for each deformable Gaussian element in the dynamic Gaussian element set, perform trajectory clustering based on each motion descriptor, and determine the dynamic Gaussian elements of each moving component based on the clustering results.

[0174] Optionally, for dynamic Gaussian set The displacement direction and scale of each dynamic Gaussian element are encoded, and dynamic Gaussian element sets are assembled accordingly. The motion descriptors corresponding to each dynamic Gaussian element. For example, for any g... i ′∈G dyn Based on the following formula, g is assembled to obtain g. i The corresponding assembly descriptor f i ′:

[0175]

[0176]

[0177] Among them, f i ′ is a dynamic Gaussian element g i The corresponding motion descriptor, Represents the dynamic Gaussian element g i The center position of the Gaussian distribution of ′ is from the k state. Transition to state k+1 The displacement vector, for The normalized vector, where ε is a very small integer to prevent the denominator from being zero during normalization.

[0178] The motion descriptors corresponding to each dynamic Gaussian element are normalized. Based on the normalized motion descriptors, the K-means clustering algorithm is used to cluster the trajectories. Dynamic Gaussian elements with the same motion direction / trajectory are grouped into the same moving part, meaning each dynamic Gaussian element corresponds to a number from 1 to n. parts One of the moving parts.

[0179] For example, any g i ′∈G dyn All correspond to l i ∈{1,…,n parts A moving part label, a static Gaussian set The static component label corresponding to each static Gaussian element can be 0. Based on this, preliminary component classification labels are obtained, including labels for each moving component and static component labels. The relationship between each dynamic Gaussian element and each moving component is determined based on the labels for each moving component.

[0180] In this embodiment, dynamic Gaussian detection is performed on each deformable Gaussian element in the deformable Gaussian field, and the dynamic Gaussian element set and static Gaussian element set are determined based on the dynamic Gaussian detection results. Multiple pairs of state image pairs are sampled from standard images of the joint object at different views and states. These state image pairs and preset prompts are input into a visual language model, which predicts the number of moving parts of the joint object. The displacement direction and scale of each dynamic Gaussian element in the dynamic Gaussian element set are encoded, and motion descriptors corresponding to each dynamic Gaussian element in the dynamic Gaussian element set are assembled. Trajectory clustering is performed based on each motion descriptor, and the dynamic Gaussian elements of each moving part are determined based on the clustering results. Through coarse-grained part segmentation, the relationship between each dynamic Gaussian element and each moving part is preliminarily determined.

[0181] Figure 8 Flowchart of the modeling method provided in the embodiments of this application Figure 7 ,like Figure 8 As shown, in step S502 above, fine component segmentation is performed based on the dynamic Gaussian elements of each moving component to obtain the boundary candidate Gaussian elements of each moving component. Each boundary candidate Gaussian element is then split, and the split Gaussian elements are corrected to generate the target Gaussian field of each moving component in each state within the joint object, including:

[0182] S701. Determine the cumulative weight of each pixel in each moving part under each viewpoint, and determine the set of visible pixels and the set of non-contributing pixels of each moving part under each viewpoint based on the cumulative weight.

[0183] Optionally, at each viewpoint, the cumulative weight of each moving part at each pixel is calculated, and the cumulative weight of the moving part at each pixel is used to characterize the visibility of the moving part at that pixel. Based on the cumulative weight of each moving part at each pixel, the set of visible pixels and the set of non-contributing pixels for each moving part at each viewpoint are determined, wherein the set of visible pixels indicates that the pixel is in the strongly visible region of the moving part, and the set of non-contributing pixels indicates that the pixel is in the invisible region of the moving part.

[0184] Specifically, from the perspective of v, the cumulative weight of the moving part p at pixel x is calculated. With other moving parts besides moving part p Maximum cumulative weight at pixel x In comparison. When satisfying At that time, pixel x is assigned to moving part p, forming the set of visible pixels of moving part p. Otherwise, a set of non-contributing components p is formed.

[0185] S702. Sample a first number of positive cue points from the set of visible pixels, sample a second number of negative cue points from the set of non-contributing pixels, and input each positive cue point and each negative cue point into the general segmentation model, and generate a mask for each moving part under each viewpoint by the general segmentation model.

[0186] Optionally, a first number of positive cue points and a second number of negative cue points are sampled from the visible pixel set and non-contribution set of each moving part, respectively, wherein the first number may be less than the second number. Each positive cue point and each negative cue point is input into a general segmentation model (Segment Anything Model, SAM), which generates a mask for each moving part from each viewpoint.

[0187] For example, from the perspective of v, the set of visible pixels of the moving part p Sampling 10 positive cue points from the non-contribution set of moving part p Twenty negative cue points are sampled, and ten positive cue points and twenty negative cue points are input into the SAM (Sensitive Aware Mask). This indicates to the SAM that these 10 pixels definitely belong to the moving part p, and these 20 pixels definitely do not belong to the moving part p. Under the visibility-aware prompt generation of the positive and negative cue points, the SAM generates a mask of the moving part p from the v-viewpoint.

[0188] S703. Project the dynamic Gaussian elements of each moving component into two-dimensional ellipses under the corresponding viewpoints, and determine the boundary candidate Gaussian elements of each moving component based on the major axis of each two-dimensional ellipse and the mask of each moving component under each viewpoint, and split each boundary candidate Gaussian element into sub-Gaussian elements within each adjacent moving component.

[0189] Optionally, the closest viewing angle v of the dynamic Gaussian elements of each moving part can be determined by light sampling. near The dynamic Gaussian elements of each moving component are projected as the closest viewing angle v. near The major axis of each two-dimensional ellipse is determined. By comparing the major axis of each two-dimensional ellipse with the size of the mask of each moving part under each viewpoint, the candidate boundary Gaussian elements of each moving part are determined.

[0190] For example, from the v-viewpoint, if the major axis of the two-dimensional ellipse after the dynamic Gaussian element projection of the moving part p is greater than the mask of the moving part p from the v-viewpoint... Then the dynamic Gaussian element is taken as the boundary candidate Gaussian element of the moving part p.

[0191] During the splitting of the boundary candidate Gaussian elements of the moving component, the target proportion of the major axis of the two-dimensional ellipse of the boundary candidate Gaussian element within the mask of the moving component is calculated. Based on the target proportion, the center position and a portion of the scale of the Gaussian distribution of the boundary candidate Gaussian element are shrunk within the mask. The portion shrunk within the mask is taken as the in-component sub-Gaussian of the moving component, and the overflow portion outside the mask is taken as the background sub-Gaussian of the moving component. The background sub-Gaussian of the moving component is shrunk into the mask of the moving component adjacent to the moving component in the subsequent recursive splitting process, so that the two-dimensional ellipse does not intersect with the masks of each moving component.

[0192] In other words, the candidate Gaussian elements of each moving component's boundary are recursively split into sub-Gaussian elements within each adjacent moving component. Dynamic Gaussian elements crossing the boundary of a moving component are split into sub-Gaussian elements that explicitly belong to adjacent moving components, thus resolving the geometric overlap problem and improving the clarity of the moving component's boundary.

[0193] S704. Based on the sub-Gaussian elements in each adjacent moving part after splitting and each non-boundary candidate Gaussian element, a second initial rendering image is obtained. Based on the photometric loss value between the second initial rendering image and the standard image, the orientation and opacity of the sub-Gaussian elements in each adjacent moving part are iteratively corrected to obtain the corrected sub-Gaussian elements in each adjacent moving part.

[0194] Optionally, during the recursive splitting of the boundary candidate Gaussian elements of each moving component, the dynamic Gaussian elements within each moving component, i.e., the non-boundary candidate Gaussian elements, are not interfered with. Based on the sub-Gaussian elements within each adjacent moving component after splitting and the non-boundary candidate Gaussian elements, a second initial rendered image is obtained, and the photometric loss value between the second initial rendered image and the standard image is calculated.

[0195] Since the position and scale of the Gaussian distribution of sub-Gaussian elements in each adjacent moving part have been corrected during the splitting process, it is only necessary to perform local correction and optimization of the orientation and opacity of the sub-Gaussian elements in each adjacent moving part through a small number of iterations based on the photometric loss values ​​of the second initial rendered image and the standard image to obtain the corrected sub-Gaussian elements in each adjacent moving part, so as to improve the consistency of the second initial rendered image.

[0196] S705. Based on the modified sub-Gaussian elements within each adjacent moving part and each non-boundary candidate Gaussian element, generate the target Gaussian field of each moving part in each state of the joint object.

[0197] Optionally, based on the modified sub-Gaussian elements within each adjacent moving part and the unsplit non-boundary candidate Gaussian elements, a target Gaussian field is generated for each moving part in each state of the joint object. The target Gaussian field can represent a Gaussian field with clear parts and well-defined boundaries in multiple states and multiple parts.

[0198] For example, the target Gaussian field of each moving part in a jointed object under each state can be represented as: in,

[0199] In this embodiment, the visible pixel set and non-contributing pixel set of each moving part under each viewpoint are determined based on the cumulative weight of each pixel in each moving part under each viewpoint. A first number of positive cue points and a second number of negative cue points are sampled from the visible pixel set and the non-contributing pixel set, respectively, and each positive cue point and each negative cue point is input into a general segmentation model, which generates a mask for each moving part under each viewpoint. The dynamic Gaussian elements of each moving part are projected into two-dimensional ellipses under the corresponding viewpoint, and the boundary candidate Gaussian elements of each moving part are determined based on the major axis of each two-dimensional ellipse and the mask of each moving part under each viewpoint. Each boundary candidate Gaussian element is split into sub-Gaussian elements within each adjacent moving part. This solves the geometric overlap problem and improves the boundary clarity of the moving parts. Based on the split sub-Gaussian elements within each adjacent moving part and each non-boundary candidate Gaussian element, a second initial rendered image is rendered. Based on the photometric loss value between the second initial rendered image and the standard image, the orientation and opacity of the sub-Gaussian elements within each adjacent moving part are iteratively corrected to obtain the corrected sub-Gaussian elements within each adjacent moving part. Based on the corrected sub-Gaussian elements within each adjacent moving component and the unsplit non-boundary candidate Gaussian elements, the target Gaussian field of each moving component in the joint object under each state is generated. Through fine component segmentation, a multi-state, multi-component Gaussian field with clear components and well-defined boundaries is obtained.

[0200] Figure 9 This is a schematic diagram of the process for generating the target Gaussian field of each moving part in a jointed object under various states, as provided in the embodiments of this application. Figure 9 Taking this as an example, we will explain in detail the complete process of generating the target Gaussian field of each moving part in the locker under each state.

[0201] Reference Figure 9 In the deformable 3D Gaussian splashing stage, the SFM algorithm is used to analyze the locker from multiple perspectives, from Z1 to Z... K Motion parameters and 3D structure are recovered from standard RGB images in K states, generating Gaussian point clouds of the locker in each state. Based on multiple Gaussian point clouds of the locker in each state, initial normalized 3D Gaussian splashing is performed to generate a normalized Gaussian field of the locker in each state. A latent variable controlled Gaussian deformation network f is then used. def In this process, latent variables under each state are introduced to shift the standardized Gaussian field, and a first initial rendered image is generated after the shift. Based on the photometric loss between the first initial rendered image and the standard image, the optimization objective is to minimize the photometric loss between the first initial rendered image and the standard image, and then jointly optimize the Gaussian deformation network f. defThe model parameters, latent variables in each state, and Gaussian parameters of each standard Gaussian element are obtained. The target Gaussian deformable network, trained from the standard Gaussian field and the latent variables in each state, yields the deformable Gaussian field of the locker in each state, characterizing the geometric structure, appearance, and motion characteristics of the locker under different states.

[0202] In the coarse-grained component segmentation stage, dynamic Gaussian detection is performed on each deformed Gaussian element in the deformed Gaussian field to obtain a set of dynamic Gaussian elements. and static Gaussian metasets Two images from different viewpoints and in different states are sampled as a pair of state images, and Y-pairs of state image pairs are obtained. The data, along with preset prompts, is input into the VLM model to estimate the number of moving parts in the locker as n. parts And through the dynamic Gaussian set Trajectory clustering is performed on each dynamic Gaussian element to preliminarily determine the relationship between each dynamic Gaussian element and each moving component.

[0203] In the refined component segmentation stage, taking a moving component p as an example, the set of visible pixels of the moving component p is determined. Non-contribution set From the set of visible pixels of the moving part p Sampling 10 positive cue points from the non-contribution set of moving part p Twenty negative cue points are sampled and input into SAM. SAM generates a mask for the moving part p from the v-viewpoint based on visibility cue perception. Based on the masks of moving parts from various perspectives, candidate Gaussian elements for the boundaries of each moving part are determined. Then, according to the mask size, the candidate Gaussian elements for the boundaries of each moving part are split into sub-Gaussian elements based on boundary awareness, thus solving the geometric overlap problem and improving the boundary clarity of the moving parts.

[0204] Based on the new Gaussian units after splitting and the unsplit Gaussian units, a second initial rendered image is obtained. Then, based on the photometric loss values ​​of the second initial rendered image and the standard image, the orientation and opacity of the new Gaussian units after splitting are locally optimized to ensure the consistency of the second initial rendered image. After local optimization, a multi-state, multi-part Gaussian field with clear components and well-defined boundaries is generated, representing the target Gaussian field of each moving component in the locker under each state.

[0205] Figure 10 Flowchart of the modeling method provided in the embodiments of this application Figure 8 ,like Figure 10As shown, in step S104 above, the motion parameters of each moving component in each target state are determined based on the target Gaussian field of each moving component in the initial state and each target state, including:

[0206] S801. Based on the Gaussian parameters of each target Gaussian element in the target Gaussian field, determine the first Gaussian point cloud of the moving part in the initial state and the second Gaussian point cloud in the target state.

[0207] Optionally, based on the Gaussian parameters of each target Gaussian element in the target Gaussian field, the first Gaussian point cloud A = {A1, A2, ..., A...} of the moving part in the initial state Z1 is determined. n} and the moving parts in the target state Z b The second Gaussian point cloud B = {B1, B2, ..., B} is given. n}

[0208] S802. Based on the first Gaussian point cloud and the second Gaussian point cloud, determine the joint type of each moving part, wherein the joint type is either translational or rotational.

[0209] Optionally, based on the first Gaussian point cloud A = {A1, A2, ..., A...} n} and the second Gaussian point cloud B = {B1, B2, ..., B n}, analyze the motion of the moving part from the initial state Z1 to the target state Z b The motion posture is determined, and the joint type of the moving parts is determined based on the motion posture.

[0210] The joint type of the moving parts can be either translational or rotational; translational joints can also be called sliding joints. For the same jointed object, the joint types of the moving parts may be different. For example, with... Figure 2 Taking a storage cabinet as an example, the drawer has a sliding joint type, and the cabinet door has a rotating joint type.

[0211] S803. If the joint type of the moving part is translation type, then determine the translation axis direction and translation distance of the moving part in the target state based on the first Gaussian point cloud and the second Gaussian point cloud.

[0212] Optionally, if the joint type of the moving part is determined to be translational, then according to the first Gaussian point cloud A = {A1, A2, ..., A...} n} and the second Gaussian point cloud B = {B1, B2, ..., B n}, determine the translation parameters of the moving part in the target state, including the translation axis direction and translation distance of the moving part in the target state.

[0213] S804. If the joint type of the moving part is rotational, then the rotation axis direction, rotation center and rotation angle of the moving part in the target state are determined according to the first Gaussian point cloud and the second Gaussian point cloud.

[0214] Optionally, if the joint type of the moving part is determined to be rotational, then according to the first Gaussian point cloud A = {A1, A2, ..., A...} n} and the second Gaussian point cloud B = {B1, B2, ..., B n}, determine the rotation parameters of the moving part in the target state, including the rotation axis direction, rotation center and rotation angle of the moving part in the target state.

[0215] In this embodiment, based on the Gaussian parameters of each target Gaussian element in the target Gaussian field, the first Gaussian point cloud of the moving component in the initial state and the second Gaussian point cloud in the target state are determined. Using the first and second Gaussian point clouds, the joint type of each moving component is determined to be either translational or rotational. If the joint type of the moving component is translational, the translation axis direction and translation distance of the moving component in the target state are determined based on the first and second Gaussian point clouds. If the joint type of the moving component is rotational, the rotation axis direction, rotation center, and rotation angle of the moving component in the target state are determined based on the first and second Gaussian point clouds. This accurately identifies the joint type of each moving component and determines the motion parameters of each moving component in each target state.

[0216] Figure 11 Flowchart of the modeling method provided in the embodiments of this application Figure 9 ,like Figure 11 As shown, in step S802 above, the joint type of each moving component is determined based on the first Gaussian point cloud and the second Gaussian point cloud, including:

[0217] S901. Based on the first Gaussian point cloud and the second Gaussian point cloud, determine the rotation angle of the moving part from the initial state to the target state.

[0218] Optionally, based on the first Gaussian point cloud A = {A1, A2, ..., A...} n} and the second Gaussian point cloud B = {B1, B2, ..., B n The Kabsch algorithm is used to calculate the centroid of the first Gaussian point cloud A. The centroid of the second Gaussian point cloud B Based on the centroid of the first Gaussian point cloud A The centroid of the second Gaussian point cloud B Calculate the covariance matrix H of the first Gaussian point cloud A and the second Gaussian point cloud B, and perform singular value decomposition on the covariance matrix H of the first Gaussian point cloud A and the second Gaussian point cloud B to obtain the rotation matrix R. Based on the rotation matrix R, calculate the motion of the moving part from the initial state Z1 to the target state Z. b The rotation angle θ.

[0219] Specifically, the motion of the moving part from the initial state Z1 to the target state Z is calculated based on the following formula. b Rotation angle θ:

[0220]

[0221] H=U∑V T

[0222] R = VU T

[0223]

[0224] in, Let A be the centroid of the first Gaussian point cloud. Let H be the centroid of the second Gaussian point cloud B, H be the covariance matrix of the first Gaussian point cloud A and the second Gaussian point cloud B, R be the rotation matrix, and θ be the value of the moving part moving from the initial state Z1 to the target state Z. b The rotation angle is tr(R), which is the trace of the rotation matrix R, i.e., the sum of the main diagonal elements of the rotation matrix R.

[0225] S902. If the rotation angle is greater than or equal to the preset angle, then the joint type of the moving part is determined to be rotation type.

[0226] Optionally, compare the movement of the moving part from the initial state Z1 to the target state Z. b The rotation angle θ is greater than or equal to the preset angle θ′. If the rotation angle θ is greater than or equal to the preset angle θ′, it indicates that the moving part has moved from the first Gaussian point cloud A in the initial state Z1 to the target state Z. b If the point cloud transformation of the second Gaussian point cloud B is a rotational transformation, then the joint type of the moving part is determined to be a rotational type.

[0227] For example, the preset angle θ′ can be 5°.

[0228] S903. Otherwise, determine that the joint type of the moving part is translational.

[0229] Optionally, if the rotation angle θ is less than the preset angle θ′, it means that the moving part has moved from the first Gaussian point cloud A in the initial state Z1 to the target state Z. b If the point cloud transformation of the second Gaussian point cloud B is approximately a translation transformation, then the joint type of the moving part is determined to be a translation type.

[0230] In this embodiment, the rotation angle of the moving component from the initial state to the target state is calculated based on the first Gaussian point cloud and the second Gaussian point cloud. When the rotation angle is greater than or equal to a preset angle, the joint type of the moving component is determined to be rotational. When the rotation angle is less than the preset angle, the joint type of the moving component is determined to be translational. This improves the accuracy of the joint type of the moving component.

[0231] The following is a detailed explanation of the process of determining the rotation axis direction, rotation center, and rotation angle of a moving part in the target state.

[0232] For rotating moving parts, calculate RI based on the rotation matrix R and the 3×3 identity matrix I, and then solve for the eigenvalues ​​λ of RI. i and eigenvectors Right now Rotation axis direction For the eigenvector whose corresponding eigenvalue is closest to 0, the eigenvector is... Normalization yields the direction of the rotation axis. The center of rotation is determined by identifying the Gaussian point at the static-dynamic boundary between the static and moving components. The rotation angle is the same as the angle θ calculated when determining the joint type, and will not be elaborated further here.

[0233] The following is a detailed explanation of the process of determining the translation axis direction and translation distance of the moving part in the target state.

[0234] For translational motion components, the centroid of the first Gaussian point cloud A is... With the centroid of the second Gaussian point cloud B The difference is used as the translation vector, and the unit vector of the translation vector is used as the direction of the translation axis. The magnitude of the translation vector is used as the translation distance.

[0235] Based on the same inventive concept, this application also provides a modeling device corresponding to the modeling method. Since the principle of the device in this application is similar to the modeling method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0236] Figure 12 A modular structure diagram of the modeling apparatus provided in the embodiments of this application, such as... Figure 12 As shown, the device includes:

[0237] The acquisition module 1201 is used to acquire standard images of the joint object from multiple perspectives and in multiple states.

[0238] The generation module 1202 is used to generate deformed Gaussian fields of the joint object in various states based on the standard image. The deformed Gaussian fields are used to characterize the geometric structure, appearance and motion characteristics of the joint object in the corresponding states.

[0239] The generation module 1202 is also used to perform component segmentation processing based on the deformed Gaussian field, and generate the target Gaussian field of each moving component in the joint object in each state.

[0240] Modeling module 1203 is used to determine the motion parameters of each moving part in each target state based on the target Gaussian field of each moving part in the initial state and each target state, and to model each moving part based on the motion parameters of each moving part in each target state, and to obtain the target model of the joint object in each target state based on the modeling results of each moving part.

[0241] As an optional implementation, the generation module 1202 is specifically used for:

[0242] Based on the standard image, a normalized Gaussian field is generated for the joint object in each state. The normalized Gaussian field includes multiple standard Gaussian elements. The Gaussian parameters of the standard Gaussian elements include the center position of the Gaussian distribution, the covariance matrix, color information, and opacity.

[0243] Based on the target latent variables of the joint object in each state, the standardized Gaussian fields are shifted to obtain the deformed Gaussian fields of the joint object in each state.

[0244] As an optional implementation, the generation module 1202 is specifically used for:

[0245] The latent variables of the target under each state and the standardized Gaussian field are input into the pre-trained target deformation network. The target deformation network predicts and generates the target offset of each standard Gaussian element in each standardized Gaussian field. The target offset includes the target offset at the center position, the target offset in the direction, and the target offset in the scale.

[0246] Based on the offset of each target, the normalized Gaussian field is offset to obtain the deformed Gaussian field of the joint object in each state.

[0247] As an optional implementation, it further includes: a training module 1204; the training module 1204 is used for:

[0248] Based on the standard image, the initial normalized Gaussian field of the joint object in each state is generated. The initial latent variables of the joint object in each state and each initial normalized Gaussian field are input into the initial deformation network. The initial deformation network predicts and generates the initial offset of each initial normalized Gaussian element in each initial normalized Gaussian field.

[0249] Based on each initial offset, each initial normalized Gaussian field is offset to obtain the initial deformed Gaussian field in each state. The first initial rendered image is obtained by rendering based on the initial deformed Gaussian field, and the photometric loss value between the first initial rendered image and the standard image is calculated.

[0250] Based on the photometric loss value, the model parameters of the initial deformable network, each initial latent variable, and the initial Gaussian parameters of each initial standard Gaussian unit are iteratively adjusted until the photometric loss value meets the preset conditions or the number of iterations reaches the prediction number, thus obtaining the target deformable network, the target latent variables in each state, and the target Gaussian parameters of each standard Gaussian unit.

[0251] As an optional implementation, the generation module 1202 is specifically used for:

[0252] Based on the deformed Gaussian field, coarse-grained component segmentation is performed to obtain the number of moving parts of the joint object and the dynamic Gaussian elements of each moving part.

[0253] Based on the dynamic Gaussian elements of each moving part, fine part segmentation is performed to obtain the boundary candidate Gaussian elements of each moving part. Each boundary candidate Gaussian element is split and the split Gaussian elements are corrected to generate the target Gaussian field of each moving part in each state in the joint object.

[0254] As an optional implementation, the generation module 1202 is specifically used for:

[0255] Dynamic Gaussian detection is performed on each deformed Gaussian element in the deformed Gaussian field to determine the dynamic Gaussian element set and the static Gaussian element set. Among them, the maximum displacement of each deformed Gaussian element in the dynamic Gaussian element set between each state is greater than or equal to a preset threshold.

[0256] Multiple pairs of images from different perspectives and states are sampled from a standard image as state image pairs. The state image pairs and preset prompts are input into a visual language model, which then predicts the number of moving parts of the jointed object.

[0257] For each deformable Gaussian element in the dynamic Gaussian element set, a corresponding motion descriptor is assembled, and trajectory clustering is performed based on each motion descriptor. Based on the clustering results, the dynamic Gaussian elements of each moving component are determined.

[0258] As an optional implementation, the generation module 1202 is specifically used for:

[0259] Determine the cumulative weight of each pixel in each moving part under each viewpoint, and based on the cumulative weight, determine the set of visible pixels and the set of non-contributing pixels for each moving part under each viewpoint.

[0260] A first number of positive cue points are sampled from the set of visible pixels, and a second number of negative cue points are sampled from the set of non-contributing pixels. Each positive cue point and each negative cue point is then input into a general segmentation model, which generates a mask for each moving part from each viewpoint.

[0261] The dynamic Gaussian elements of each moving component are projected into two-dimensional ellipses at the corresponding viewpoints. Based on the major axis of each two-dimensional ellipse and the mask of each moving component at each viewpoint, the boundary candidate Gaussian elements of each moving component are determined, and each boundary candidate Gaussian element is split into sub-Gaussian elements within each adjacent moving component.

[0262] Based on the sub-Gaussian elements within each adjacent moving component after splitting and each non-boundary candidate Gaussian element, a second initial rendered image is obtained. Then, based on the photometric loss value between the second initial rendered image and the standard image, the orientation and opacity of the sub-Gaussian elements within each adjacent moving component are iteratively corrected to obtain the corrected sub-Gaussian elements within each adjacent moving component.

[0263] Based on the modified sub-Gaussian elements within each adjacent moving part and each non-boundary candidate Gaussian element, the target Gaussian field of each moving part in the joint object under each state is generated.

[0264] As an optional implementation, the modeling module 1203 is specifically used for:

[0265] Based on the Gaussian parameters of each target Gaussian element in the target Gaussian field, the first Gaussian point cloud of the moving part in the initial state and the second Gaussian point cloud in the target state are determined.

[0266] Based on the first Gaussian point cloud and the second Gaussian point cloud, the joint type of each moving part is determined, wherein the joint type is either translational or rotational.

[0267] If the joint type of the moving part is translational, then the translation axis direction and translation distance of the moving part in the target state are determined based on the first Gaussian point cloud and the second Gaussian point cloud.

[0268] If the joint type of the moving part is rotational, then the rotation axis direction, rotation center and rotation angle of the moving part in the target state are determined based on the first Gaussian point cloud and the second Gaussian point cloud.

[0269] As an optional implementation, the modeling module 1203 is specifically used for:

[0270] Based on the first Gaussian point cloud and the second Gaussian point cloud, determine the rotation angle of the moving part from the initial state to the target state.

[0271] If the rotation angle is greater than or equal to the preset angle, then the joint type of the moving part is determined to be rotational.

[0272] Otherwise, determine that the joint type of the moving part is translational.

[0273] This application also provides an electronic device, such as... Figure 13 The diagram shown is a schematic representation of the structure of an electronic device provided in an embodiment of this application, including: a processor 131, a memory 132, and a bus 133. The memory 132 stores machine-readable instructions executable by the processor 131 (e.g., ...). Figure 12 The device in the embodiment acquires the execution instructions corresponding to the module 1201, the generation module 1202, the modeling module 1203, and the training module 1204. When the electronic device is running, the processor 131 communicates with the memory 132 through the bus 133. When the machine-readable instructions are executed by the processor 131, the steps of the modeling method in the above embodiment are executed.

[0274] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the modeling method described above.

[0275] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0276] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0277] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A modeling method characterized by, The method comprises the following steps: acquiring standard images of a joint object in multiple views and multiple states; generating a deformation Gaussian field of the joint object in each state according to the standard images, the deformation Gaussian field being used to represent geometric structure, appearance and motion characteristics of the joint object in the corresponding state; performing coarse-grained component segmentation processing according to the deformation Gaussian field to obtain the number of motion components of the joint object and dynamic Gaussian primitives of each motion component; determining the cumulative weight of each pixel point in each motion component in each view, and determining the visible pixel set and the non-contributing set of each motion component in each view according to the cumulative weight; sampling a first number of positive hint points from the visible pixel set and a second number of negative hint points from the non-contributing set, and inputting each positive hint point and each negative hint point into a general segmentation model to generate a mask of each motion component in each view by the general segmentation model; projecting the dynamic Gaussian primitives of each motion component into two-dimensional ellipses in the corresponding view, and determining boundary candidate Gaussian primitives of each motion component according to the major axis of each two-dimensional ellipse and the mask of each motion component in each view, and splitting each boundary candidate Gaussian primitive into intra-sub-Gaussian primitives of each adjacent motion component; rendering a second initial rendering image according to the split intra-sub-Gaussian primitives of each adjacent motion component and each non-boundary candidate Gaussian primitive, and iteratively correcting the direction and opacity of each intra-sub-Gaussian primitive of each adjacent motion component according to the photometric loss value between the second initial rendering image and the standard image to obtain corrected intra-sub-Gaussian primitives of each adjacent motion component; generating target Gaussian fields of each motion component in each state of the joint object according to the corrected intra-sub-Gaussian primitives of each adjacent motion component and each non-boundary candidate Gaussian primitive; determining motion parameters of each motion component in each target state according to the target Gaussian fields of each motion component in the initial state and each target state, modeling each motion component according to the motion parameters of each motion component in each target state, and obtaining a target model of the joint object in each target state according to the modeling results of each motion component.

2. The method of claim 1, wherein, The method further comprises the following steps: generating a standardized Gaussian field of the joint object in each state according to the standard images, the standardized Gaussian field comprising a plurality of standard Gaussian primitives, the Gaussian parameters of the standard Gaussian primitives comprising the center position, the covariance matrix, the color information and the opacity of the Gaussian distribution; offsetting each standardized Gaussian field according to the target latent variables of the joint object in each state to obtain the deformation Gaussian field of the joint object in each state.

3. The method of claim 2, wherein, The method further comprises the following steps: offsetting each standardized Gaussian field according to the target latent variables of the joint object in each state to obtain the deformation Gaussian field of the joint object in each state. inputting the target latent variables in each state and the standardized Gaussian fields into a pre-trained target deformation network, and generating target offsets of each standard Gaussian cell in each standardized Gaussian field by the target deformation network, wherein the target offsets include a center position target offset, a direction target offset, and a scale target offset; performing offsetting on each standardized Gaussian field according to the target offsets to obtain a deformation Gaussian field of the articulated object in each state.

4. The method of claim 3, wherein, Before the generating of the target offsets of each standard Gaussian cell in each standardized Gaussian field by the target deformation network, the method comprises: generating initial standardized Gaussian fields of the articulated object in each state according to the standard images, inputting initial latent variables of the articulated object in each state and the initial standardized Gaussian fields into an initial deformation network, and generating initial offsets of each initial standard Gaussian cell in each initial standardized Gaussian field by the initial deformation network; performing offsetting on each initial standardized Gaussian field according to the initial offsets to obtain an initial deformation Gaussian field in each state, rendering a first initial rendering image according to the initial deformation Gaussian field, and calculating a photometric loss value of the first initial rendering image and the standard images; iteratively adjusting model parameters of the initial deformation network, the initial latent variables, and initial Gaussian parameters of each initial standard Gaussian cell according to the photometric loss value until the photometric loss value meets a preset condition or an iteration number reaches a preset number of iterations, to obtain the target deformation network, the target latent variables in each state, and target Gaussian parameters of each standard Gaussian cell.

5. The method of claim 1, wherein, The coarse-grained component segmentation processing according to the deformation Gaussian field comprises: performing dynamic Gaussian detection on each deformation Gaussian cell in the deformation Gaussian field to determine a dynamic Gaussian cell set and a static Gaussian cell set, wherein a maximum displacement of each deformation Gaussian cell in the dynamic Gaussian cell set between states is greater than or equal to a preset threshold value; sampling a plurality of pairs of images in different states under different viewing angles from the standard images as state image pairs, inputting the state image pairs and a preset prompt into a visual language model, and predicting the number of moving components of the articulated object by the visual language model; assembling a corresponding motion descriptor for each deformation Gaussian cell in the dynamic Gaussian cell set, performing trajectory clustering according to each motion descriptor, and determining the dynamic Gaussian cell of each moving component according to a clustering result.

6. The method of claim 1, wherein, The determining of the motion parameters of each moving component in each target state according to the target Gaussian fields of each moving component in a starting state and in each target state comprises: determining a first Gaussian point cloud of the moving component in the starting state and a second Gaussian point cloud of the moving component in the target state according to Gaussian parameters of each target Gaussian cell in the target Gaussian fields; and determining a joint type of each motion component according to the first Gaussian point cloud and the second Gaussian point cloud, wherein the joint type is a translational type or a rotational type; if the joint type of the motion component is the translational type, determining a translational axis direction and a translational distance of the motion component in the target state according to the first Gaussian point cloud and the second Gaussian point cloud; if the joint type of the motion component is the rotational type, determining a rotational axis direction, a rotation center and a rotation angle of the motion component in the target state according to the first Gaussian point cloud and the second Gaussian point cloud.

7. The method of claim 6, wherein, The determining the joint type of each motion component according to the first Gaussian point cloud and the second Gaussian point cloud comprises: determining a rotation angle of the motion component from a start state to the target state according to the first Gaussian point cloud and the second Gaussian point cloud; if the rotation angle is greater than or equal to a preset angle, determining that the joint type of the motion component is the rotational type; otherwise, determining that the joint type of the motion component is the translational type.

8. An electronic device, comprising: comprise: a processor, a memory and a bus, the memory storing machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the processor executes the machine readable instructions, and executes the steps of the modeling method in any one of claims 1 to 7.

Images