A hinge object 3D model generation method and system

Abstract

The application relates to the technical field of three-dimensional content generation, and discloses a hinge object 3D model generation method and system. The method comprises the following steps: receiving a single image and a natural language instruction, generating a three-dimensional initial grid and synthesizing an operation video; double sampling a grid surface to obtain a target geometric point cloud, extracting dense geometric features and anchor point features, and simultaneously extracting spatial and motion features from a video frame to obtain video features; splicing category labels, view angle labels, anchor point features and video features into initial semantic streams, obtaining global semantic features through self-attention, performing bidirectional cross-modal attention to obtain updated global and dense geometric features; respectively predicting joint parameters and component segmentation; performing a differentiable spatial transformation on active components, constraining deformation through forward kinematics loss, mapping the segmentation result back to a grid to derive a hinge model. The application can generate a high-fidelity, physically consistent and simulative hinge model only by using a single image and a single instruction.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional content generation technology, specifically to a method and system for generating a 3D model of a hinged object. Background Technology

[0002] Hinge objects, composed of rigid components connected by kinematic joints, play a crucial role in linking static scene perception with dynamic physical interaction. Reconstructing high-fidelity and physically consistent digital twin models of hinge objects has become an important prerequisite for large-scale embodied intelligent simulation training and robot understanding. However, this task remains extremely challenging due to unknown kinematic structures, diverse input commands, and the difficulty of ensuring both geometric accuracy and physical plausibility.

[0003] Existing methods for 3D reconstruction of hinged objects can be broadly categorized into three types: multi-view observation-based methods, methods incorporating external priors, and methods relying on single-modal input. While these methods improve reconstruction reliability to some extent, they still suffer from significant limitations in practical applications. First, multi-view methods are extremely costly, their performance depending on high-quality camera pose estimation and dense view coverage, thus limiting scalability. Second, methods using movable part masks as priors lead to excessive reliance on auxiliary signals rather than actively understanding the moving structure from static geometry. Finally, methods that rely solely on static geometry inference based on a single image or a single 3D mesh are prone to severe empirical bias. Lacking dynamic temporal evidence, these static inference methods often rely on rote memorization of shape distribution patterns from the training set, resulting in vague kinematic structures lacking realistic physical plausibility when faced with unseen complex structures. Summary of the Invention

[0004] To overcome the limitations of existing 3D hinge object generation methods, which rely on multi-view input and suffer from empirical bias due to static inference, this invention proposes a method and system for generating 3D hinge object models. This invention innovatively uses a video generation model as a world simulator, synthesizing an operational video containing explicit physical dynamic evidence by providing a single image and natural language operation instructions. Simultaneously, this invention designs a video hinge feature extraction network capable of deep bidirectional fusion of dynamic motion cues extracted from the video with static 3D geometric information. Based on this, the network can accurately predict the object's component segmentation results and rigorous kinematic parameters, ultimately directly outputting a 3D hinge object model asset suitable for physical simulation.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for generating a 3D model of a hinged object, comprising: It receives a single image and natural language instructions, generates an initial 3D mesh, and synthesizes the operation video; The surface of the initial 3D mesh is uniformly sampled and sharp edge sampled, and merged into a target geometric point cloud. Dense geometric features and geometric anchor point features are extracted by graph convolution and positional encoding. Spatial features and inter-frame motion features are extracted from the video frames of the operation video, and video features are obtained by time gating. The category label, view label, geometric anchor point features and video features are concatenated into an initial semantic stream. The global semantic features are obtained through self-attention. Bidirectional cross-modal attention is performed based on the dense geometric features and the global semantic features to obtain the updated global semantic features and the updated dense geometric features. The updated global semantic features are input into the motion parameter decoder to predict joint parameters; the updated dense geometric features and geometric anchor features are input into the point cloud decoder to predict the component segmentation category of each 3D point in the target geometric point cloud. The set of active component points is obtained based on the predicted component segmentation categories. Differentiable spatial transformations are performed on the active component point sets using the predicted joint parameters. The consistency between the predicted deformation and the actual deformation is constrained by the positive kinematic loss. The component segmentation categories of each 3D point are mapped back to the 3D initial mesh, and a hinge model with kinematic parameters is derived.

[0006] In one embodiment, the operation video is synthesized through a video generation model that is fine-tuned using low-rank adaptive technology. The video generation model is constrained to follow the kinematic laws of rigid bodies and generate continuous operation videos based on input natural language operation commands.

[0007] In one embodiment, the process of uniformly sampling and sharply sampling the surface of the initial 3D mesh, merging them into a target geometric point cloud, and extracting dense geometric features and geometric anchor point features through graph convolution and positional encoding specifically includes: The uniform point set obtained by uniform sampling Edge point set obtained by edge sampling Merge into a target geometric point cloud, for the first... A three-dimensional point, with the three-dimensional point Nearest neighbor set Construct edge features, encode them using local graph convolution, and then superimpose them with the 3D Fourier position code of the 3D point. To obtain local geometric features : ; ; in, Indicates the first The coordinates of a three-dimensional point Indicates frequency level, Indicates the total number of frequency levels. This represents a local graph convolution encoder. Represents the position-encoded projection matrix; for The first in The coordinates of each 3D point are analyzed; the local geometric features of each 3D point are concatenated into dense geometric features; and the dense geometric features are pooled to obtain geometric anchor point features. .

[0008] In one embodiment, the step of extracting spatial features and inter-frame motion features from video frames of the operating video, and obtaining video features through time gating, specifically includes: For the operation video, the first video frames Spatial features are extracted using a frozen visual pre-trained model. Calculate the absolute pixel difference between adjacent frames. Motion features are obtained through a motion feature encoder. Then, a time-gated adapter containing one-dimensional time convolution is used to generate gating coefficients. Spatial features, gated motion features, and three-dimensional spatiotemporal position coding are fused into frame features. : ; ; ; ; ; in, This represents a frozen visual pre-trained model. For the first One video frame, Represents a motion feature encoder. This represents a one-dimensional convolution along the time dimension. For feature splicing, express function, Representation layer normalization, Indicates the video frame number and the height and direction coordinates of the pixel Width direction Three-dimensional spatiotemporal position encoding of coordinates; Frame features of each video frame Arranged in chronological order, video features are obtained. .

[0009] In one embodiment, the step of performing bidirectional cross-modal attention based on dense geometric features and global semantic features to obtain updated global semantic features and updated dense geometric features specifically includes: With dense geometric features For query, global semantic features Dense-to-semantic cross-attention is performed on the keys and values ​​to obtain updated dense geometric features injected with video motion cues. Then, using the global semantic features as the query and the updated dense geometric features as the key and value, semantic-to-dense cross-attention is performed to obtain the updated global semantic features. : ; ; in, This indicates scaling dot product attention.

[0010] In one embodiment, the joint parameters include joint type, joint axis parameters, and range of motion; the range of motion includes the initial state and relative motion. When the joint type is rotational, the motion parameter decoder outputs six-dimensional Plück coordinates containing direction vectors and spatial moment vectors to determine the joint axis parameters and output the initial state and angle range of rotation. When the joint type is translational, the motion parameter decoder outputs a unit three-dimensional axis as the joint axis parameter, and outputs the initial state and relative displacement of the translation.

[0011] In one embodiment, the component segmentation category of each 3D point in the predicted target geometric point cloud specifically includes: For the target geometric point cloud The first in Each of the three-dimensional points is in a set of 16 nearest neighbors. and 32 nearest neighbor set The updated dense geometric features, geometric anchor point features, and relative coordinates of the 3D point are aggregated and stitched together to obtain multi-scale point features. Then predict the occupancy classification logical value of the 3D point. and contrast feature vectors and according to Determine whether the 3D point belongs to a fixed base or a moving part: ; ; ; in, Indicates the aggregation of neighborhood features. Indicates splicing, This represents a point cloud decoder. Represents the geometric point cloud of the target The Middle Part segmentation categories for 3D points express The Middle The updated dense geometric features corresponding to each 3D point express The Middle Geometric anchor point features corresponding to each 3D point For the first The coordinates of a three-dimensional point express The Middle The coordinates of a three-dimensional point express The Middle The updated dense geometric features corresponding to each 3D point express The Middle Geometric anchor point features corresponding to each 3D point express The Middle The coordinates of a three-dimensional point; This refers to the softmax function.

[0012] In one embodiment, the step of performing a differentiable spatial transformation on the set of points of the moving parts using predicted joint parameters, and constraining the consistency between the predicted deformation and the actual deformation through a positive kinematic loss constraint, specifically includes: Obtain the set of points predicted as active components from the target geometric point cloud. The predicted joint axis parameters and relative motion Acting on Each three-dimensional point in and with differentiable space transformation Calculate the average between predicted deformation and actual deformation Distance, resulting in differentiable positive kinematic loss : ; in, and These represent the actual joint axis parameters and the actual relative motion, respectively. It is an L1 norm; For rotational joints , It is a unit direction vector. It is a spatial moment vector. The point on the axis of rotation that is closest to the origin. For translational joints , The unit is the translation axis.

[0013] In one embodiment, mapping the component segmentation categories of each 3D point back to the initial 3D mesh to derive a hinge model with kinematic parameters specifically includes: The KD tree is used to map the component segmentation categories of 3D points back to the initial 3D mesh, cut out independent rigid sub-components, and export them as a unified robot description format or an asset file compatible with the physics engine.

[0014] In a second aspect, the present invention provides a 3D model generation system for hinged objects, used to implement the method of any embodiment of the first aspect, comprising: The initialization module is used to receive a single image and natural language instructions, generate a 3D initial mesh, and synthesize the operation video; The feature extraction module is used to uniformly sample and sharply sample the surface of the initial 3D mesh, merge them into the target geometric point cloud, and extract dense geometric features and geometric anchor point features through graph convolution and position encoding; it extracts spatial features and inter-frame motion features from the video frames of the operation video, and obtains video features through time gating; The cross-modal fusion module is used to concatenate category labels, viewpoint labels, geometric anchor features and video features into an initial semantic stream, obtain global semantic features through self-attention, and perform bidirectional cross-modal attention based on dense geometric features and global semantic features to obtain updated global semantic features and updated dense geometric features. The decoding module is used to input the updated global semantic features into the motion parameter decoder to predict joint parameters; and to input the updated dense geometric features and geometric anchor features into the point cloud decoder to predict the part segmentation category of each 3D point in the target geometric point cloud. The optimization export module is used to obtain the set of active part points based on the predicted part segmentation category, perform differentiable spatial transformation on the active part point set using the predicted joint parameters, and constrain the consistency between the predicted deformation and the actual deformation through positive kinematic loss. The part segmentation category of each 3D point is mapped back to the 3D initial mesh, and the hinge model with kinematic parameters is exported.

[0015] The system and method in this invention correspond to each other; the specific technical solutions applicable to the method are also applicable to the system.

[0016] Compared with the prior art, the beneficial technical effects of the present invention are: This invention overcomes the bottleneck of empirical bias in static geometric inference: compared with traditional methods that rely solely on single images or static meshes, this invention provides the model with intuitive physical intuition through synthesized instruction-driven dynamic video. This enables the model to infer moving parts based on actual dynamic physical evolution, rather than simply memorizing the geometric shape distribution in the training set. Therefore, it maintains extremely high structural control accuracy and physical plausibility even when faced with rare or out-of-distribution object structures.

[0017] This invention significantly improves the accuracy of hinge component segmentation and kinematic parameter prediction: through a unique dual-stream point cloud sampling (combining uniform sampling and edge sampling) and a bidirectional cross-modal attention mechanism, this invention can accurately delineate complex motion boundaries at a dense 3D point cloud level. In authoritative benchmark tests, compared to the current state-of-the-art static inference methods, this invention achieves a groundbreaking performance leap in both 3D volume intersection-over-union (gIoU) and component segmentation accuracy in both static and fully moving states.

[0018] This invention enables low-cost, high-fidelity automated production of physical assets: requiring only a single image and a simple natural language command, it can output an end-to-end 3D model containing physical joint parameters (such as rotation axes, unit translation axes, and range of motion defined by six-dimensional Plück coordinates). This model is compatible with general-purpose physical simulation engines, reducing the costs of data acquisition and manual modeling for building digital twin environments in embodied intelligence, robot interactive training, and mixed reality applications. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0020] Figure 2 This is a schematic diagram of the framework structure of the present invention. Detailed Implementation

[0021] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] like Figure 1 As shown, a method for generating a 3D model of a hinged object according to the present invention includes the following steps: S1 receives a single image and natural language instructions, generates a 3D initial mesh, and synthesizes the operation video; S2 performs uniform sampling and sharp edge sampling on the surface of the initial 3D mesh, merges them into the target geometric point cloud, and extracts dense geometric features and geometric anchor point features through graph convolution and positional encoding; extracts spatial features and inter-frame motion features from the video frames of the operation video, and obtains video features through temporal gating. S3 concatenates category labels, viewpoint labels, geometric anchor features and video features into an initial semantic stream, obtains global semantic features through self-attention, and performs bidirectional cross-modal attention based on dense geometric features and global semantic features to obtain updated global semantic features and updated dense geometric features. S4. Input the updated global semantic features into the motion parameter decoder to predict joint parameters; input the updated dense geometric features and geometric anchor point features into the point cloud decoder to predict the component segmentation category of each 3D point in the target geometric point cloud. S5: Obtain the set of active component points based on the predicted component segmentation categories, perform a differentiable spatial transformation on the set of active component points using the predicted joint parameters, and constrain the consistency between the predicted deformation and the actual deformation through positive kinematic loss; map the component segmentation categories of each 3D point back to the 3D initial mesh, and derive the hinge model with kinematic parameters.

[0023] Figure 2 This is the overall structure of the present invention, which will be described in detail in several parts below.

[0024] 1. Task request and multimodal data initialization.

[0025] When a task to generate a 3D model of a hinged object is received, the system first receives a single image input by the user, along with the corresponding natural language operation instructions. A basic 3D generation model is used to transform the input image into a static initial 3D mesh. Simultaneously, a finely tuned video generation model is used, combined with the input image and the natural language operation instructions, to synthesize a continuous dynamic operation video, serving as explicit dynamic evidence of the target's physical operation.

[0026] In a preferred embodiment, the 3D generation model can be at least one of Hunyuan3D 1.0, SAM 3D, or Seed3D 1.0; the video generation model can be Wan2.1-I2V-14B-480p, and fine-tuned using low-rank adaptive technology. After Wan2.1 generates 17 frames of operational video, one intermediate frame is removed to obtain 16 frames of operational video for subsequent processing. .

[0027] 2. Dual-stream point cloud coding and multimodal feature extraction.

[0028] To accurately analyze the topological structure of objects, this invention performs double sampling on the initial 3D mesh: on the one hand, it extracts a uniformly distributed point cloud to ensure global shape coverage; on the other hand, it extracts a point cloud with sharp edges to clearly preserve key high-frequency geometric boundary details. Regarding visual features, this invention utilizes a visual pre-trained model to extract spatial features from video frames and calculates inter-frame pixel differences to extract dynamic motion features, subsequently fusing them with temporal information into a unified video feature sequence.

[0029] Specifically, for the target geometric point cloud The first in A three-dimensional point, with its Nearest neighbor set Construct edge features, encode them using local graph convolution, and then superimpose them with the 3D Fourier positional code of the 3D point to obtain local geometric features. .

[0030] For uniform point sets respectively and edge point set The above encoding is performed, and the resulting local geometric features are concatenated into dense geometric features. and to Pooling is used to obtain geometric anchor point features. .

[0031] Operation video Unified to 16 frames; for the first video frames Spatial features are extracted using a frozen visual pre-trained model. Calculate the absolute pixel difference between adjacent frames. Motion features are obtained through a motion feature encoder. Then, a time-gated adapter containing one-dimensional time convolution is used to generate gating coefficients. It integrates spatial features, gated motion features, and three-dimensional spatiotemporal position coding into video features. .

[0032] In a preferred embodiment, the visual pre-trained model employs DINOv2 ViT-B / 14 with frozen parameters to extract spatial block features from each video frame of the 16-frame operational video.

[0033] 3. Bidirectional cross-modal fusion of static topology and dynamic vision.

[0034] Category tag Learnable camera view markers , from the target geometric point cloud Dense geometric features Geometric anchor point features obtained by convergence and video features Concatenate into an initial global semantic feature stream And through self-attention Obtain global semantic features Subsequently, dense-to-semantic cross-attention and semantic-to-dense cross-attention are performed sequentially to obtain the updated dense geometric features. and updated global semantic features .

[0035] 4. Task-specific decoding and kinematic parameter prediction.

[0036] After bidirectional cross-modal fusion, the updated global semantic features will be... The global time-stamped input motion parameter decoder is used to classify joint types. And regress joint axis parameters and range of motion Updated dense geometric features With geometric anchor point features Input point cloud decoder, targeting the target geometric point cloud Each 3D point prediction in the dataset occupies a classification logical value and a contrastive feature vector, and is obtained through... and Multi-scale nearest neighbor feature aggregation distinguishes between fixed base and moving parts.

[0037] 5. Physical constraint optimization and simulation model export.

[0038] From the target geometric point cloud Obtain the set of points predicted to be active components. The predicted joint axis parameters and relative motion Acting on Each three-dimensional point in and with differentiable space transformation Calculate the average between predicted deformation and actual deformation Distance, resulting in differentiable positive kinematic loss .

[0039] After training, the target geometric point cloud is generated using a KD tree. Point-by-point component segmentation categories are mapped back to the high-resolution 3D initial mesh. Split the fixed base sub-mesh and the moving part sub-mesh, and add joint types. Joint axis parameters and range of motion Export the 3D model as a Unified Robot Description Format or MuJoCo XML format.

[0040] This invention innovatively introduces a video generation model as a simulator of the physical world. By transforming implicit generation dynamics into explicit physical motion evidence, it completely eliminates the reliance on multi-view images and manually labeled masks.

[0041] This invention designs a video hinge feature extraction network and its bidirectional cross-modal fusion mechanism, which realizes the deep alignment of local high-frequency three-dimensional geometric details with global dynamic video spatiotemporal cues, effectively eliminating physical ambiguity in static geometric inference.

[0042] This invention introduces a differentiable positive kinematic loss function for the generation process, directly binding abstract kinematic parameters with explicit three-dimensional spatial deformation, ensuring that the generated hinge 3D model has extremely high physical consistency and out-of-the-box simulation capabilities.

[0043] Example: Upon receiving the target task, this invention acquires a static object image. and a natural language operation command describing the target action. For example, "Open the left door of the cabinet." The 3D model was generated using Hunyuan3D 1.0, which uses images... Convert to a 3D initial mesh without joint properties The video generation model uses Wan2.1-I2V-14B-480p, finely tuned with low-rank adaptive technology, based on the image. and operating instructions Generate 17 consecutive video frames, and remove one intermediate frame to obtain 16 operation video frames. 3D models can also be generated using SAM 3D or Seed3D 1.0.

[0044] For 3D initial mesh A uniform point set is formed by sampling 2048 uniform points on its surface. An edge point set was constructed by sampling 2048 sharp edge points. ,Will and Merged into a target geometric point cloud containing 4096 three-dimensional points The two point sets are respectively passed through The local image convolutional encoding is fused with 3D Fourier positional encoding, and the resulting features are concatenated into 768-dimensional dense geometric features. The geometric anchor point features are obtained through pooling. .

[0045] For the target geometric point cloud The first in A three-dimensional point, with its Nearest neighbor set Construct edge features, encode them using local graph convolution, and then superimpose them with the 3D Fourier positional code of the 3D point to obtain local geometric features. 3D Fourier position coding and local geometric features Calculate according to the following formula: ; ; in, Indicates the first Three-dimensional point coordinates, Indicates frequency level, Indicates the total number of frequency levels. This represents a local graph convolution encoder. Represents the position-encoded projection matrix; for each uniform point set and edge point set The above encoding is performed, and the resulting features are concatenated into dense geometric features. and to Pooling is used to obtain geometric anchor point features. .

[0046] Operation video Unified to 16 frames; for the first frame Spatial features were extracted using a frozen DINOv2 visual pre-trained model. Calculate the absolute pixel difference between adjacent frames. Motion features are obtained through a motion feature encoder. Then, a time-gated adapter containing one-dimensional time convolution is used to generate gating coefficients. It integrates spatial features, gated motion features, and three-dimensional spatiotemporal position coding into video features. Calculate according to the following formula: ; ; ; ; ; in, This represents a frozen visual pre-trained model. Represents a motion feature encoder. This represents a one-dimensional convolution along the time dimension. express function, Representation layer normalization, Indicates the video frame number and the height and direction coordinates of the pixel Width direction Three-dimensional spatiotemporal position encoding of coordinates; [and] each Video features are obtained by arranging them in chronological order. .

[0047] This embodiment uses 6 dual-stream Transformer blocks to implement bidirectional cross-modal fusion, and each cross-attention layer contains 12 attention heads.

[0048] Category tag Learnable camera view markers , from the target geometric point cloud Dense geometric features Geometric anchor point features obtained by convergence and video features Concatenate into an initial global semantic feature stream And through self-attention Obtain global semantic features Subsequently, dense-to-semantic cross-attention and semantic-to-dense cross-attention are performed sequentially to obtain the updated dense geometric features. and updated global semantic features Calculate according to the following formula: ; ; ; ; ; in, This indicates scaling dot product attention. , , These represent the query matrix, key matrix, and value matrix, respectively. Indicates the dimension of the key vector; This is an updated dense geometric feature after injecting video motion cues. To utilize Global semantic features updated in reverse.

[0049] Updated global semantic features The global time stamp is input to the motion parameter decoder. The motion parameter decoder first uses a classifier to determine the joint type. It can be either a rotational or translational joint; the output of a rotational joint is a unit direction vector. and space moment vector The six-dimensional Plück coordinates constituted and initial state and relative angle Translational joint output unit translation axis and initial state and relative displacement The updated dense geometric features and geometric anchor features Input the point cloud decoder and perform the following multi-scale nearest neighbor aggregation and point-by-point classification: ; ; ; in, and They represent the first The 16 nearest neighbor set and the 32 nearest neighbor set of a 3D point Indicates the aggregation of neighborhood features. This represents a point cloud decoder. Represents the geometric point cloud of the target The Middle The component segmentation category is a 3D point.

[0050] From the target geometric point cloud Obtain the set of points predicted to be active components. The predicted joint axis parameters and relative motion Acting on Each three-dimensional point in and with differentiable space transformation Calculate the average between predicted deformation and actual deformation Distance, resulting in differentiable positive kinematic loss Calculate according to the following formula: ; in, and These represent the actual joint axis parameters and the actual relative motion, respectively.

[0051] For rotational joints , It is a unit direction vector. For spatial moment vectors: ; .

[0052] For translational joints , Unit translation axis: ; To bypass Rotation The rotation matrix, It is the point on the rotation axis that is closest to the origin.

[0053] During the inference phase, the target geometric point cloud is constructed using a KD tree. The component segmentation categories of each 3D point are mapped back to the initial 3D mesh. Cut out the fixed base sub-mesh and the moving part sub-mesh; combine the predicted joint type. Joint axis parameters and range of motion Export as a simulable asset file in the Unified Robot Description Format (URDF) or MuJoCo XML format.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0055] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0056] This invention also provides a system. The system can be a system that uses software (applications), modules, components, servers, clients, etc., employing the methods described in the embodiments of this specification, combined with necessary hardware implementation. Since the implementation schemes and methods for solving the problem are similar, the specific system implementations in the embodiments of this specification can be found in the implementations of the foregoing methods, and repeated details will not be elaborated upon. Although the system is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0057] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0059] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for generating a 3D model of a hinged object, characterized in that, include: It receives a single image and natural language instructions, generates an initial 3D mesh, and synthesizes the operation video; The surface of the initial 3D mesh is uniformly sampled and sharp edge sampled, and merged into a target geometric point cloud. Dense geometric features and geometric anchor point features are extracted through graph convolution and position encoding. Spatial features and motion features between adjacent frames are extracted from the video frames of the operation video, and then video features are obtained by time gating. The category label, view label, geometric anchor point features and video features are concatenated into an initial semantic stream. The global semantic features are obtained through self-attention. Bidirectional cross-modal attention is performed based on the dense geometric features and the global semantic features to obtain the updated global semantic features and the updated dense geometric features. The updated global semantic features are input into the motion parameter decoder to predict joint parameters; the updated dense geometric features and geometric anchor features are input into the point cloud decoder to predict the component segmentation category of each 3D point in the target geometric point cloud. The set of active component points is obtained based on the predicted component segmentation categories. Differentiable spatial transformations are performed on the active component point sets using the predicted joint parameters. The consistency between the predicted deformation and the actual deformation is constrained by the positive kinematic loss. The component segmentation categories of each 3D point are mapped back to the 3D initial mesh, and a hinge model with kinematic parameters is derived.

2. The method for generating a 3D model of a hinged object according to claim 1, characterized in that, The operation video is synthesized through a video generation model, which is fine-tuned using low-rank adaptive technology. The video generation model is constrained to follow the kinematic physical laws of rigid bodies and generate continuous operation videos based on input natural language operation commands.

3. The method for generating a 3D model of a hinged object according to claim 1, characterized in that, The process involves uniformly sampling the surface of the initial 3D mesh and sampling sharp edges, merging them into a target geometric point cloud, and extracting dense geometric features and geometric anchor point features through graph convolution and positional encoding. Specifically, this includes: The uniform point set obtained by uniform sampling Edge point set obtained by edge sampling Merge into a target geometric point cloud, for the first... A three-dimensional point, with the three-dimensional point Nearest neighbor set Construct edge features, encode them using local graph convolution, and then superimpose them with the 3D Fourier position code of the 3D point. To obtain local geometric features : ； ； in, Indicates the first The coordinates of a three-dimensional point Indicates frequency level, Indicates the total number of frequency levels. This represents a local graph convolution encoder. Represents the position-encoded projection matrix; for The first in The coordinates of each 3D point are analyzed; the local geometric features of each 3D point are concatenated into dense geometric features; and the dense geometric features are pooled to obtain geometric anchor point features. .

4. The method for generating a 3D model of a hinged object according to claim 1, characterized in that, The extraction of spatial features and inter-frame motion features from video frames of the operation video, followed by temporal gating to obtain video features, specifically includes: For the operation video, the first video frames Spatial features are extracted using a frozen visual pre-trained model. Calculate the absolute pixel difference between adjacent frames. Motion features are obtained through a motion feature encoder. Then, a time-gated adapter containing one-dimensional time convolution is used to generate gating coefficients. Spatial features, gated motion features, and three-dimensional spatiotemporal position coding are fused into frame features. : ； ； ； ； ； in, This represents a frozen visual pre-trained model. For the first One video frame, Represents a motion feature encoder. This represents a one-dimensional convolution along the time dimension. For feature splicing, express function, Representation layer normalization, Indicates the video frame number and the height and direction coordinates of the pixel Width direction Three-dimensional spatiotemporal position encoding of coordinates; Frame features of each video frame Arranged in chronological order, video features are obtained. .

5. The method for generating a 3D model of a hinged object according to claim 1, characterized in that, The process of performing bidirectional cross-modal attention based on dense geometric features and global semantic features to obtain updated global semantic features and updated dense geometric features specifically includes: With dense geometric features For query, global semantic features Dense-to-semantic cross-attention is performed on the keys and values ​​to obtain updated dense geometric features injected with video motion cues. Then, using the global semantic features as the query and the updated dense geometric features as the key and value, semantic-to-dense cross-attention is performed to obtain the updated global semantic features. : ； ； in, This indicates scaling dot product attention.

6. The method for generating a 3D model of a hinged object according to claim 1, characterized in that, The joint parameters include joint type, joint axis parameters, and range of motion; the range of motion includes the initial state and relative motion. When the joint type is rotational, the motion parameter decoder outputs six-dimensional Plück coordinates containing direction vectors and spatial moment vectors to determine the joint axis parameters and output the initial state and angle range of rotation. When the joint type is translational, the motion parameter decoder outputs a unit three-dimensional axis as the joint axis parameter, and outputs the initial state and relative displacement of the translation.

7. A method for generating a 3D model of a hinged object according to claim 1, characterized in that, The component segmentation category of each 3D point in the predicted target geometric point cloud specifically includes: For the target geometric point cloud The first in Each of the three-dimensional points is in a set of 16 nearest neighbors. and 32 nearest neighbor set The updated dense geometric features, geometric anchor point features, and relative coordinates of the 3D point are aggregated and stitched together to obtain multi-scale point features. Then predict the occupancy classification logical value of the 3D point. and contrast feature vectors and according to Determine whether the 3D point belongs to a fixed base or a moving part: ； ； ； in, Indicates the aggregation of neighborhood features. Indicates splicing, This represents a point cloud decoder. Represents the geometric point cloud of the target The Middle Part segmentation categories for 3D points express The Middle The updated dense geometric features corresponding to each 3D point express The Middle Geometric anchor point features corresponding to each 3D point For the first The coordinates of a three-dimensional point express The Middle The coordinates of a three-dimensional point express The Middle The updated dense geometric features corresponding to each 3D point express The Middle Geometric anchor point features corresponding to each 3D point express The Middle The coordinates of a three-dimensional point; This refers to the softmax function.

8. A method for generating a 3D model of a hinged object according to claim 1, characterized in that, The process of performing a differentiable spatial transformation on the point set of moving parts using predicted joint parameters and constraining the consistency between predicted deformation and actual deformation through positive kinematic loss specifically includes: Obtain the set of points predicted as active components from the target geometric point cloud. The predicted joint axis parameters and relative motion Acting on Each three-dimensional point in and with differentiable space transformation Calculate the average between predicted deformation and actual deformation Distance, resulting in differentiable positive kinematic loss : ； in, and These represent the actual joint axis parameters and the actual relative motion, respectively. It is an L1 norm; For rotational joints , It is a unit direction vector. It is a spatial moment vector. The point on the axis of rotation that is closest to the origin. For translational joints , The unit is the translation axis.

9. A method for generating a 3D model of a hinged object according to claim 1, characterized in that, The process of mapping the component segmentation categories of each 3D point back to the initial 3D mesh and deriving a hinge model with kinematic parameters specifically includes: The KD tree is used to map the component segmentation categories of 3D points back to the initial 3D mesh, cut out independent rigid sub-components, and export them as a unified robot description format or an asset file compatible with the physics engine.

10. A 3D model generation system for hinged objects, used to implement the method as described in any one of claims 1 to 9, characterized in that, include: The initialization module is used to receive a single image and natural language instructions, generate a 3D initial mesh, and synthesize the operation video; The feature extraction module is used to uniformly sample and sample sharp edges of the surface of the initial 3D mesh, merge them into the target geometric point cloud, and extract dense geometric features and geometric anchor point features through graph convolution and position encoding. Spatial features and motion features between adjacent frames are extracted from the video frames of the operation video, and then video features are obtained by time gating. The cross-modal fusion module is used to concatenate category labels, viewpoint labels, geometric anchor features and video features into an initial semantic stream, obtain global semantic features through self-attention, and perform bidirectional cross-modal attention based on dense geometric features and global semantic features to obtain updated global semantic features and updated dense geometric features. The decoding module is used to input the updated global semantic features into the motion parameter decoder to predict joint parameters; and to input the updated dense geometric features and geometric anchor features into the point cloud decoder to predict the part segmentation category of each 3D point in the target geometric point cloud. The optimization export module is used to obtain the set of active part points based on the predicted part segmentation category, perform differentiable spatial transformation on the active part point set using the predicted joint parameters, and constrain the consistency between the predicted deformation and the actual deformation through positive kinematic loss. The part segmentation category of each 3D point is mapped back to the 3D initial mesh, and the hinge model with kinematic parameters is exported.

Images